My Article-to-Reel Pipeline in One Diagram
One published article in, one 9:16 MP4 out, and the five stages that decide everything between.
The voice is fine, but the captions drift a syllable late. The captions are fine, but the motion is dead. The motion is fine, but the last scene’s text overflows its card. Each of those is a different stage failing, and knowing which stage is the entire reason I drew the diagram.
This post is that diagram. One published article goes in. One 9:16 MP4 comes out. Five stages in between, and a cache layer that quietly decides whether re-rendering costs me money or nothing.
If you read the post Zero, this is the same map zoomed in one level. Vocabulary first, then five posts of detail.
The whole machine — five stages, one shared JSON contract, one cache.
Preferred to watch? Here is 30 seconds TLDR.
Hi, I’m Daniel — a software engineer, solo builder, and the person behind Digital Craft Workshop.
I’ve spent ten years writing TypeScript, .NET, and React for a living, and the last few of them building things alongside the day job: B2B SaaS, AI tools like Article Forge, internal infrastructure, and a Unity 2D narrative game. Some shipped. Some I killed. Both are useful.
Digital Craft Workshop is for developers, AI builders, and indie founders who’d rather understand the craft than ride the hype — the kind of reader who likes Domain-Driven Design, distrusts gurus, and wants AI that’s cheap, practical, and accountable to one engineer in a workshop.
Each essay is something you can read in one sitting and use the same week: an architecture pattern, an AI workflow a single developer can actually run, a build log, or a postmortem on a project that didn’t make it. Built carefully. Explained plainly. Owned by you.
What goes in, what comes out
The input is a published article — not a draft. That word is load-bearing. The pipeline scrapes the cover image and any inline diagrams straight off the live post, so the article has to exist at a public URL first.
A draft has nothing to scrape.
The output is a 1080×1920 H.264 MP4, somewhere between 30 and 90 seconds, depending on the article. The audio is normalized to −14 LUFS, so it sits at the same loudness as everything else in an Instagram or YouTube Shorts feed.
You download it, you upload it. That’s the whole last mile.
Between those two ends sits one button in my article editor. Clicking it is a POST:
POST /api/articles/{id}/video → start a render job, returns immediately
GET /api/articles/{id}/video → poll job status until the MP4 URL appears
The POST doesn’t wait around. It writes a job row to Postgres, kicks off the render in the background, and hands control back in a second.
The render itself takes 45 to 90 seconds; the UI polls the GET endpoint until the MP4 URL shows up. Vocabulary for the rest of the series: the thing the button creates is a render job, and it moves through the five stages below in order.
The five stages
Stage 1 — Scenario. Claude Haiku 4.5 reads the full article and writes a screenplay: three to five scenes, each one a single narration line, a short on-screen text label, and an image assignment. The output is JSON. This file is the contract that the whole pipeline runs on. Part 2 builds this stage.
Stage 2 — Voice. ElevenLabs narrates each scene’s line in a clone of my voice.
The audio matters, but the output I actually came for is the per-word timestamps — every word tagged with the millisecond it starts and ends. The $5 tier hands you those. Most tools don’t. Part 3.
Stage 3 — Captions. Those timestamps become an ASS subtitle file with karaoke timing — each word turns red at the exact millisecond the voice says it. The captions are pinned to a fixed point near the bottom of the frame so they never jump between scenes. Part 4.
Stage 4 — Visuals. Every scene is built from three image layers, rendered with sharp and SVG at 2160×3840: a background, the static brand chrome (the DCW header, the red bar), and the animated content. Then FFmpeg’s zoompan filter adds a slow drift — the Ken Burns move. Part 5.
Stage 5 — Compose. FFmpeg fuses one scene at a time: visuals on the bottom, captions and chrome on top, narration mixed and loudness-normalized. Then it concatenates the scenes with a 0.4-second pause between each. The result is the MP4.
Part 6 swaps the bottom layer of this stage from a still image to real video. But it takes more time, and I will post about it in the last post of the series.
One real run, end to end
Abstract stages are easy to nod along to and hard to picture. So here is one render, start to finish.
The input was my post “The $2 Bill That Made Me Self-Host My Memory” — around 1,400 words, already live on Substack.
Stage 1. Haiku read all 1,400 words and returned five scenes, 135 words of narration in total. One API call, about a tenth of a cent.
Stage 2. ElevenLabs narrated those 135 words — roughly 740 characters of text-to-speech — in my cloned voice. That is 740 of the 30,000 characters the $5 plan gives me each month.
Stage 3. The timestamps came back, one entry per word, and became a 135-line ASS caption file.
Stage 4. Five scenes, three layers each — fifteen PNGs rendered at 2160×3840.
Stage 5. FFmpeg fused them into a 1080×1920 MP4: 55 seconds long, about 17 MB.
Button to finished file: roughly 70 seconds. Marginal cost: about a cent of ElevenLabs and a rounding error of Haiku. Every other post in this series is just one of those five stages, taken slowly.
The contract between stages
Here is the thing that lets this be five separate posts instead of one 6,000-word tangle: the stages don’t know about each other. They each know about one JSON file.
Stage 1 writes it. Stages 2 through 5 read it. Trimmed to a single scene, it looks like this:
{
“scenes”: [
{
“narration”: “Two weeks ago I clicked a button and a video fell out.”,
“visualText”: “ONE BUTTON”,
“imageRole”: “gallery”,
“imageName”: “workshop-bench”
}
],
“durationSeconds”: 45
}
The voice stage reads narration. The caption stage reads narration again, plus the timestamps ElevenLabs hands back.
The visual stage reads visualText and imageName. Nobody reaches across a stage boundary. Restyle the captions, and stage 1 never notices.
That isolation is why a bug lives in exactly one box — and why I can hand you one stage per week without the posts collapsing into each other.
The cache that makes iteration free
The obvious problem with a pipeline that calls two paid APIs: every render costs money. A Haiku call to write the scenario, an ElevenLabs call to narrate it. Re-render a reel six times in an evening, tuning the caption color, and you’ve paid six times for narration that never changed.
So the last successful render’s scenario JSON and per-scene audio get cached — keyed on the narration text itself.
Change the brand color, the caption style, the motion, the scene timing — the narration text is identical, so the cache hits. The re-render spends $0 in AI credits.
Change a single word of narration and the key for that one scene stops matching, so that scene — and only that scene — gets fresh TTS. There’s a ?fresh=1 escape hatch for when I want the whole thing rebuilt from scratch.
In practice, that means a typical evening of tuning — six or seven re-renders chasing a caption color or a scene cut — costs me the price of the first render and nothing after.
The cache is what makes this a tool I actually use instead of one I’m scared to click. If iteration costs a dollar a try, I’d stop iterating — I’ve watched that exact dynamic quietly kill side projects of mine before.
What’s still wrong
The machine runs end-to-end. One button, about 45 seconds of finished reel — voiced, captioned, on-brand.
But stage 4 is lying to you.
The “motion” is a Ken Burns zoom — a slow pan across a still image. On its own, it looks fine. Side by side with anything genuinely animated, it reads as a slide deck wearing a video costume.
Every stage does what it claims — except stage 4. The motion stage is the one that should make the reel feel alive, and it doesn’t.
That’s the named obstacle this series walks toward. Part 5 builds the Ken Burns layer properly and is honest about its ceiling. Part 6 throws it out for real per-scene AI animation running on a 12GB GPU in the corner of my office.
Next Wednesday, Part 2: how Claude Haiku turns an article into a screenplay — and why that JSON file, not the GPU, is the most important part of the whole machine.
One question before Part 2: which of the five stages would you build first — and what does your current video or reel workflow cost you per render right now? Tell me in the comments.
If “five stages, each failing in its own box” is an architecture you’d actually want to build, restack this. Someone in your feed is mid-build on the same machine.
Ok, I clearly have the patience of a gnat, lol.
Read your post dated May 15th and see you're halfway there.
This is SO great. I already have ElevenLabs and a digital character for my brand... I can't wait to try this.