Programmatic Video Engine
I write the source material. Code and local ML manufacture the finished video.
Overview
Making video by hand does not scale, so I built a pipeline that manufactures it. I write or record the source material, and everything after that is automatic: transcription, alignment, asset selection and the final render. There is no editor in the loop and no cloud service on the bill. Every stage, including the ML, runs locally on my own hardware at zero inference cost.
The pipeline
The composition itself is code. I wrote the video template once in Remotion, which treats a video as a React component tree, and every video since reuses it. A Node.js orchestrator runs the stages in order. First, a Whisper-class local model transcribes the narration and aligns it against the audio, so the pipeline knows what is being said and when. That timing data is what lets on-screen elements land in step with the voice without anyone scrubbing a timeline.
From there, a small local LLM plans the video. It reads the material and emits a JSON shot-list: a plain data document describing what should be on screen and when. Fixed code I wrote turns that shot-list into a concrete Remotion composition, and @remotion/renderer renders it programmatically to H.264. Batch scripts then produce each entry in two formats from the same source, a long-form version and a vertical short, in one pass.
Data, not code
The design decision the whole system rests on is that the model never writes code. It only ever emits JSON conforming to a shape my renderer understands, and everything that actually constructs the video is deterministic code I wrote and can test. That makes model quality almost irrelevant: a small free local model can fill in a data structure far more reliably than it could ever write working rendering code. It is also what keeps the pipeline entirely local and entirely free. I do not need a frontier model to fill in a shot-list, so nothing leaves the machine and nothing costs money per video.
How it runs
The whole thing runs on a memory-constrained machine, and it only works because rendering is staged per entry rather than held as one giant job. Each video is prepared, rendered and written out before the next begins, which keeps peak memory inside what the hardware can actually give. I started the project in April 2026 and it is ongoing. A human provides the source, data describes the video, and fixed code manufactures it.
The hardest problem
The hard problem was getting a reliable, fully automated video pipeline out of a memory-constrained machine without paying for any cloud inference. Two decisions made it work. First, the local model only ever emits JSON data, never code. Everything that constructs the video is fixed, deterministic code, and the model just fills in a shot-list. That makes model quality almost irrelevant, so a small free local model is enough, and it keeps the entire pipeline local at zero inference cost. Second, rendering is staged per entry so each video fits in RAM: prepare, render, write out, move on to the next. Neither decision is exotic on its own, but together they turn modest hardware into a machine that reliably converts source material into finished video with nobody at an editing desk.
Start a project
Want something built like this?
Everything on this page was designed, built and hosted by one person. Tell me what you need and I reply with scope, cost and timeline, usually within two working days.