For all the hype surrounding generative AI, the infrastructure powering it has a glaring bias: it was built for text. While Large Language Models (LLMs) have benefited from years of hyper-focused optimization, making them faster and cheaper to run, image and video generation (diffusion models) have largely been bolted onto existing systems.
The result? Generating AI video is still stubbornly slow and prohibitively expensive.
Last week, an AI infrastructure startup called MachGen emerged from stealth to change that. Founded by Kismat Singh and Manoj Krishnan, MachGen is building a high-performance inference and fine-tuning stack designed specifically for diffusion models from the ground up.
Their pitch is simple but ambitious: push diffusion latency and compute costs down by an order of magnitude to unlock interactive, real-time video generation.Here’s what they laid out in a blog post:
Founder Pedigree from NVIDIA, Google, Intel, and Meta
MachGen’s team has spent their careers making AI models run fast at the world’s biggest tech companies.
The founders’ collective resume includes co-building the TRT high-performance inference teams at Nvidia, leading Datacenter AI software at Intel, driving the vLLM inference effort at Google, and building the large-model training infrastructure behind PyTorch at Meta.
Having watched the LLM inference stack mature from clunky research prototypes into highly optimized, production-grade systems serving trillions of tokens, Singh and Krishnan saw a familiar pattern emerging with diffusion.
“We didn’t start MachGen to shave a few percent off existing bills,” the founders noted in their launch announcement. “When a video that takes several minutes can be generated in seconds, and at a fraction of the cost, you don’t just get a cheaper version of today’s products, you unlock interactive generation, real-time personalization, customized advertising at scale.”
Why the LLM Playbook Fails for Video
The core technical problem MachGen is solving stems from the fact that diffusion models are fundamentally different beasts than LLMs.
The optimizations that made LLMs cheap (like paged KV caches and continuous batching) are built around an autoregressive pattern, meaning they predict one token at a time. Diffusion models, however, are non-autoregressive and compute-bound throughout their entire “denoising” loop.
Because the industry has over-indexed on LLMs, existing serving frameworks treat diffusion as an afterthought. To fix this, MachGen re-engineered the stack across four key pillars:
- Attention: MachGen uses lower-precision recipes and sparse attention techniques to slash the compute budget, which typically dominates video generation.
- Caching: Since standard LLM KV caching doesn’t transfer, MachGen built proprietary caching recipes that exploit the spatial and temporal redundancy unique to diffusion models.
- Kernels: While open-source GPU kernels for LLMs are abundant, diffusion kernels are scarce. MachGen developed highly optimized kernels specifically for diffusion architectures.
- Parallelism: To minimize communication overhead between multiple GPUs, they implemented tailored kernels that overlap communications with data-independent computations.
The Numbers
MachGen claims these foundational changes yield massive performance leaps without sacrificing quality or resorting to distilled models.
According to the company’s internal benchmarks, they are achieving a 4x to 6x reduction in latency and a 2x to 3x reduction in cost compared to standard inference deployments.
- Image Generation: HiDream runs in 1 second (vs. 6 seconds baseline), and Flux runs in 1.5 seconds (vs. 6.2 seconds).
- Video Generation: Wan 2.2 runs in 16.5 seconds (vs. 98 seconds), and LTX 2.3 completes in 10.7 seconds (vs. 67 seconds).
MachGen is bringing this optimized stack to market through several avenues, catering to developers of all sizes. Offerings include direct Model APIs for instant access, MachGen Cloud (a fully managed service), a Self-Hosted option for enterprise VPCs, and a Serverless platform where customers bring their own models while MachGen handles GPU provisioning and autoscaling.
The North Star: World Models
While democratizing video generation is MachGen’s immediate go-to-market strategy, the company is playing a longer game.
Their ultimate target is world models, AI systems that can understand, simulate, and reason about the physical world. This is the next frontier of AI, heavily backed by billions in funding and spearheaded by labs linked to Fei-Fei Li, Yann LeCun, and xAI.
Crucially, today’s world models are computationally similar to diffusion models.
“There is essentially no production-grade inference solution for world models today. We intend to be the first to build it,” the founders stated. “Diffusion is where we earn our keep now; world models are our North Star. When they arrive, performance won’t be a nice-to-have. It’ll be the thing that determines whether they’re usable at all.”
