Most AI products that process video never actually analyse most of what they’re given.
That sounds strange, but it’s usually the result of a decision that feels obvious at the time. If you need AI vision capabilities – to soup up security camera analysis, for example, or content moderation – the obvious choice is to plug into a third-party API. It works, it’s quick to integrate and you can get it up and running in a day.
But these services don’t analyse every frame. Every frame needs compute to process, so many tools keep costs down by sampling the video at set intervals instead. Everything between those frames is invisible. And the model doing the sampling matters as much as the sampling itself. Quality diverges at both ends of the pipeline: what the vision model sees, and how well it describes what it saw.
I came to that conclusion after building and self-hosting my own vision infrastructure instead of relying on an API. It took longer, it was much harder, and there were plenty of problems to solve along the way. But it made me realise that the harder choice early on was the right one for the product.
What vision AI can and can’t see
Every vision AI model comes with a sampling rate: this is how often the model actually looks at the video.
Looking at every frame is expensive, so providers have to limit how much they process. Most video is recorded at 24-60 frames per second, but many lower-cost vision services only analyse around one frame per second to keep compute costs down.
The documentation for one of the most widely used vision APIs states that video is sampled at one frame per second by default, and warns that fast action sequences can lose detail at that rate. The problem is that fast action is precisely what makes a highlight. A goal, a headshot or a crash can happen entirely between two sampled frames, and the model simply never sees it.
Even a modest increase changes what gets captured: moving from one frame per second to 1.4 takes in around 40 per cent more of the frames. On rented infrastructure you pay for that at the provider’s prices, but on your own, it’s a tuning decision.
What’s doing the looking
Sampling is only half the picture. The other half is the strength of the model reading those frames.
One of the largest tools in this space has said publicly that it runs its visual description processing on the budget tier of a major model provider. Budget tiers exist because premium ones cost more, and the gap between them is measurable.
On Video-MME, the industry-standard benchmark for video understanding, deployments of open-weight models can now reach or exceed the scores of the premium tiers.This gap is mostly about comprehension, how well the model understands what it is looking at, rather than how much visual detail it can pick out. A weaker model tags the scene – a person, a desk, some text on screen. A stronger one follows the plot: who did what, how the situation changed, why the moment mattered. Both produce a written description, but one reads like a list of objects and the other like an account of what actually happened
In other words, a team running its own infrastructure can get premium-grade vision without paying premium API prices, while at least one of the biggest rented pipelines is running on the cheapest option.
The difference shows up in small details. Vision models process each frame at a set level of detail, and reading small on-screen text takes more of it than the standard settings provide. So the small print of a video, a player’s health bar, a score ticker, a caption, can go unread unless you pay for more processing.
In every product of this kind, the AI that picks the moment doesn’t watch the video, but reads a description of it. A description produced by a budget model sampling once a second comes out flat, more a list of moments than a story, and the clips built from it inherit that.
Why rented pipelines ration compute
Most vision APIs charge based on how much video they process, and video is far more expensive to analyse than text. A chatbot handles a few thousand tokens per conversation, while a vision model processes thousands of frames from a single video. As usage grows, that pricing forces every rented pipeline to ration compute somewhere.
As a result, many businesses turn to the budget tier of what’s available and sample as sparsely as they can get away with. Those are the two decisions that cap quality.
The pros and cons of self-hosting
Running your own models isn’t easy. You need GPUs, infrastructure and enough engineering to keep everything running reliably. A few years ago that would have been unrealistic for most startups.
Today it’s much more achievable. Open-weight models have improved quickly, and AI coding tools have made deploying and managing them much easier than they used to be.
To be clear, self-hosting does not mean training your own model. That remains the territory of heavily funded labs, for now. It means taking open-weight models and tuning the infrastructure and pipeline around them. The work is deployment engineering, not model training.
Owning the infrastructure lets you spend compute where quality shows. The cost saving is a welcome bonus rather than the reason to do it.
We still rent our reasoning model. At that stage of the pipeline, the open models that compete with the frontier are giants that need a datacentre, while a vision model that matches the premium tier now fits on a single large GPU. Self-hosting only makes sense where the extra compute shows up in the product.
Questions to ask before choosing a vision model
If video understanding is just one feature in your product, renting a model is probably the right decision.
But if this is the core you want to build your product around, it’s worth taking a closer look at the alternatives. Run the numbers at ten times and a hundred times your current usage. Ask providers how often their models sample video, and what it costs to raise that rate, and test them with footage that matters for your use case.
Using an API is the right decision for most prototypes, as long as you don’t treat it as permanent. As your product grows, ask yourself whether the convenience you gained at the start is limiting the strength of your core features. An API is convenient, but unless you’re paying for the premium tier, it’s handing your product a weaker picture of the video than a self-hosted open-weight model can now deliver. For us, the cost saving was a bonus. The better product was the point.
