In spring 2025, the generative AI industry entered a new phase of development. Following the rapid adoption of text-based AI assistants, companies began deploying video agents capable of holding natural conversations in real time. One of the companies leading this transition is Tavus, a Y Combinator alum backed by investors including Sequoia Capital and Scale Venture Partners. The company created the Conversational Video Interface (CVI), which it describes as the world’s fastest interface of its kind: a real-time multimodal framework that lets an AI agent see, hear, and respond naturally, face to face. In March 2025, Tavus introduced Phoenix-3, Raven-0, and Sparrow-0 — a family of models powering AI agents that perceive visual context, read nonverbal cues, and keep up with the rhythm of a live conversation at sub-second latency.
We spoke with Senior Software Engineer Nikita Puzyrenko about the technology behind real-time conversational AI, the engineering challenges of building emotionally intelligent video agents, and why Developer Experience has become just as important as the AI models themselves.
Over the past year, the industry has been actively discussing the shift from text-based AI interfaces to video agents. Why do you believe that shift is the next stage in the evolution of human-AI interaction rather than another interface fashion?
For decades, people have had to adapt to computers: we learned commands, then menus and interfaces, and eventually prompts. Conversational Video Interfaces, the term we use at Tavus for this class of system, begin to reverse that relationship. Instead of humans learning the machine’s language, the machine finally starts to learn ours.
Human communication is much richer than an exchange of words. We respond to facial expressions, tone, timing, eye contact, all the small signals that tell us the other participant is actually engaged. Text strips out almost all of those signals, and voice restores only some of them. Face-to-face interaction creates a sense of presence and engagement that neither can match.
That has practical value far beyond novelty. Think of interviewing, education, coaching, sales, customer support. The role the AI plays is different in each case, but the underlying need is the same: people communicate more naturally when they can see, hear, and respond to the other participant.
A skeptic might say a video agent is just a chatbot with a face, a cosmetic layer on top of the same language model.
I understand the skepticism, but it misses what actually changes. CVI is not a visual layer painted over a chatbot; it is a step toward a more human form of computing. When a system can perceive how you say something, not only what you say, and can respond at the right moment with the right expression, interacting with it stops feeling like operating software and starts feeling like communicating. That difference is not cosmetic; it changes what people are willing to trust the technology with.
Tavus positions its CVI as one of the fastest real-time human-AI interaction systems available today. What were the most critical engineering challenges that had to be solved to achieve truly natural conversations with sub-second latency?
The hardest part is that users experience latency and naturalness end to end, not component by component.
A real-time conversation requires the system to receive the user’s speech, understand whether they have finished speaking, generate a relevant response, synthesize audio, render the replica, and deliver all of it through a real-time media connection. Every stage contributes to the final experience, and the user only ever perceives the sum.
So there is no single bottleneck you could optimize and call it done?
Exactly. Optimizing one component is never enough. A fast response can still feel unnatural if the system interrupts the user, replies at the wrong moment, produces an abrupt visual transition, or becomes unstable when network conditions change. You can shave milliseconds off the model’s response and still lose the entire effect to one poorly timed turn.
So the real objective was never to reduce a latency metric. It was to preserve the rhythm of conversation: knowing when to listen, when to respond, and how to transition naturally between those states. That required close coordination between conversational timing, rendering, streaming, and the product experience built around them.
At the same time, modern AI video agents must do far more than generate speech and video. They need to understand conversational context, recognize human emotions, and respond naturally. What technological breakthroughs made these capabilities possible?
The major breakthrough was not any single model — it was bringing several specialized capabilities together inside one real-time system.
Phoenix-3 enabled full-face rendering. Earlier systems focused mostly on lip movement, which is exactly why they looked artificial: a real face speaks with the eyebrows, the cheeks, dozens of micro-movements, not just the mouth. Raven-0 added real-time perception: the ability to read visual context and nonverbal cues, so the agent understands not only what is said but how it is said. And Sparrow-0 addressed conversational timing — interpreting pauses, pacing, and the subtle signals that indicate whether a person has actually finished speaking.
The language model supplies reasoning, context, and personality, but intelligence alone was never enough. Natural communication requires perception, timing, and rendering to work as one system: understand what someone said, consider how it was communicated, determine when a response is appropriate, and express that response visually.
Together, these capabilities produced something qualitatively different. The result is more than a face delivering generated speech; it is an interface capable of participating in the full rhythm of a conversation.
Which parts of this system did you work on yourself — and what turned out to be the hardest problem there?
One of the most interesting challenges I worked on was the replica’s active-listening behavior. Natural conversation depends just as much on listening as it does on speaking, yet that is surprisingly easy to overlook.
Our replicas already looked natural while speaking, but the listening state was less convincing. The transition between the two could produce a visible jump, and while the other participant was talking, it sometimes looked as if the replica was no longer fully present in the conversation.
Solving it meant working across several parts of the system at once: the replica-training experience, the processing layer, and the CVI rendering pipeline. I updated the training flow to capture a neutral listening segment alongside the speaking footage, then extended the processing and rendering flow to use that data during a live conversation. This allowed the replica to maintain a natural listening state while the other person was speaking and transition more smoothly between listening and responding.
The result was smoother transitions between speaking and listening and a more natural presence throughout the conversation. It mattered because a convincing conversation is not only about how an AI speaks. It is also about how it listens.
Tavus is also known among developers for its self-serve platform. The intuitive assumption is that a company builds the technology first and then wraps a developer platform around it. Why invest serious engineering effort there — and how were you involved?
Let me correct the premise slightly, because the sequence matters. We did not build the platform after the technology was already in place. The platform was built alongside the technology, in parallel, and it was created specifically to showcase those capabilities and make them accessible to everyone. That parallel development continues to this day.
When I started working at Tavus, the existing platform was oriented toward video generation, and real-time conversational experiences required a fundamentally different developer journey. Together with one other engineer, I led the rebuild of the new Portal from the ground up: I owned the frontend architecture, took an active part in product and design decisions, developed the component system, implemented the backend changes the frontend needed, and collaborated with the backend team on API design.
The difficult part was not presenting a collection of features. We had to translate a complex technology into an experience developers could grasp quickly: a developer should be able to arrive with an idea, bring a replica and persona to life, try a conversation, and move naturally from that first experiment to integrating Tavus into their own product. And because we were building inside a fast-moving startup, the architecture had to survive constant change, including new capabilities, modified workflows, experiments, without rebuilding the Portal each time.
After launch, product analytics and user feedback showed us how developers actually used the platform, where they hit errors, and which parts of the experience needed work. That feedback loop let us iterate quickly and made the path from discovering Tavus to testing a real use case much shorter.
It also created a clear route to enterprise adoption: a team could experience the technology, validate a concrete use case, and begin a larger integration with a working result in hand rather than an abstract idea. The Developer Portal was never just a dashboard around an API. It is the primary place where developers experience Tavus technology, understand its value, and turn it into a product.
Your examples repository and React component library are public on GitHub. What role do they play in helping developers move from documentation to a working CVI integration?
Documentation is the source of truth for an API’s capabilities, but working code helps developers understand how those capabilities fit together inside a real application.
I contributed to the public `tavus-examples` repository, including practical examples of building CVI experiences with our React component library. Instead of asking developers to understand every API concept and assemble the entire experience from scratch, the examples gave them a working starting point they could run, inspect, and adapt to their own product.
The React component library addressed a related problem. Most teams should not need to build the common parts of a conversational interface from scratch, such as setting up the experience, managing the conversation lifecycle, and rendering the call UI. The library provided reusable building blocks that helped developers get a solid experience running quickly while preserving the flexibility to customize it.
The documentation, examples, and component library each served a different purpose. Documentation described the platform, examples demonstrated proven integration patterns, and the component library accelerated implementation. Together, they shortened the path from exploring CVI to shipping it inside a real product.
Can you quantify the effect?
The clearest measure is time to integration. Work that used to take teams weeks, including wiring up real-time video, handling conversation events, and building the interface around them, now takes minutes with ready components and working examples. But honestly, the more important effect is on uncertainty, not just effort. Examples help developers avoid common integration mistakes and reach a working experience faster. The goal was never only to help them write less code. It was to help them evaluate their idea sooner.
Today, many experts are talking about the transition from AI assistants to autonomous AI agents. In your view, how should human-AI interfaces evolve when systems move beyond responding to prompts and begin making decisions, planning actions, and pursuing goals independently?
The autonomy of an agent is largely determined by the system designed around it, not only by the model inside it.
An agent that can only answer a question will return text. The same agent connected to tools, workflows, clear instructions, and an observable execution environment can complete a much larger task. A development agent, for example, could take an issue from a project tracker, inspect the relevant codebase, implement a change, run checks, create a pull request, and come back with screenshots, video, or metrics demonstrating the completed work. The same underlying intelligence will appear far less autonomous if all it is given is a text box and no ability to act.
As agents become more capable, interfaces must evolve from displaying answers to communicating progress, decisions, uncertainty, and results. People need to understand what an agent is doing, which tools it is using, and whether the outcome meets the original objective. And greater autonomy should come with explicit permissions, observable progress, clear success criteria, and checkpoints for actions whose consequences are difficult to reverse.
So the limit is not only the intelligence of the model. It is also the quality and completeness of the system designed around it.
As AI agents become more autonomous, the conversation is increasingly shifting from replacing humans to enabling effective Human-AI Collaboration. How do you see the future of Human-AI Collaboration, and what role can Conversational Video Interfaces play in shaping it?
I do not think AI will settle into one fixed role. It may act as an assistant, a coach, a researcher, a coworker, or a combination of these, depending on the task and the person using it.
The immediate value is faster iteration. A person can delegate research, analysis, or implementation, let several tasks run in parallel, and review the results instead of performing every intermediate step manually. That makes it faster to learn a new subject, explore different approaches, and move from an idea to a working result.
Human review will remain important, but its nature will change. Instead of producing every intermediate artifact, people will increasingly define objectives, evaluate outcomes, apply judgment, and decide what should happen next.
And this is exactly where CVI comes in. As agents take on longer and more complex tasks, trust and communication become the scarce resources. Face-to-face interaction gives an agent a way to explain its work, communicate uncertainty, respond to visual and conversational cues, and adapt the conversation in real time.
The future is not simply about machines completing more tasks. It is about creating systems that help people learn faster, explore more possibilities, and extend what they are capable of doing.
