In legacy industries like healthcare, the biggest opportunities live in “messy” problems where outcomes hinge on trust, empathy, and timing. In the age of Agentic AI, the temptation is to solve these with pure compute. But automation alone isn’t enough, and human-driven processes don’t scale.
If you’re building in this space, your job is to reconcile two competing truths:
- The importance of a human touch in earning trust and driving action
- The reliance of agentic systems on high-fidelity, “expert-in-the-loop” data for effectiveness
At Headway, we faced this tension while helping primary care physicians (PCPs) refer patients to behavioral healthcare. Here is how we moved from manual workflows to a scalable, agentic-ready model.
Human-in-the-Loop as a Data Engine
We often hear the Y Combinator mantra: “Do things that don’t scale.” In the nascent era of Agentic AI, this roughly translates to: “Get scrappy to capture high leverage, high value data that doesn’t yet exist.”
When we launched our first referral product—a simple “easy refer” button—it looked like a success on paper. But within a month, 50% of doctors churned. They didn’t want a button; they wanted a handoff. They missed the social workers they used to know by name.
We responded by virtually embedding Licensed Clinical Social Workers (LCSWs) into our referral flow. It was unscalable, but it was also a goldmine for agentic learning. By having LCSWs handle the “messy” middle:
- Capturing Intent: We recorded why a social worker chose one therapist over another for a specific patient.
- Decision Logic: Every manual intervention became a labeled data point for how to navigate complex insurance or clinical barriers.
The Lesson: Don’t just use humans to bridge a gap; use them to build a decision-log that will eventually serve as the “world model” for your AI agents.
Identifying High-Leverage Decision Bottlenecks
As we moved toward conversion, we realized that trust is asymmetric. One “high-leverage” moment can power the entire funnel. We discovered that when a doctor said, “John is going to reach out to you; he’s a therapist I trust,” response rates doubled, even if the subsequent text was automated.
In an agentic system, this is where Contextual Injection happens. An agent shouldn’t just “act”; it needs to know whose authority it is operating under. By identifying these leverage points, you can design agents that don’t just “execute tasks,” but “inherit trust.”
Implementing Agentic Guardrails
As we began layering in automation (outreach texts, follow-ups, scheduling, etc.) we hit the “Uncanny Valley.” One patient sharply asked: “You’re not even a real person, are you?”
This taught us the importance of Guardrails and Transparency. In healthcare, an agent hallucinating a tone or a clinical fact isn’t just a bug; it’s a liability. We implemented three layers of guardrails:
- Semantic Boundaries: Ensuring the agent never strays into diagnostic territory.
- Deterministic Fallbacks: If the AI detects high emotional distress or a “logic loop,” it immediately pings a human supervisor.
- Transparency by Design: We stopped trying to “mask” the AI. We told patients: “Technology helps us move faster, but John (a real human) is monitoring this to ensure you get care.”
Surprisingly, patients didn’t mind us incorporating AI into our experiences; what they minded was any attempt to hide it. When the guardrails were visible, trust actually increased.
The Flywheel of Decision Data
Today, Headway’s referral product operates nationwide. But the real “moat” isn’t just the network; it’s the rich decision-making data captured from years of human-in-the-loop processes.
Building in vertical AI (especially in high-sensitivity fields) requires a shift in mindset:
- Phase 1: Human-in-the-loop for trust and data capture.
- Phase 2: Agent-in-the-loop for scale and efficiency.
- Phase 3: Human-on-the-loop for oversight and edge-case handling.
If you are building healthcare AI products, don’t rush to remove the human. Instead, celebrate the outsized leverage of human processes to generate rich training data early on, and deploy human expertise to shape the nature of the decision making that will eventually be the domain of the agents you build.
