
Telephone triage runs on a short intake note and a quick decision. A nurse reads the caller’s symptoms, works through a published guideline, and decides whether the caller needs an ambulance, the emergency department, an appointment, or care at home. In the United States, much of that work follows guidelines written by Schmitt-Thompson Clinical Content (STCC), which are used by more than 400 health systems and health plans and about 10,000 physician practices.
Our team spent nine development iterations building an LLM-based medical triage system that automates the first pass of that decision against STCC’s guidelines. On a curated 50-scenario open benchmark, a directly prompted language model chose the correct disposition 42% of the time. The system we built, called ALLMTS, reached 98%. The study behind those numbers is on arXiv and was accepted at the GenAI4Health Workshop at NeurIPS 2026.
Most of the 56-point gap comes from one design choice. The model reads the guideline and answers its questions, and ordinary code turns those answers into the final disposition.
What a raw model gets wrong in triage
Asked directly, a general-purpose model escalates too often. On the full set of 566 expert-validated scenarios, the raw baseline over-triaged about 44% of cases, and up to 46% on the open subset. Under-triage is the failure that can hurt a patient, so a model that leans toward caution can look responsible. At those rates, though, it reproduces the problem a triage line exists to solve: routine cases flowing to the most expensive point of care.
The second problem is the audit trail. A prompted model returns a disposition and a plausible explanation, but it cannot point to the clinical rule that produced the answer. Clinical governance teams need that trace before they let any system near a patient.
Let the guideline decide and the model read
STCC’s clinical guidelines are decision trees. Each one lists yes or no Triage Assessment Questions (TAQs) in descending order of acuity, and each question is tied to a disposition such as “Go to ED Now” or “Home Care.” The first question answered “yes” sets the disposition. Nurses are trained to work this way.
The system follows the same order of work in four stages:
- Extract the patient’s age and gender, and use them to filter the guideline set.
- Structure the primary and secondary reasons for the visit.
- Retrieve the matching guideline.
- Answer every TAQ in that guideline as yes or no, in parallel, under a closed-world rule: if the intake note does not state it, the answer is no.
A short piece of deterministic code then takes the first positive TAQ in acuity order and returns its disposition. The model never writes the disposition itself. Each model call has a narrow reading task with an output that can be checked, and every disposition traces back to a named question in a named guideline.
How the guideline reaches the model matters
Early versions retrieved guidelines from pre-parsed PDF files. Parsing broke the multi-step TAQ logic into fragments, so the model often worked from an incomplete decision tree. We moved the guidelines into a relational database with structured records that keep each question, its acuity order and its disposition together. Each retrieval now returns the complete question sequence in the right order.
The lesson applies well beyond medicine. If the source content is logic, store it as logic. Turning a rulebook into prose and asking a model to rebuild the rules at runtime adds an error source that no prompt can fully remove.
Results against a nurse panel
The scenarios were drafted with a language model, then reviewed and refined by two STCC nurse editors and validated by a panel of expert triage nurses. On the 50 open scenarios, the same five-nurse panel reached a 94.4% correct disposition rate.
| Measure | Raw model | ALLMTS |
| Accuracy, 50-scenario open benchmark | 42% | 98% |
| Accuracy, 566 scenarios (14 dispositions) | 47% | 95% |
| Weighted F1, 566 scenarios | 48% | 96% |
| Over-triage, 566 scenarios | about 44% | about 2% |
| Under-triage, 566 scenarios | not reported | 4% |
| Open scenarios with the same acuity band on all ten runs | 28 of 50 | 47 of 50 |
The paper describes 98% as comparable to the nurse panel, and we agree with that wording. On 50 scenarios, a single case moves the score by two points, and nurses on a live line work under time pressure that a benchmark does not reproduce. The fairer reading is that the system matches expert performance on validated cases and behaves consistently enough to be audited.
Consistency deserves its own line in any evaluation. A model that gives the right answer on one run and a different answer on the next cannot be governed, whatever its average accuracy.
We ran every open scenario ten times. The baseline held the reference acuity band on all ten runs for 28 of 50 scenarios. ALLMTS held it for 47.
“Research, testing and validation of the content is at the core of what we do,” says Laurie O’Bryan, RN, Nurse Editor at STCC.
Where the system still fails
The remaining errors fall into two groups. In the first, the system selects the correct guideline but finds no positive TAQ, so it has no disposition to return. In the second, it selects the wrong guideline for a caller whose chief complaints overlap or are ambiguous. Both are retrieval and reading problems, and both are visible in the trace, which is the practical benefit of keeping the decision outside the model.
The evidence also has a limit. All 566 scenarios are synthetic. They were drafted with a language model, then refined by nurse editors and validated by nurses, not taken from live calls. The next step for this kind of system is a prospective evaluation on real intake notes, run alongside nurses before it is trusted with any part of the decision.
What transfers to other rule-bound decisions
Prior authorization, claims adjudication and compliance screening share the structure of triage: a published rulebook, a short case description and a decision that must be explained to an auditor. Four practices from this project apply directly.
First, when a rulebook exists, have the model answer the rulebook’s questions and compute the outcome in code. Second, store the rules as structured records so retrieval returns them whole. Third, measure against human experts on the same cases, and report over- and under-decisions separately, because a single accuracy figure hides the direction of the errors. Fourth, run each case repeatedly and treat run-to-run stability as a metric in its own right.
None of this requires a particular model or vendor. We compared several model generations during development, and newer ones handled complex multi-step scenarios better. The largest gain, however, came from narrowing the model’s job to reading, and leaving the decision to the rules the clinicians already trust.
Sources
- Janiszewski, Stopa, Lenkiewicz, Achtelik, Gonczarek, ALLMTS: Automated LLM-based Medical Triage System, arXiv, accepted at the GenAI4Health Workshop at NeurIPS 2026: https://arxiv.org/abs/2610.32469
- Vstorm, Reaching expert-level medical triage with guideline-executing LLMs (STCC case study): https://vstorm.co/case-study/reaching-expert-level-medical-triage-with-guideline-executing-llms/


