AI & Technology

The Trust Problem

By Jon Sargent, founder of On Spot Solutions LLC and the creator of ATLAS

Capturing knowledge is the easy part. The hard part is trust — a confident wrong answer at midnight is worse than no answer at all. 

There is a standard against which any knowledge-preservation system in the trades should be measured. Call it the midnight standard: a technician is standing in front of a piece of equipment that is not working, at a site they do not know well, at an hour when no one senior is reachable. They consult the system. The system gives them an answer. Is that answer reliable enough to stake their work on? Reliable enough that acting on it will not make the situation worse, damage equipment, create a safety hazard, or generate liability for everyone in the chain? 

If the answer is yes, the system earns its place. If the answer is sometimes, or probably, or for most things, the gap between that and yes is where real damage happens. The trades are not a context where a plausible-sounding wrong answer is merely inconvenient. 

The hallucination problem is real 

Language models hallucinate. This is not a fringe failure mode or an early-generation flaw being rapidly engineered away — it is a structural property of how these systems work. A model generates the most statistically probable continuation of the input it has received. When that input touches something the model has seen a lot of, the output is usually accurate. When it touches something uncommon, edge-case, or dependent on very specific technical context, the model can produce confident, fluent, completely wrong answers. 

A Stanford study on legal applications of AI found that even well-performing models cited non-existent cases in legal briefs — not occasionally, but regularly. The same class of error is possible in technical service contexts: a system that sounds authoritative recommending a diagnostic path that is wrong for the specific hardware revision, or a safety interlock sequence that applies to a different product generation. The technician who does not know enough to catch the error is precisely the technician the system is supposed to be helping. 

Verification is the discipline that makes it usable 

The answer is not to wait for perfect systems. It is to build verification into the architecture of how knowledge systems are used. This means sourcing answers to actual prior cases rather than generating them from patterns. It means flagging confidence levels explicitly so a technician knows when they are on solid ground and when they should seek confirmation. It means building escalation paths into the workflow so that an answer the system is uncertain about routes to a human rather than being presented as definitive. 

This is not a novel idea in high-stakes domains. Aviation has built layers of verification into every consequential decision. Medicine has clinical decision support tools that flag rather than replace physician judgment. The trades can adopt the same orientation: AI as a first resource that surfaces the relevant prior experience, with verification built in for anything consequential enough to get wrong. 

The liability question 

Who is responsible when an AI-guided diagnosis goes wrong? This is not a hypothetical. As knowledge-assistance tools move from demo to deployment, the liability question will move with them. The organizations building these systems are aware of it, and the answer being developed in practice is not to shift liability to the AI but to document the human decision-making layer clearly enough that responsibility remains with the person who acted, informed by a tool that surfaced relevant information. 

That framing is workable — and it is actually consistent with how expertise transfer has always functioned. The senior technician who trained the junior one is not liable when the junior makes an error. The organization that built a training program with a known gap is in a different position. Knowledge systems that are honest about their limitations, that flag uncertainty, and that keep human judgment in the decision loop are defensible in a way that systems presenting confident answers to every query are not. 

What comes next in this series 

Trust in a system is partly technical and partly human. Part 6 addresses the human side — the dignity and ownership questions that arise when you ask a veteran to transfer a lifetime’s worth of expertise into infrastructure someone else will profit from. Part 6: Who Owns What a Lifetime of Work Taught You? 

Related Articles

Back to top button