A refinery switched to a new crude blend. Within days, corrosion in the overhead lines accelerated past anything the operating team had seen. Yet the plant’s process model, well-calibrated to years of historical conditions, saw nothing wrong. Every parameter sat inside specification, and the optimizer kept running.
The model couldn’t see how chloride distribution, acid formation, and neutralizer chemistry had shifted under the new feed. It was pattern-matching against a past that no longer existed. A purely data-driven AI model trained on the same history would have missed it the same way, for the same reason.
The failure wasn’t a bad model. It was a model running with nothing underneath it that understood the chemistry.
That failure matters now because AI is being layered into heavy industry at speed — steel mills, power grids, pharmaceutical plants, chemical refineries. These industries answer to physics. In the rush to embed AI into the physical layer, we are building a new kind of industrial blind spot: the gap between statistical confidence and physical reality.
Why This Failure Mode Is New
The person who built the original process model knew where it ran out. They knew which crude blends behaved strangely and which conditions the correlations never covered. That knowing wasn’t captured in the data. It was the residue of having done the work by hand: build a model the slow way and you learn its edges whether you mean to or not.
The accuracy gap between model and reality was real, but it was static, and a human stood inside it.
Take the human out of the loop and the gap doesn’t disappear. It changes character. A bounded error, caught and corrected one decision at a time by an operator, becomes a propagating error. It repeats across every recommendation, every minute, every unit the system touches. The same few percent that was a rounding error when an engineer read it now compounds at machine speed. AI doesn’t fail quietly. It fails with confidence, at scale, in well-formatted outputs.
Watch a room of veteran process engineers react to an AI claim and you will hear the same instinct over and over, dressed as different objections: Where was this validated? What conditions has it seen? What happens at the edges?
This is not technophobia. These are the questions of people who have been wrong before and remember what it cost.
The Disappearing Safety Buffer
For decades, that gap stayed contained, not because anyone solved it, but because the chemistry lived in a silo. If you wanted the science, you went to specialized software, learned the thermodynamics deeply enough to use it well, and ran it yourself. The science was powerful, but the delivery was narrow. The grounding was implicit. It rode along inside the person at the keyboard.
The science, in other words, was never the bottleneck. The delivery was.
What’s changed is that the delivery stopped waiting. Engineers want the chemistry inside the simulator they run all day, the monitoring system wired to their sensors, the platform their whole site runs on. Increasingly the request doesn’t even come from them. Third-party software companies now want first-principles chemistry living inside their products, because it’s the one part they’ve decided not to approximate.
That spread is a good thing. It’s also where the danger moves, because implicit knowledge doesn’t commute.
Push the same chemistry into a dozen surfaces, accessed through plain language instead of an interface someone trained on for years, and the specialist who used to stand inside the gap isn’t there anymore. The refinery had one. The next thousand instances won’t. What kept the error bounded was the person who knew what the model didn’t.
What “Grounded” Actually Means
A statistical AI model needs to have seen conditions like yours before to be accurate. A first-principles model does not.
Give a first-principles model the actual composition, temperature, and pressure of your system, and it solves for what will happen from the underlying science of how ions, molecules, and phases behave.
Physics alone isn’t the whole answer. What makes these models trustworthy in the field is decades of validation against experimental data across thousands of chemical species. The physics provides the framework; the accumulated reconciliations provide the calibration. Together they predict behavior in conditions no one has operated before. That is a different thing from interpolating inside a training set.
To be honest about the boundary: grounding matters most where systems move. In stable processes with consistent feedstock and good instrumentation, statistical models work well and physical validation may add little. The problem starts at the transitions: new feed, upset, unfamiliar chemistry. That is where AI is being asked to act autonomously.
The Part That Can’t Be Retrofitted
Here is the thing most discussions of AI and trust miss. The chain of custody on an answer can’t be back-filled. It’s built at the moment the answer is earned: the prediction that missed, the engineer who went out to the unit to find out why, the correction folded back in.
That window doesn’t reopen. You can refactor a bad software system whenever you like, but you can’t return to the day an operational decision was made and recover the human reasoning behind it.
The value that’s hard to copy was never the software’s final answer. It was the institutional memory of being in the room when the answer was wrong.
That is the difference between confidence and grounding. A model can be trained to produce a confident number. It can’t be trained backward into having been checked by an engineer who was there, against a plant that no longer runs that way. The physics can be assembled from the literature. The record of being wrong in front of a running plant can’t.
Where This Leaves Industrial Operators
The instinct to put AI in the operational layer is right. The speed and scale are real, and the leaders furthest along are doing work worth respecting. The open question for a CEO isn’t whether to use AI. It’s what sits underneath it when conditions move outside what it has seen.
If the same chemistry is going to live in a simulator, a dashboard, and an external platform all at once, the thing that has to travel with it isn’t just the final recommendation. It’s the answer’s provenance — where it was born, what it was validated against, where it stops being reliable. The same chemistry has to give the same answer in every surface, and carry the proof of its own grounding into each one. That’s the part that’s hard to build, and it’s the part worth building.
The most sophisticated operators already grasp this. The ones running first-principles chemistry live against sensor data, flagging deviations automatically across an enterprise, have built the most automated version of this that exists in the field. And they still put a human in front of the action.
The system produces the reading. A person who knows what the reading means decides what to do with it. That isn’t timidity. It’s the bounded gap, preserved on purpose.
Stop asking how accurate the model is. Start asking a harder pair of questions: where does this model run out, and what catches it when it does?
An AI optimizer can propose a setpoint in milliseconds. Whether that setpoint respects the chemistry is a separate question, and it needs an answer that doesn’t come from the same statistical machinery that made the proposal.
Frameworks like the NIST AI Risk Management Framework point in this direction, making “valid and reliable” the base of trustworthy AI rather than a feature added late. NIST is now extending the framework toward AI in critical infrastructure. But that kind of validity means performing as intended under expected conditions, a bar the refinery’s model cleared right up until the feed changed. Physical grounding demands more. Before a digital system acts on the physical world, something structural has to be able to say whether the action is physically possible. That check is not a layer on top. It is the foundation underneath.
The refinery story ended well, as it happens. The divergence was caught offline, against the new feed composition, before the plant had time to fail. Somewhere in that loop, something knew what the chemistrywould do under conditions it had never been handed before. And a person was still there to act on it.
Andy Rafal is the CEO of OLI, which has built and validated first-principles electrolyte chemistry models for industry since 1971.
