U.S. property insurers that use AI to handle claims tend to describe the safeguard in the same words: a person reviews the recommendation before anything happens to the customer. That sentence is easy to write into a policy and much harder to prove after the fact. The proof has to live in the claim file, and in many files it is thin. One worked example shows where it breaks.
A hypothetical property claim that goes wrong in three places
The case below is invented for illustration. It does not describe any real insurer, vendor, or policyholder.
A homeowner reports water damage after a supply line under the kitchen sink fails. A claims model reads the first notice of loss, the photos and the base policy form, then recommends a denial because the damage looks like it built up over weeks rather than happening suddenly. The policy, however, carries an endorsement that adds limited cover for this kind of leak. That endorsement sits in a separate document the model was never given.
An adjuster opens the file the next morning. The screen shows a score, a one-line reason and the photos, but not the list of endorsements attached to the policy. Under the team’s guidelines, the adjuster can override a model recommendation only with a supervisor’s sign-off, and the supervisor queue is running two days behind.
Meanwhile the denial letter has already gone out. The workflow generated it the evening before, because letter generation was triggered by the model’s output rather than by the end of the human review.
The adjuster eventually finds the endorsement, and the claim is reopened and paid. The final outcome is right, but look at what the file can show. It records a denial and then a reversal. It says nothing about why the model missed the endorsement, what the reviewer could see, or why the customer heard “no” before anyone had looked.
Why “a human approved it” is not evidence on its own
US insurance regulators have said that human involvement matters, but they treat it as one input to a risk judgment rather than a safe harbour. The National Association of Insurance Commissioners (NAIC), which coordinates standards among state insurance regulators, adopted a model bulletin on insurers’ use of AI in December 2023. It asks each insurer to scale its controls to its own assessment of risk, and one of the factors it lists is “the extent to which humans are involved in the final decision-making process.” That factor only helps an insurer that can show what the involvement actually consisted of.
The bulletin also says an insurer’s AI program should cover systems acquired from third parties, not just those built in-house. Its stated aim is that decisions made or supported by a vendor’s system meet “the legal standards imposed on the Insurer itself.” Buying the denial model from someone else does not move the record-keeping job anywhere else.
Six records worth keeping for every AI-assisted claim decision
The list below is good practice, not a legal checklist. No single regulation requires these six items in this form. They are the minimum a claims team would need to reconstruct the hypothetical case above, written so that someone outside the claims department could follow them.
- The model’s recommendation and its basis: the output, the model or ruleset version, and the inputs it used, including any policy documents it did not receive.
- What the reviewer saw: a record of the information on screen when the reviewer decided, so anyone can later tell whether the missed endorsement was in view.
- The reviewer’s authority: whether that person could override the recommendation alone, needed someone else’s sign-off, or could only escalate.
- The final decision and the reason for it: the outcome, who made it, and the policy provision, fact or guideline that explains any departure from the model.
- The notice the customer received: the letter version, the date it was sent, the reason it gave, and the route it offered for disputing the decision.
- The order of events: timestamps for the recommendation, the review and the notice, so it is possible to check what happened first.
In the hypothetical case, record six would have exposed the main problem at a glance. A letter that goes out before the review finishes is a risk in this particular workflow design. Whether it breaks the law in a given state depends on that state’s claims-handling rules and the facts of the claim. It is still the failure a customer notices first, and it is cheap to prevent by tying the notice to the end of review rather than to the model’s output.
The NAIC bulletin does say an insurer’s AI program should include ways of telling affected consumers that AI systems are in use. That is a different question from when a decision letter is released. Both deserve an owner.
A simple way to test the records is to pick three recent claims where the model recommended one outcome and a person chose another. Try to rebuild each one from the six records alone. Where the trail breaks, the review is no longer something the insurer can demonstrate.
Check the rules in each state before rolling out further
A claims AI workflow that works in one state is often switched on in others, and that is the moment to check the records again. The NAIC bulletin is a template, not a law, and it applies in a state only if that state’s insurance regulator adopts it. More than half the states have done so, usually through a bulletin from the insurance department. A few states, including California, Colorado, New York and Texas, run their own insurance-specific rules or guidance instead.
The NAIC publishes an adoption map with a reference list of each state’s instrument. The map notes that it does not judge whether a state’s action contains every element of the model. For checking the current text rather than a summary, a state-by-state tracker of NAIC AI bulletin adoption links each state’s instrument to the regulator’s official source.
The state-specific rules differ in scope, and scope is the first thing a claims team should check. Texas’s 2020 bulletin reminds insurers that they are responsible for the accuracy of data used in rating, underwriting and claims handling, even when a third party supplies it. California’s 2022 bulletin addresses claims practices alongside marketing, rating and underwriting, and reminds insurers of their obligation to pay claims in a way that treats similarly situated people alike. New York’s 2024 circular letter, by contrast, is directed at AI systems and external data in underwriting and pricing.
None of these documents prescribes the six records above. What they show is that the same workflow can sit under different expectations in different states. Before expanding, read the instrument that applies in each new state, confirm which of your claims tools it reaches, and make sure the records you keep would let you answer its questions.
The test that matters
Human review is a control only if it can change an outcome and the file can prove it. In the hypothetical claim, the reviewer lacked the endorsement, needed permission to disagree, and was overtaken by an automated letter. None of that is unusual, and none of it shows up in a sentence that says a person approved the decision. The six records are how an insurer finds out, before a customer or a regulator does, whether its human review is real.
Sources
- NAIC, Model Bulletin: Use of Artificial Intelligence Systems by Insurers, adopted December 4, 2023: https://content.naic.org/sites/default/files/cmte-h-big-data-artificial-intelligence-wg-ai-model-bulletin.pdf.pdf
- NAIC, Implementation of NAIC Model Bulletin (adoption map and reference list): https://content.naic.org/sites/default/files/legal-adoption-map-ai-model-bulletin.pdf
- Texas Department of Insurance, Commissioner’s Bulletin B-0036-20, September 30, 2020: https://www.tdi.texas.gov/bulletins/2020/B-0036-20.html
- California Department of Insurance, Bulletin 2022-5, June 30, 2022: https://www.insurance.ca.gov/0250-insurers/0300-insurers/0200-bulletins/bulletin-notices-commiss-opinion/upload/BULLETIN-2022-5-Allegations-of-Racial-Bias-and-Unfair-Discrimination-in-Marketing-Rating-Underwriting-and-Claims-Practices-by-the-Insurance-Industry.pdf
- New York Department of Financial Services, Insurance Circular Letter No. 7, July 11, 2024: https://www.dfs.ny.gov/industry-guidance/circular-letters/cl2024-07
About the author
Simon Li is the editor of InsureAI Wire, which tracks how US insurance regulators are responding to AI.
