AI adoption has outpaced the frameworks meant to govern it, and the fallout now lands on compliance and risk teams as much as on IT. Governance, Risk and Compliance (GRC) reviews are stalling AI tools in procurement, CFOs are questioning token spend that isn’t producing measurable gains, and most organizations can’t say with confidence what the AI running inside their environment actually does.
Frameworks are almost always a lagging indicator, but this time they showed up early. NIST published its AI Risk Management Framework fifty-seven days after ChatGPT launched. ISO/IEC 42001 followed within thirteen months, and the EU AI Act entered into force in August 2024. Cloud waited four years for its first cross-industry control framework, and open source waited twenty-three years for SBOM minimum elements.
The gap opened anyway. Ernst & Young (EY) surveyed 202 senior AI decision-makers at US public companies with more than $1 billion in revenue. 98% have formal AI governance policies, and 47% have skipped those processes for urgent deployments. 91% use agentic AI in pilot or production, and 49% haven’t updated their governance framework for it.
I recently joined Chris Steffen, VP of Research at Enterprise Management Associates (EMA), for a fireside chat webinar to talk through where that gap is showing up and what GRC teams can do about it.
The frameworks govern the organization, and the model goes untested
ISO/IEC 42001, the NIST AI RMF, the EU AI Act, and SOC 2 each describe what a company must do to enforce governance on an AI system. None makes an assessable claim about what the system does.
Part of the reason is timing. These frameworks were shaped around a more classic idea of data science: a model built for one scenario, trained on known data, and tested before release. Agentic systems act with a degree of autonomy and pursue their goal however they can. LLMs are probabilistic. Cybersecurity is deterministic, built on saying this system can do that and nothing else. Those two worlds are now colliding in production.
Chris made a point about hallucinations that I keep coming back to. Models are rewarded for producing an answer. Not even the correct answer, just any answer.
“The very first answer that every single AI solution should always be giving you is three very simple words: I don’t know,” he said. “As a manager, as an executive, if the response is I don’t know, that’s better than an answer that is a false answer.”
Consider a hypothetical insurer that certifies a claims triage model under ISO/IEC 42001. It documents a policy, assigns roles, runs an impact assessment, and defines criteria for verifying and validating the model. A quarterly test then shows the model wrongly denies 40% of valid claims from one group of policyholders. The auditor checks whether the criteria were defined, applied on schedule, and retained. All three check out, so the certificate issues. The standard lets the insurer set its own criteria and gives the auditor no basis to call them too weak.
SOC 2 has the same boundary. It tests whether controls are operated, and a model inference is not a control operation. Processing Integrity tests conformance to a specification that management writes, so it can confirm a summary was produced and delivered without confirming the summary is true.
NIST acknowledges the underlying problem in its own framework, citing the lack of consensus on robust and verifiable methods for measuring AI risk. You can’t write an assessable requirement against a measurement that doesn’t exist yet, so the frameworks govern process and leave system behavior to the organization.
We’ve solved this before. In human capital management, the Equal Employment Opportunity Commission set out mathematical analyses for how a hiring model performs: accuracy, overfitting, and adverse impact on protected classes.
When I’ve built those models, it didn’t matter whether they were random forests or neural networks. We tested every one of them against those measures before release. That’s the next step for AI governance: does the model perform the way it should, and does it run afoul of the law? Both are testable, and the companies building these systems are responsible for testing them.
Chris and I agree on where that responsibility lands. “You don’t get to ‘AI’ yourself out of liability,” he said.
Why AI tools get blocked in procurement
Here’s what GRC teams run into. A vendor presents an ISO/IEC 42001 certificate and a clean SOC 2 Type II. The certified organization wrote its own scope and chose which controls to exclude, and those documents usually stay confidential. A certificate covering the whole product looks identical to one covering a single summarization feature.
The supply chain makes it harder. When a product runs on a frontier model behind an API, the model provider almost always sits outside the SOC 2 examination as a carved-out subservice organization. The auditor confirms management reviewed the provider’s own report, which typically covers the hosting platform and sits behind an NDA. A company can hold a clean ISO/IEC 42001 certificate and a clean SOC 2 Type II without ever running an evaluation against the model it ships.
Reviewers who slow these deals down are responding to a real gap. Some teams have gone further and now pass on any tool with an AI component because the extra review isn’t worth the delay. I understand the impulse, but it’s a blunt instrument.
Chris sees the same pattern in his conversations with CISOs and CIOs. “I think the GRC teams are so overwhelmed right now that they are applying a human approach to a nonhuman problem,” he said. “We’re just going to flatly, unilaterally deny something because we don’t have the bandwidth to understand exactly what the risk assessment is going to look like.”
The concerns behind it are specific, and vendors can answer them. Data separation by customer has been a basic control since the sixties. When a vendor trains a shared model on customer data, that separation disappears, because the data now lives inside the model where prompt injection or another attack might surface it for someone else. At Strike Graph we go through enterprise vendor reviews constantly, and the conversations that clear quickly come down to three answers. We host our own models, so customer data doesn’t go to a third-party provider. We don’t train on customer data, because synthetic data makes that unnecessary. And we can show where the AI produces measurable value.
Token spend without results is a governance failure
McKinsey’s 2025 State of AI survey report found that 88% of organizations use AI in at least one function, and 39% attribute any EBIT impact to it, most of that under 5%. When spend climbs and productivity doesn’t follow, the tool usually went in without measurable objectives. Asking what productivity gain a tool is expected to deliver, and checking whether it showed up, is not a hard conversation to have.
ISO/IEC 42001 already gives organizations the instrument for it. It requires measurable AI objectives, defined monitoring, and a management review that can act on the results, including stopping a project. Most companies run it as a documentation exercise. The ones that use it get results: among organizations in the EY survey that run formal AI reviews, 64% modified at least a quarter of their AI systems and 25% stopped at least a quarter.
There’s a technical problem underneath the budget one. Teams routinely point the most expensive models at tasks that don’t need them. Choosing AI should look more like building IT infrastructure, with the right model matched to each job.
When Strike Graph tested open-source LLMs for mapping security controls across frameworks, accuracy was acceptable and the cost was dramatic. We built our own dataset and trained our own model, which raised accuracy by five to six percentage points at one-seventeenth the size of the nearest large LLM. It’s cheap enough to run that customers use it without limits.
Architecture determines exposure and cost
The architecture behind an AI tool shapes its risk more than its feature list does. Prompt injection has mitigations and no fix, because instructions and data travel as the same tokens and the model can’t tell them apart. Simon Willison’s lethal trifecta names the combination that makes exfiltration possible: private data, untrusted content, and the ability to communicate externally in one system. Remove one leg per system. An agent that reads untrusted email and holds customer records shouldn’t also get an outbound connection.
A security executive I spoke with recently put it simply: until you’re confident in every outcome, put the agent on an island.
BYOD shows why this works. Policy and training never solved it. Android Work Profile and Apple User Enrollment did, by building the data boundary into the platform. Buyers should expect vendors to explain what their AI can reach, which models it runs on, and how those choices affect both cost and exposure.
Testing methodology is a trust signal
Every framework leaves the evaluation criteria to the organization being assessed, so buyers need to ask for them directly. The question that trips vendors up most is the simplest one: how accurate is your model? Few can answer it.
A trustworthy vendor can name the metric it tests against, the threshold it set, the conditions it tested under, and how often it retests when the model version changes. Qualitative output like a natural-language response is harder to measure, but it can be done. We send those responses to experts in the field to rate their validity, and we don’t release a model without at least that measure of accuracy.
A system that can give different answers to the same input needs testing across a range. Good testing isolates one factor at a time. We used to adjust a single field across a series of records, resubmit them, and watch how the outcomes shifted. Apply the same harness to log files or claims data and you learn how far the model’s answers can move, which is what a risk owner needs to know.
The EU AI Act’s Article 15(3) offers a usable template for disclosure: declare the level of accuracy, the metrics used, and the testing conditions in the documentation that ships with the system.
What GRC teams should evaluate before approving an AI tool
Here are some key questions to ask before approving an AI tool:
- Ask for the certificate’s scope statement and list of excluded controls, and confirm the AI component falls inside that scope. For SOC 2, check whether the AI sits within the system description boundary.
- Ask what testing methodology the vendor uses and what threshold it holds itself to.
- Ask whether your data trains the model, and how the vendor records data provenance.
- Ask which systems, data, and external connections the AI can reach.
Then follow these steps to hit achievable milestones:
- Start with discovery before writing policy. Your risk profile, the tools you’re adopting, and your tolerance for error are specific to you, and a testing program built without that context can become too expensive to run.
- Don’t spend political capital reinventing AI assessment. The NIST AI RMF and ISO/IEC 42001 are community-built starting points, even if you use them only to inform what to look for.
- Trim what doesn’t apply and keep what does.
That approach changes the GRC team’s role. Instead of reacting to each tool a business owner brings forward, GRC can hand the business a menu of tools that already meet its requirements.
Stay flexible, but also be resilient
The regulatory ground will keep shifting. The EU AI Act has pulled back or postponed parts of its rollout, and CMMC paused its phased rollout for a time. Those adjustments are normal when standards arrive this close to the technology. Nothing stops a company from documenting the control operations that address the risks it has analyzed and validating them with evidence. That work carries over to whatever framework comes next.
That evidence also has to keep pace with the models. “You’re not doing a risk analysis once and hoping for the best,” Chris said. “These AI models are changing at least monthly, if not weekly, and so it might change your risk profile.”
It’s also fine to move deliberately. A friend of mine who served in the military used to tell me that first to market is the first target downrange. Let your validated opportunities, your efficiency goals, and your risk concerns set the pace.
Watch the conversation on demand
Justin Beals, CEO and Founder of Strike Graph, joined Chris Steffen, VP of Research at EMA, for a fireside chat on why procurement teams are slowing AI purchases, what token sprawl costs organizations, and why architecture should be a top consideration when evaluating AI vendors. The session is built for compliance leaders, risk officers, GRC directors, and business decision makers evaluating AI tools.
[Watch on demand: https://insights.strikegraph.com/webinar-watch-ai-governance-gap
Strike Graph’s free AI Vendor Evaluation Checklist covers 35 questions on data exposure, testing, architecture, and governance fit.
