Most organizations now have an AI governance framework. Legal drafted an acceptable use policy, HR shared it with employees, IT weighed in, and security reviewed the technology. Many companies also updated vendor contracts, revised internal policies and briefed the board.
That work was essential. But it was built for a different type of AI.
The governance frameworks developed over the past two years assumed employees would use generative AI as an assistant. A person asks a question, AI provides an answer, and a human decides what happens next. Human judgment remains at the center of every decision.
Agentic AI changes that model. Instead of responding to prompts, these systems receive a goal, break it into tasks, use multiple tools, make decisions and adapt as they work, often with little or no human intervention.
As organizations begin deploying agentic AI across legal, finance, procurement and customer operations, governance must evolve too. Policies designed for chatbots are no longer enough.
Having served as general counsel at four technology companies, I’ve seen both sides of technology adoption: the pressure to innovate quickly and the consequences when governance fails to keep pace. Before deploying agentic AI, every leadership team should be able to answer five critical questions.
-
What happens when the AI gets it wrong?
Product demonstrations showcase success. Governance planning should focus on failure.
An agentic AI system may send inaccurate information to a customer, retrieve data it should never access or take an action that appears logical based on incomplete information. Unlike traditional AI tools, it may complete several steps before anyone notices something has gone wrong.
Before deployment, ask vendors about known limitations, testing methods and failure scenarios. Request model or system documentation that explains how the product performs outside ideal conditions. If a vendor cannot clearly explain its risks, that’s an important indicator of the product’s maturity.
-
Who is responsible when something goes wrong?
This is often the most overlooked question.
Every agentic AI deployment typically involves three parties: the foundation model developer, the platform provider and your organization. Each controls different parts of the system, but responsibility rarely aligns neatly with control.
In practice, software providers frequently limit their contractual liability. That often leaves the organization deploying the technology carrying most of the legal and operational risk.
Before signing any agreement, map where responsibility begins and ends for every party involved. If your organization assumes most of the risk, ensure leadership understands that before implementation begins.
-
Does your vendor contract reflect agentic AI?
Many AI contracts were written before autonomous agents became widely available. As a result, they often fail to address the risks these systems create.
Several provisions deserve particular attention. Your vendor should notify you before making significant model changes, since updates can alter how an agent behaves. You should also have access to activity logs that allow you to reconstruct decisions if an incident occurs.
Contracts should clearly define data retention practices, specify whether customer data is used for model improvement and establish meaningful protections if autonomous actions cause harm. Finally, organizations should negotiate the right to terminate agreements if safety performance deteriorates after deployment.
These provisions may seem detailed, but negotiating them before deployment is far easier than after an incident.
-
What authority should the AI actually have?
One of the biggest governance mistakes is giving AI systems more authority than intended.
An AI agent only knows what its permissions allow. If it can send emails, it will send emails. If it can approve transactions or modify documents, it will do so whenever those actions support its assigned objective.
Organizations should create an authority matrix before deployment. Every action should fall into one of three categories: autonomous, requiring human approval or prohibited entirely.
External communications deserve particular caution. Any action that sends information outside the organization should generally require human approval, regardless of how capable the technology appears.
-
Is your board receiving meaningful oversight?
Many board updates still summarize AI with a single statement that the company is “using AI responsibly.” That is no longer sufficient.
Agentic AI affects cybersecurity, regulatory compliance, vendor management, intellectual property and reputation. Boards do not need technical explanations of large language models, but they do need visibility into how autonomous systems operate inside the business.
Effective reporting should include an inventory of deployed AI agents, their business purpose, executive ownership, risk classification, significant model changes and any incidents or near misses. Building this reporting process early makes governance far more manageable as adoption scales.
Governance Must Evolve With the Technology
The governance work organizations completed over the past two years was not wasted. It established policies, raised awareness and created a foundation for responsible AI adoption.
However, agentic AI represents a fundamental shift. Generative AI produced recommendations for humans to evaluate. Agentic AI increasingly takes action on behalf of the business.
That changes the governance challenge entirely.
The organizations that succeed with agentic AI will not necessarily be those that adopt it first. They will be the ones that establish clear accountability, appropriate controls and meaningful oversight before autonomous systems become embedded in everyday operations.
Legal leaders should be involved from the beginning, not called in after the first serious incident. In the era of agentic AI, governance is no longer about managing prompts. It’s about managing autonomous decision-making.
