The Governance Question Changes When Systems Can Act
AI agents present a different governance problem. Earlier AI tools generally supplied information for human assessment; agents can also retrieve records, choose tools, coordinate tasks, and act within connected systems. Software has long automated consequential decisions, but language-model agents can interpret broadly stated goals and select intermediate steps no operator expressly prescribed. The authority granted to an agent therefore warrants as much scrutiny as the reliability of its answers.
Competence and authorization should not be confused. An agent may correctly spot an inconsistency in a contract without having permission to amend the agreement, contact a counterparty, or alter a customer record. An action within its technical permissions may still be improper if it rests on a contested interpretation, grants an unapproved concession, or affects someone who cannot participate. Governance must specify which judgments an agent may assist, which decisions it may execute, and who bears responsibility for the result.
Judgment Is More Than Accurate Prediction
Arguments about AI and judgment often assume judgment is a capacity people possess and machines do not. In practice, professional judgment brings together factual assessment, prediction, interpretation, valuation, and authority, not necessarily exercised by the same person. AI can perform some of this work well enough that repeating a defined task adds little value. Competence at a task, however, is not a grant of decision-making authority.
For governance purposes, it helps to distinguish three kinds of judgment. Epistemic judgment assesses the credibility, completeness, and uncertainty of evidence; practical judgment weighs competing interests and foreseeable consequences; institutional judgment concerns who may decide on behalf of an organization or another person, under which rules, and with what duty to give reasons. These functions overlap. Still, an agent’s proficiency in classifying documents or predicting outcomes does not entitle it to resolve a disputed issue.
Suppose a contract-review agent flags a limitation-of-liability clause outside an organization’s preferred range. Detecting the deviation is one task; deciding whether to accept it requires attention to the commercial relationship, negotiating history, insurance coverage, and the kinds of losses the organization is prepared to bear. Sending alternative language to the counterparty is another act, one that may shape expectations even before a contract exists. Treating all three as a single “contract review” function obscures the allocation of responsibility.
Agency Introduces Distinct Sources of Risk
The risk posed by an agent depends in part on the systems it can access, the changes it can make, and the opportunity to stop it before harm occurs. An error confined to a draft is different from an erroneous instruction delivered to a bank, employee, or contractual counterparty. Access to external tools joins ordinary software-permission risks to the difficulty of interpreting open-ended instructions. NIST’s 2026 design tenets for agent identity address this combination through distinct identities, bounded delegation, access decisions, and audit records [1].
An agent’s ability to complete a task is not evidence, on its own, that it can safely operate without supervision. AgentDojo, a peer-reviewed evaluation environment, tested agents encountering untrusted material in settings including email and online banking; researchers observed failures on legitimate tasks and susceptibility to certain prompt-injection attacks [2]. The findings depend on the particular tasks, systems, and attacks tested and cannot be reduced to a general failure rate. Even so, a successful demonstration provides little basis for granting consequential permissions.
Not every failure involves an attacker. An agent might pursue a narrow performance target and miss a contractual exception, use an outdated record, or assemble otherwise permitted steps into a prohibited outcome. In multi-agent workflows, mistaken assumptions may pass between systems without being noticed. Evaluation needs to follow the entire course of action, including intermediate decisions, rather than score each model response separately.
Human Oversight Must Be Substantive
A human being in the process does not necessarily provide effective oversight. A reviewer may lack the time, information, expertise, or practical ability needed to challenge an agent’s proposed action. Researchers have documented overreliance on automation for decades; in a 2024 experiment, allowing participants to monitor and adjust algorithmic recommendations increased their acceptance of the recommendations while reducing accuracy on the task studied [3, 4]. Although the experiment did not involve AI agents, its findings undermine the assumption that requiring approval ensures independent review.
A meta-analysis spanning 106 experimental studies likewise found that combinations of humans and AI performed worse on average than the stronger of the two working alone, although they often outperformed humans on their own [5]. The results differed substantially by task and study design and should not be read as a prediction about any particular deployment. Organizations therefore need evidence about the human-and-system process itself rather than separate assessments of the human and the model. Human review may strengthen accountability without improving the underlying decision.
Effective oversight requires a reviewer to understand the proposed action, examine its supporting evidence, decline it without undue difficulty, and intervene before the consequences become irreversible. Articles 14 and 26 of the European Union’s AI Act reflect parts of this approach: the former addresses oversight of high-risk systems, and the latter requires covered deployers to assign oversight to people with appropriate competence, training, authority, and support [6]. These provisions concern high-risk systems, and a July 2026 amendment postponed their application until December 2027 or August 2028, depending on system type [6]. The Act’s emphasis on reviewers’ actual capacities is relevant beyond the systems to which those provisions apply.
Governance as the Design of Decision Rights
NIST’s voluntary AI Risk Management Framework divides risk management into govern, map, measure, and manage functions [7]. For agents, however, the starting point should be the decisions being delegated, not merely the models being deployed. An organization should identify the people whose interests each workflow might affect, the information an agent may consult, the systems it may alter, and the actions that call for separate approval. It should distinguish read-only analysis from drafting, bounded execution, and conduct that may have legal, financial, or personal effects, then calibrate review according to possible harm, uncertainty, reversibility, and available remedies.
Those boundaries should appear in technical permissions, not just written instructions. An agent should have an identity distinct from an employee’s, credentials limited to its assigned task, and restrictions that prevent a second agent from receiving broader authority than the first. NIST’s design tenets specify distinct identification, narrowing authority across successive delegations, and activity records maintained independently of the agent [1]. Such controls cannot determine whether a business choice is sound, but they can prevent the use of authority never granted.
A task-completion score cannot establish whether an agent is suitable for deployment. Tests should include incomplete records, conflicting instructions, malicious retrieved material, changed permissions, mistaken actions, and recovery after a person intervenes. NIST’s 2026 ARIA evaluation manual incorporates model testing, red teaming, and user testing, offering a way to assess both model behavior and the system in which it is used [8]. In addition to speed and completion rates, evaluations should record unauthorized acts, missed escalations, the seriousness of errors, and whether corrections succeed.
Accountability also depends on preserving evidence of what happened and providing a means of correction. For an agent-mediated decision, the record should show the instructions then in effect, sources consulted, tool use, permissions, approvals, and resulting changes; a retrospective narrative is no substitute. Anyone affected by a consequential decision should have a way to challenge an error and reach a person empowered to fix it. Logs may facilitate an investigation, but they offer little remedy on their own.
Professional Responsibility and Institutional Accountability
Legal practice makes the distinction between assistance and delegation particularly clear. The American Bar Association’s Formal Opinion 512 explains that using generative AI does not relieve lawyers of duties concerning competence, confidentiality, supervision, and independent verification appropriate to the task [9]. The opinion does not prescribe rules for autonomous agents, but its underlying requirement remains: lawyers are accountable for their clients’ work. Assigning work to software does not transfer professional duties.
Outside legal practice, a manager may authorize an agent to recommend hiring decisions while retaining responsibility for the criteria applied, the information used, and access to review. Responsibility is not limited to whoever clicks “approve”: designers set available actions, leaders establish incentives, and procurement or compliance teams determine conditions of use. The OECD AI Principles similarly connect accountability to each actor’s role, the context, and their ability to act [10].
Preserving the Conditions for Judgment
Human review does not invariably produce better results, just as strong automated performance does not establish that human judgment is unnecessary. Deciding which work to automate requires a specific account of where decision-making authority belongs. Some activities can proceed automatically subject to later audit; others warrant prior approval, independent evidence, or a rule against autonomous execution. These choices should be documented and reconsidered as the system, the organization, and the effects of deployment change.
With AI agents, an organization can delegate substantial discretion by setting permissions that appear merely technical. Final sign-off cannot compensate for poorly defined authority or ineffective review. It must identify decision-makers who possess the information, authority, and resources to intervene, explain outcomes, and correct errors. Those obligations remain with the institution however much work the agent performs.
References
[1] Harold Booth, Bill Fisher, Ryan Galluzzo, and Tom Clancy, “Design Tenets for Agent Identity Management Systems,” NIST Cybersecurity Insights (October 27, 2026). https://www.nist.gov/blogs/cybersecurity-insights/design-tenets-agent-identity-management-systems
[2] Edoardo Debenedetti et al., “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents,” Advances in Neural Information Processing Systems 37 (2024). https://arxiv.org/abs/2406.13352
[3] Raja Parasuraman and Victor Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse,” Human Factors 39(2) (1997). https://doi.org/10.1518/001872097778543886
[4] Daniela Sele and Marina Chugunova, “Putting a Human in the Loop: Increasing Uptake, but Decreasing Accuracy of Automated Decision-Making,” PLOS ONE 19(2) (2024). https://doi.org/10.1371/journal.pone.0298037
[5] Michelle Vaccaro, Abdullah Almaatouq, and Thomas W. Malone, “When Combinations of Humans and AI Are Useful: A Systematic Review and Meta-Analysis,” Nature Human Behaviour 8 (2024). https://doi.org/10.1038/s41562-024-02024-1
[6] Regulation (EU) 2024/1689, Artificial Intelligence Act, Articles 14 and 26, as amended by Regulation (EU) 2026/1744. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
[7] National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (2023). https://doi.org/10.6028/NIST.AI.100-1
[8] Theodore Jensen et al., ARIA Evaluation Planning Manual: Elements of ARIA-Style AI Evaluations, NIST AI 200-3 (September 2026). https://doi.org/10.6028/NIST.AI.200-3
[9] American Bar Association Standing Committee on Ethics and Professional Responsibility, Formal Opinion 512, “Generative Artificial Intelligence Tools” (July 29, 2024). https://www.americanbar.org/content/dam/aba/administrative/professional_responsibility/ethics-opinions/aba-formal-opinion-512.pdf
[10] Organisation for Economic Co-operation and Development, OECD AI Principles, Principle 1.5: Accountability (adopted 2019, updated 2024). https://oecd.ai/en/dashboards/ai-principles/P9
