Enterprise AI Agents Need Restraint: Confidence Is Not Authority

As AI agents move from answering questions to taking actions inside enterprise systems, a new design principle is emerging among CIOs: restraint. The ability of an AI agent to recognize when it lacks the authority, context, or judgment to proceed may matter more than its ability to complete a task.

CIO.com spoke with technology leaders at PwC, Celigo, and CarGurus to surface the governance frameworks that separate pilots from production-ready agentic deployments.

Key Takeaways

• "Confidence is not authority": an AI agent may be 99% confident that a database should be deleted or a refund should be approved, but that does not mean the organization has delegated that decision to the system. Allan Dabre, Technology Compliance and AI Lead at PwC, argues this distinction must be deliberately designed into every agentic workflow
• The "agent harness" concept: a controls layer external to the model that defines what an agent can and cannot do, replacing policy documents with enforceable software configuration
• "Least agency" principle (Matt Graney, CPO at Celigo): give agents the minimum autonomy needed to complete the job. Too many tools create confusion, especially as context windows and task complexity grow
• Guardrails should sit outside the model. An agent that judges its own outputs provides weaker control than an external guardrail that checks inputs and outputs before downstream actions execute
• CarGurus handles 70% of dealer support cases end-to-end with agents, but routes consequential actions through human approval with a simple prompt: "I am about to do this. Do you want me to proceed?" (Matt Quinn, CTO at CarGurus)
• Accountability must be pre-defined before deployment: the business defines the outcome, technology builds the agent, risk and compliance set the guardrails, and governance monitors behavior. The authority to pause or retire an agent should be settled before production, not negotiated during an incident

The next phase of enterprise AI maturity will be defined not by agents that always answer or always complete the task, but by agents that know when not to act.

Read the full article on CIO.com