Enterprise AI Agent Governance: Human-in-the-Loop Approval, Escalation and Audit Trails

The moment an agent can write to a system — not just summarise or retrieve, but create, send, approve or change — it stops being an analytics feature and becomes an actor with delegated authority. The right mental model is not 'a smarter chatbot' but 'a new junior team member who never sleeps, works very fast, and occasionally misunderstands'. You would not give that person unsupervised authority to release payments on day one.

So governance is not a compliance afterthought bolted on before go-live; it is the architecture. The useful question for every agent is: what is the worst thing this can do in a single action, and who should see it before it happens? The answer sets the control, and it differs enormously between drafting an email and committing a purchase order.

A single policy for all agent actions is either too loose for the dangerous ones or too heavy for the trivial ones. Classify each action the agent can take by blast radius. Reading data and producing a draft for a human is low risk — let it run freely. Actions with internal consequence that are easily reversed — creating a draft record, flagging an item — are medium. Actions that spend money, commit the organisation, communicate externally, or change a live operational system are high risk and get the strongest controls.

This tiering is concrete, not abstract. For a supply-chain agent: 'summarise today's OTIF exceptions' is free; 'draft a chaser email to the carrier' is low; 'raise a replenishment PO under USD 5,000' is medium with approval; 'release a PO over USD 25,000 or email the customer about a delay' is high with mandatory human sign-off. Write the list of actions, assign a tier to each, and the control model follows.

A human-in-the-loop gate pauses the agent before a consequential action and routes it to a named person for approval, with the context needed to decide — what the agent proposes to do, why, and the inputs it used. On the Microsoft stack this is a Power Automate approval surfaced in Teams or Outlook: the agent prepares the action, the human approves or rejects with one tap, and only then does it execute.

Place gates by the risk tier, not everywhere. Gating a draft wastes the human's attention and trains them to ignore the prompts; gating a payment release is exactly where their attention belongs. The design goal is that every approval a human is asked for is one that genuinely needed a human — anything else erodes the control through fatigue, which is the failure mode that quietly defeats most approval systems.

Separate from fixed approval gates, the agent needs rules for when to hand off even on actions it is normally allowed to take. Low confidence in its own interpretation, inputs that fall outside the policy it was given, conflicting data, or an edge case it has no clear path for — all of these should trigger escalation to a human rather than a best-guess action. An agent that proceeds confidently when it is actually uncertain is the most dangerous configuration.

Make escalation a first-class, designed path, not an error state. Define the conditions explicitly (confidence below a threshold, value above a limit, a field outside an allowed set), name who each type of case escalates to, and give the human the full context. A well-governed agent does more of the routine work precisely because the organisation trusts it to stop and ask on the unusual cases.

Every action the agent takes, every input it used, every decision it made and every approval or rejection must be logged to a durable, tamper-evident record. This is not optional for anything that touches money, customers or regulated processes — when someone asks six weeks later why a PO was raised or a customer was emailed, 'the AI did it' is not an answer, but a full trail showing the inputs, the agent's reasoning, the approver and the timestamp is.

The audit trail also makes the agent improvable: patterns of escalation and rejection show where its judgement is weak and where the thresholds are wrong, and reviewing rejected actions is the fastest way to tune the policy. Log to a governed store — in a Fabric estate, into OneLake alongside the operational data — so the agent's behaviour is analysable with the same tools as everything else.

Approval fatigue is the quiet killer. If the agent asks for too many approvals, or asks for low-value ones, humans start rubber-stamping, and a rubber-stamped approval is no control at all — it is theatre with a name attached. The entire effectiveness of a human-in-the-loop model rests on the thresholds being set so that every request genuinely warrants a human's attention. Getting those thresholds right is the real work, and it needs tuning against real behaviour.

Governance also cannot exceed the quality of the data and permissions underneath it — an agent acting on wrong data, correctly approved, still does the wrong thing, so data quality and access control are prerequisites, not details. And an audit trail is retrospective: it proves what happened and enables improvement, but it does not prevent the first instance of a bad action. Prevention comes from the risk tiering and the gates; the trail is how you learn and answer for it.

Before an agent is allowed to act, do the governance design — it is a short exercise that prevents expensive incidents.

• List every action the agent can take, and assign each a risk tier by its single-action blast radius
• Set human-in-the-loop approval gates on the medium and high tiers only, surfaced in Teams or Outlook
• Define escalation conditions (low confidence, out-of-policy, over a value limit) and name who each escalates to
• Log every action, input, decision and approval to a durable, tamper-evident trail in a governed store
• Start the agent in a low-authority configuration, review the audit trail, and widen its authority only as the trail earns trust