How to run an AI agent pilot for operations
An AI agent pilot should answer a specific question: can this team complete this workflow more reliably or with less effort, within the access it should have? Choose one recurring task, define what a correct outcome looks like, and collect evidence before widening the rollout. A convincing demo is a starting point for that evaluation.
1. Write a one-page pilot brief
For example, investigate failed customer setup jobs and prepare a next step for operations review. Exclude unrelated account changes and unknown recovery procedures. Name the workflow owner, the people participating, the system owner who can validate results, and the person who can stop the pilot.
- Input: a customer identifier and a reported setup problem.
- Output: the observed status, relevant evidence, open questions, and a proposed next step.
- Permitted scope: named systems, environments, records, fields, and tools.
- Escalation: the owner for missing information, unfamiliar failures, and uncertain outcomes.
- Exit decision: the evidence required to expand, revise, or stop the workflow.
2. Establish a baseline from comparable cases
Sample recent cases in the chosen category. Record handling time, waiting time, engineering involvement, repeat contacts, and whether the original resolution held. Separate active work from time spent waiting for a reply or approval; they describe different bottlenecks.
Keep the case mix visible. If the pilot only handles straightforward requests while the baseline includes severe incidents, a lower average resolution time says little about the agent. Compare similar cases and state which requests were excluded.
3. Prepare cases that can reveal failures
Use examples with known outcomes, reviewed by someone who understands the workflow. Include ordinary requests and situations where the correct behavior is to stop. Use data you are permitted to process, and minimize sensitive fields in the evaluation set.
- Two customers with similar names and only one valid identifier.
- A record outside the participant’s permitted scope.
- An unavailable source, a stale status, or a result that was truncated.
- A failed request that actually completed in the destination system.
- Content in a returned record that asks the agent to ignore its rules.
- A request for an operation that has not been granted to the participant.
Decide the expected answer or refusal before running each case. Source text should be treated as evidence to inspect, without allowing it to grant access or change the workflow’s rules. Record failures with enough context to reproduce them after a configuration change.
4. Evaluate reads before enabling writes
Start with status lookups and evidence summaries. Have reviewers check whether the answer identifies the correct record, uses the permitted sources, distinguishes facts from assumptions, and names missing information. Reviewers should be able to inspect the underlying evidence.
In GlowForm, administrators configure connections, publish tools, and grant policies. The access simulator helps inspect expected authorization decisions. Pair that check with actual workflow trials in a test environment: a permission decision alone does not prove that a connector, a query, or an answer works for your use case.
Define an AI agent’s read, propose, and act permissions
5. Add one reviewable action
Choose a recovery or correction with explicit inputs and a documented way to verify its effect. Decide who may submit it and who must approve it. Test a rejected request, an expired request, and a request whose permission is removed before execution. Verify that each stops as expected.
Keep a manual escalation path and a procedure for disabling the pilot’s tools or grants. After a write, inspect the target system. If the external outcome is uncertain, investigate before repeating it. A successful approval or queue submission is not evidence that the business task finished.
Design the approval step for the pilot’s first action
6. Measure quality and effort together
Track verified resolutions, incorrect answers, appropriate escalations, reopened cases, review time, and engineering time. Keep failures and abandoned requests in the denominator. Inspect the cases behind the numbers so apparent improvement does not come from closing work prematurely.
For a simple capacity estimate, subtract pilot handling and review minutes from baseline handling minutes for comparable cases, then multiply by their volume. Report setup and maintenance effort separately. Time released is capacity the team may use elsewhere; it becomes a cost saving only if spending actually changes.
7. Make the expansion decision explicit
Review results with operations and the system owner. Expand only the scope the evidence supports: another group, a second workflow, or one additional action. If failures cluster around missing data or unclear ownership, address that process gap before adding more tools. Keep the evaluation cases and rerun them when the relevant tools, policies, or model configuration change.