Before your first AI agent pilot: a practical checklist

A good first AI agent pilot is one bounded workflow with a measurable baseline, low-consequence permissions, representative evaluation cases, a spending ceiling and a named human escalation path. It should end with a decision to stop, revise or expand. A fluent demo is evidence that the interface works; it is not evidence that the workflow is ready for live responsibility.

This checklist is a set of source-based recommendations for the person accountable for a pilot. It is intentionally product neutral. Anthropic's guidance on effective agents recommends starting with the simplest workable design and adding agentic complexity only when it improves results. That is a useful constraint when a pilot is under pressure to look impressive.

1. Choose a workflow with edges

Write the workflow as a trigger, authorised inputs, expected output and clear end state. Prefer work that is frequent enough to evaluate, narrow enough for a domain owner to judge, and reversible when it goes wrong. Avoid an open-ended brief such as “help everyone with customer service”. A better candidate is “classify new support requests into the existing queue and draft a suggested response for human approval”.

• Name the user and domain owner.
• Record the current process, quality level, handling time and known failure cases where those measures are available.
• Define what is outside the pilot, including markets, data classes and actions.
• State the business decision the pilot will inform.

2. Set permissions before prompts

List every system and action separately. Begin with the least privilege that can test the workflow. Read-only access plus a draft for approval is usually enough to learn whether the agent understands the task. Writing to a sandbox is a different permission from writing to production. Sending a message, changing a price or issuing a refund needs its own control and should not arrive as a side effect of broad credentials.

Illustrative permission map

May read: new tickets, approved knowledge base
May produce: category suggestion, response draft
Must ask: missing order number, conflicting policy
Must escalate: threat, payment dispute, personal-data request
May not: send, refund, edit customer data, change policy

3. Build the evaluation set first

Collect representative past cases and remove data the pilot does not need. Include normal tasks, ambiguous inputs, missing information, hostile instructions, tool failures and cases that must be refused or escalated. Define the expected behaviour before running the agent. Depending on the task, grading can combine exact checks, a model-based rubric and human review, but the domain owner must calibrate what acceptable means. Anthropic's evaluation guide explains why task-specific graders and regression cases need to be established early.

• Measure task success and harmful failures, not just completion.
• Record sample size and inspect the distribution instead of quoting one average.
• Keep model, prompt, tool and policy versions with each run.
• Require the agent to fail safely when a tool or source is unavailable.

4. Put a ceiling on operation

Set a maximum number of live cases, a calendar end date and a total spending ceiling before the pilot begins. Include inference, tool, hosting, observability and human review costs. Cost saved needs a defensible baseline; absent that, report the cost of running the pilot and the quality observed. Add per-run limits for tool calls, retries and elapsed time so one broken case cannot consume the whole budget.

The implementation should match the task. Microsoft Foundry's Agent Service overview distinguishes managed prompt agents, hosted custom agents and direct API approaches. A pilot does not earn orchestration complexity merely because a platform makes it available.

5. Run in shadow, then widen carefully

First run the agent against evaluation cases. Then let it observe or draft alongside the current workflow while a human still performs the real action. Compare the two. Only widen permissions when the acceptance criteria are met across representative cases and the remaining failures are understood, detectable and recoverable. A domain owner should be able to pause the pilot without waiting for a platform team or vendor.

6. Hold the stop decision

Schedule the decision while the scope is still fixed. Stop if the workflow is too variable to judge, required data cannot be handled safely, harmful failures remain hard to detect, or review costs erase the value. Revise when the use case is sound but tools, instructions or evaluation coverage are weak. Expand only when the evidence supports the next specific permission or population.

A pilot succeeds when it produces a trustworthy decision, including a decision to stop.

Keep the decision record with the baseline, evaluation results, incidents, costs, unresolved risks and next owner. That package is more valuable than a polished demonstration because it tells the organisation what it can safely do next. For the wider delivery discipline around agent projects, see What 15 years of e-commerce delivery taught me about shipping AI agents.