How to Evaluate an AI Agent Before Production
فارسی
The short answer: an AI agent is a system that manages steps toward a goal, selects tools, observes what happened, and adjusts or hands control back to a person. A polished demo does not establish that it is production-ready. The useful question is whether it completes a defined task with inspectable evidence, acceptable cost, and safe control boundaries.
This guide provides a seven-part scorecard for product and engineering teams. Download the AI agent evaluation scorecard in CSV format, replace the example weights when your risk profile requires it, and score representative tasks before expanding permissions.
What makes a system an agent?
A chatbot response or a single model call is not automatically an agent. OpenAI’s practical guide describes agents as systems that independently accomplish tasks by managing workflow execution and selecting tools within guardrails. Anthropic describes a self-directed loop in which the system plans, acts, observes, and adjusts until the task is complete or human input is needed.
The distinction matters because tool access changes the failure surface. Summarizing an invoice is a model task. Finding invoices, extracting fields, reconciling a policy, updating an accounting system, and escalating an exception is an agent workflow. Each additional action can increase value, but it also adds permissions, external state, and recovery requirements.
Do not use an agent where deterministic automation is enough
If the path is stable, rules are explicit, and every valid state is known, conventional software is usually easier to test and cheaper to operate. Agents are more promising when the workflow depends on contextual judgment, messy documents, or exceptions that made a rules engine brittle. Even then, start with the smallest useful scope. A read-only agent with three well-defined tools is a better first experiment than an assistant with immediate access to email, payments, files, and production administration.
| Workflow | Safer starting point | Warning sign |
|---|---|---|
| Stable and repetitive | Deterministic code or rules | Adding an LLM without a decision need |
| Context-heavy decision | Narrow agent with read tools | No measurable completion state |
| Irreversible action | Mandatory human approval | Autonomous send, delete, or payment on day one |
| Many specialist roles | Prove one agent first | Premature multi-agent orchestration |
The seven-part AI agent scorecard
Build a set of 20 to 50 tasks sampled from the intended workflow. For every task, define the expected final state, valid inputs, permitted actions, and stop conditions. Run critical cases repeatedly. Score each dimension from zero to five, preserve the execution trace, and calculate the weighted result.
1. Task success — 25%
Measure whether the external state is correct, not whether the final prose sounds convincing. Did the right record change? Was the requested artifact created? Were policy constraints satisfied? Prefer deterministic checks, system read-back, or an independent reference answer. When human judgment is necessary, write the rubric before viewing the output.
2. Evidence quality — 15%
Important claims and actions should be traceable to sources or system records. Track supported-claim rate, invalid citations, stale sources, and evidence that does not actually support the conclusion. Fluency is not a substitute for provenance, especially in research, finance, legal operations, or executive reporting.
3. Repeatability — 15%
One success is anecdotal. Run each high-impact task at least five times under controlled conditions and report the distribution, worst case, and failure rate. NIST’s work on agent hijacking evaluations shows why repeated attempts can materially change a risk estimate. The same principle applies to ordinary stochastic failures.
4. Tool and data safety — 15%
Classify tools by read versus write access, reversibility, sensitive data, and financial impact. Test hostile content in a document, an untrusted link, contradictory instructions, and incomplete inputs. The agent should treat retrieved content as data, not as authority to redefine its goal. Least privilege, isolated environments, authentication, and server-side authorization matter more than a single prompt guardrail.
5. Human control — 10%
Define approvals before the test. External messages, deletion, payment, public publication, and contractual commitments are common checkpoints. A reliable agent should infer routine technical steps while escalating choices of intent, preference, or material risk. Measure both correct escalation and unnecessary interruption; an agent that asks on every step has also failed to deliver useful autonomy.
6. Recovery and safe stopping — 10%
Inject API errors, timeouts, missing evidence, partial data, and a tool response that contradicts the plan. The system should retry within a limit, choose a safe alternative, or stop with a precise account of incomplete work. Infinite loops, concealed failure, and guessed completion should all score zero for that case.
7. Cost and time per accepted task — 10%
Do not optimize on model-call price alone. Add tokens, tool charges, retries, elapsed time, and human review, then divide by accepted tasks. Track median and 95th-percentile latency so a small number of very slow runs remain visible. Establish a quality baseline with the strongest viable configuration before testing cheaper models.
A five-step evaluation plan
- Write the task contract. Define inputs, final state, tools, forbidden actions, and stop conditions.
- Create representative cases. Include ordinary work, edge cases, missing data, and high-risk actions.
- Capture a baseline. Fix the model and settings, repeat runs, and save traces, cost, and duration.
- Inject failure and attack conditions. Break tools, add untrusted instructions, and remove required evidence.
- Set a release decision. Define minimum scores, pilot scope, monitoring, and rollback before users depend on the agent.
Anthropic’s agent-evaluation guidance argues for assessing both the final outcome and the multi-step trajectory. Google’s Agent Development Kit similarly separates response quality from execution-path evaluation against predefined test cases. An outcome-only grader can miss a dangerous shortcut or the wrong tool sequence that happened to produce a plausible result.
Architecture is part of the evaluated system
Model benchmarks do not transfer automatically to your product. The tested system includes the model, instructions, tools, permissions, environment, memory, retry policy, and human checkpoints. Record every one of those in the evaluation report. A model update, new tool, broader permission, or changed system prompt can invalidate the baseline and should trigger a targeted rerun.
The NIST AI Risk Management Framework treats evaluation as part of a broader process for mapping, measuring, managing, and governing risk. Product teams do not need to reproduce a government program, but they should evaluate the system in its actual context rather than reporting a laboratory accuracy number as a deployment guarantee.
Cross-border and multilingual deployment
Language support, regional availability, billing access, and permission to process data are separate questions. Check current official terms, supported regions, data controls, and account requirements before creating a dependency. Do not treat an unofficial access workaround as an architecture.
For Persian workflows, include Persian and Latin numerals, right-to-left layouts, mixed-language names, Solar Hijri and Gregorian dates, half-spaces, scanned documents, and entity collisions. A translated English benchmark will not expose all of the failures Persian-speaking users see. Preserve test cases that reflect real input quality without storing unnecessary personal data.
Single agent or multi-agent system?
More agents do not guarantee better results. Maximize a single agent’s capability first. Split roles when instructions or tools have become so diverse that the system repeatedly selects the wrong capability, or when isolation is valuable. Once multiple agents coordinate, evaluation must include the network: handoff loss, conflicting state, duplicated work, and consistency across the final decision.
The Phronesis Index project is a public preprint exploring consistency detection in networks and multi-agent systems. Its reported results are author-reported and not peer-reviewed, so it is linked here as a research direction and example of a system-level evaluation problem, not as independent validation of this scorecard.
Bottom line
Buy or ship an agent only after it can complete a narrow, valuable task under repeated tests. Measure task success, evidence, repeatability, tool safety, human control, recovery, and total cost per accepted result. If the system cannot recognize missing evidence, stop before an irreversible action, or explain what remains incomplete, it is not ready for broader autonomy.
See Sepehr Bayat’s projects and research, or browse the English blog for future practical work on AI products and multi-agent systems.