The Agentic Revenue CompanyRevenue as a Service
AI agents wiki · Testing & evaluation

The new definition of done for AI agents

An AI feature is not done because the code compiles and the prompt sounds convincing. It is only “done” when an automated evaluation runs reliably – before rollout and in operation.

Why classic tests are no longer enough

Classic software is predictable: input A and function B produce output C. This determinism makes robust tests possible. Generative AI is stochastic – the same prompt can produce a different result on Monday than on Tuesday. Unit tests as known from software development do not cover this.

If you want to use AI agents reliably, you cannot rely on “vibe checks” that work today and fail with a customer tomorrow. You need a dedicated evaluation layer – especially where hallucinations are a compliance risk.

The new evaluation paradigm

Classic tests are mostly binary: pass or fail. AI evaluations also contain binary checks but often assess on a spectrum. An evaluation is therefore not a single script but a pipeline of checks – from strict syntax checks to semantic quality judgements.

Layer 1: deterministic checks
Code, regular expressions and schema validation secure the structure: was valid JSON produced in the right schema? Was the right tool called with the required arguments? Is a field such as an email address filled in correctly? This layer comes first (fail fast) – if the structure is broken, expensive semantic checks are pointless.
Layer 2: model-based checks
Whether an answer is helpful, appropriate or free of hallucinations can hardly be checked with code. Here a language model acting as LLM-as-a-Judge assesses against a rubric. People can do this too, but not across thousands of cases with every change.

What makes an LLM judge reliable

  • A capable model as judge – ideally stronger than the production model, especially if that one was chosen smaller for latency reasons.
  • A strict rubric with clear gradations from failure to success; vague instructions produce noise.
  • Reference answers (golden outputs) reviewed by people, so the judge compares against a verified benchmark.
  • A justification per criterion, so failures can be traced and fixed.

Offline evaluation: the gate before rollout

The offline pipeline is for regression testing: errors, drift and latency issues are detected before a change goes to production. It is based on a versioned golden dataset that reflects real use – standard cases, edge cases and deliberate manipulation attempts. Its size depends on the breadth of the tasks; a few hundred cases are one possible example. Synthetic cases can speed up curation, but domain experts must review and approve them.

Scoring is hybrid and weighted. Example for an agent that sends emails: part of the points go to layer 1 (correct tool call, valid schema), part to layer 2 (subject matches the intent, content free of hallucinations, recipients correct). If a deterministic check fails, the case fails immediately.

The pipeline runs as a blocking step before a change is merged. Which pass rate applies is something you set per risk – higher in regulated areas than for internal helper tasks. Every fix is followed by a full run, because a fix in one place can cause new errors elsewhere.

Online evaluation: signals from operation

After rollout, evaluation becomes telemetry. Deterministic checks are fast enough to run on all traffic. An LLM judge, by contrast, reviews a sample asynchronously, for example a few percent of cases – where the data protection framework allows – so that latency and cost on the critical path do not rise.

  • Explicit signals: positive or negative user ratings and free-text feedback
  • Implicit signals: frequent regeneration, conspicuously many apologies or refusals
  • Deterministic checks in production: schema and tool conformity to detect drift or API changes early
  • Asynchronous judge ratings on a sample

The loop of continuous improvement

Evaluation is not set-and-forget. Datasets age because questions and conditions change: an agent that performs well on familiar topics can fail immediately on a new one. Without a feedback loop, a deceptive picture emerges – good scores in the test lab while quality for real users declines.

  • Capture: a negative signal in operation
  • Triage: flag the session for human review
  • Root cause: analyse the gap and improve the system
  • Extend the dataset: add the case with a corrected expected answer to the golden dataset
  • Safeguard: include the new edge case in all future runs

Definition of done and the outcome

The new definition of done shifts the standard: measure instead of guess, iterate instead of hope, treat quality as ongoing operation. An agent is never finished – it is as good as its evaluation and the people who act on its signals.

For agents in sales this means: the evaluation criteria derive from the business goal and the mandate, that is, from what an agent may prepare, recommend or execute with approval. In AI-first coaching you clarify goals, mandate and your AI-first operating model. You pay for results – not for activity.

From tool to outcome

In AI-first coaching you clarify which outcomes AI agents should be accountable for in your go-to-market and within which framework. REVAGO then designs, builds and runs your system. You pay for results – not for activity.

Frequently asked questions

What is the definition of done for AI agents?

For AI agents, a feature is only done when an automated evaluation runs reliably: against a curated golden dataset before rollout and with signals from operation afterwards. Compiling code and a convincing prompt are not enough.

Why are unit tests not enough for AI agents?

Unit tests assume deterministic behaviour. Language models respond stochastically, and qualities such as helpfulness or faithfulness cannot be checked by code alone. Model-based checks and human-validated reference answers therefore complement classic tests.

What is a golden dataset?

A golden dataset is a versioned collection of test cases with an exact input and an expected answer reviewed by people. It reflects real use, including edge cases and manipulation attempts, and is continuously extended with new cases from operation.

What is the difference between offline and online evaluation?

Offline evaluation checks changes against the golden dataset before rollout and blocks on regressions. Online evaluation monitors live operation with user and behavioural signals, deterministic checks and judge ratings on a sample.

Further reading in the wiki

A neutral overview, not a recommendation of individual providers. REVAGO is not affiliated with the listed projects. Descriptions are based on the projects’ public information; features and licences may change. Selection based on awesome-agents (kyrolabs).