Evals for the tasks agents actually perform.
Turn a real production journey or qualified workflow into a controlled task. Compare agents, models, documentation, interfaces, prompts, and product treatments under the same success criteria.
Comparable runs
Measure more than a final pass.
Completion
Did the run satisfy the task criterion?
Friction
Where did the run encounter avoidable effort?
Latency
How long did the task take?
Tokens
What model usage did it require?
Retries
How many attempts were needed?
Failure mode
What prevented the run from completing?
Production ↔ controlled
The same task, two kinds of evidence.
Production reconstruction discovers what matters. Evals make treatments comparable. Production verification shows what changed in the real product.
| Metric | Source | Answers | Limitation |
|---|---|---|---|
| Production ATCR | Reconstructed real product behavior | What agents actually completed | Varies with available production evidence |
| Eval pass rate | Controlled runs with shared criteria | What comparable runs completed | Controlled results are not production proof |
Illustrative matrix
Compare the run, not just the model.
Filter a claim-safe example by product treatment while keeping completion, friction, latency, tokens, retries, and failure mode visible together.
| Agent | Model | Treatment | Stage | Completion | Friction | Latency | Tokens | Retries | Failure mode | Evidence |
|---|---|---|---|---|---|---|---|---|---|---|
| Agent A | Model 1 | Baseline | configure | Blocked | 4/5 | 41s | 8,650 | 2 | Auth scope unresolved | |
| Agent A | Model 1 | Docs treatment | complete | Completed | 1/5 | 29s | 6,120 | 0 | - | |
| Agent B | Model 2 | Interface treatment | complete | Completed | 2/5 | 33s | 7,010 | 1 | - | |
| Agent C | Model 3 | Docs treatment | complete | Completed | 1/5 | 31s | 6,550 | 0 | - |
How each metric is defined
Completion uses the task criterion. Friction counts avoidable recovery work. Latency, tokens, and retries describe the controlled run. Failure mode names the observed blocker.
Evaluate the task that matters.
Bring one workflow, one success criterion, and the agent experience you want to improve.