Compare AI tools on tasks you genuinely need to complete, using the same inputs and a scoring rubric chosen before inspecting the answers. Representative tasks show how each tool fits the work you expect to complete.
Anthropic’s evaluation guidance separates tasks and grading methods, including deterministic, model-based and human checks. Use that distinction without treating one provider’s examples as independent evidence that its own tool will win.
Include the time spent checking and editing when comparing results. A deliverable is useful once it is ready for its intended purpose, so that final stage belongs in the assessment. This gives both tools the same practical standard to work towards.
Build a small representative set
Include ordinary work, difficult cases and examples where the correct response is to identify missing information. Remove private material unless you have authority to use it in the systems being compared. Keep expected answers or review criteria outside the generated responses.
Record the tool, version or available model information, settings, date and features used. Compare equivalent access where possible, and state differences that cannot be removed.
Score the useful result
| Criterion | Check |
|---|---|
| Task completion | Did it produce the requested usable result? |
| Accuracy | Do independent checks support important claims? |
| Constraints | Were required details and limits preserved? |
| Correction work | What did a reviewer need to change? |
| Operation | What time or manual intervention was required? |
Hide tool names during subjective scoring where practical. Use independent expected answers or source checks for the factual parts of the assessment.
Keep the failures in the decision
Repeat consequential cases enough to investigate variability, documenting what you actually ran. Report the sample’s limits and retain rejected outputs. This article supplies an evaluation design, not results from a comparison we have not performed. Describe the choice in terms of the tasks assessed, including the results, review effort and limitations observed.
Use a small scoring rubric
Here is a fictional rubric for a document-extraction task. Choose the criteria and importance for the actual work before comparing tools.
| Criterion | Observable check |
|---|---|
| Completeness | All required fields supplied or marked unresolved |
| Fidelity | Values match the supplied source |
| Format | Output fits the agreed structure |
| Review effort | Corrections and time recorded |
| Limits | Unsupported or ambiguous cases identified |
Keep an expected answer for each task where one can be established. If a criterion needs human judgment, write a short description of what a satisfactory result looks like.
Follow one result through correction
Suppose two tools extract dates from the same fictional notice. One returns quickly but omits a condition; the other includes it and needs a formatting adjustment. Record both the omission and the formatting work, then compare the finished deliverables under the rubric.
Repeat representative cases with the tool version, settings and supplied material recorded. Include difficult inputs such as an ambiguous date or a missing field when they occur in the intended workload. If the comparison is small, describe the outcome as evidence about that set of tasks. The practical conclusion can be narrow: which tool fitted this workflow, how much review it needed and what further checking would be useful.
Choose one ordinary task and one demanding case from the work you expect to do. Define the expected result for each before comparing responses. Keeping the corrections and review time beside the score shows which tool fits the workload and where extra checking remains useful.
Sources consulted
- Demystifying evals for AI agents — Anthropic; consulted 3 November 2026.