7 Benchmarks for Evaluating AI Agents—and What Each Measures

An AI agent cannot be evaluated by one general knowledge score. A coding agent must modify a repository and pass tests; a browser agent must reach the correct website state; a service agent must follow policy while using tools; and a computer-use agent must complete work across real applications. Each capability needs a benchmark designed around that environment.

Even then, a published result measures the complete system—not just the underlying model. The agent scaffold, system prompt, tool definitions, browser or desktop interface, action budget, retries, environment version, and evaluator can all change the score. Compare results only when those conditions are documented.

Seven benchmarks at a glance

Benchmark	Main signal	Typical evaluation	Major blind spot
SWE-bench Verified	Repository-level bug fixing	Generated patch passes task tests	Does not measure general software work or product judgment
GAIA	General assistant research and tool use	Correct final answer across multi-step tasks	Final-answer scoring reveals little about safe process
WebArena	Long-horizon browser operation	Requested answer or website end state is correct	Self-hosted sites do not reproduce every live-web failure
τ-bench family	Tool use, conversation, and policy compliance	Task state satisfies policy and user goal	Results depend on simulated-user behavior and domain design
ARC-AGI	Few-shot abstraction and adaptation	Correct novel task outputs or interactive progress	ARC-AGI-2 is not a workflow or tool-use benchmark
OSWorld	Computer use across desktop applications	Execution-based checks of the resulting system state	Environment setup and release differences complicate comparison
AgentBench	Breadth across multiple agent environments	Environment-specific success metrics	Older task environments may not reflect a production workflow

1. SWE-bench Verified: repository-level software repair

SWE-bench Verified is a human-reviewed subset of SWE-bench. An agent receives a real code repository and an issue description, then must generate a patch that satisfies the benchmark's tests. The Verified split was created to remove ambiguous or broken tasks from the original set.

Use it when: you need evidence that a coding agent can inspect an unfamiliar Python repository, reason about an issue, modify multiple files, and produce a test-passing fix.

Do not infer: a high score does not establish skill at greenfield architecture, feature discovery, security review, code explanation, other programming languages, or collaboration with a human team. It also does not isolate the base model: repository tools, context retrieval, patch strategy, and test budget are part of the evaluated system.

Report the exact split, harness, allowed tools, maximum steps, test policy, and whether the run used hints, retries, or test-time selection. For an example of a repository-aware coding system, see the MiMo Code agent overview.

2. GAIA: general assistant tasks with tools

GAIA contains real-world questions that can require browsing, reasoning, multimodal input, document handling, code execution, and other tools. The questions are designed to have unambiguous final answers while varying in difficulty.

Use it when: you are evaluating a research-style assistant that must find, combine, and verify information rather than answer from memory.

Do not infer: a correct final answer does not prove that the system followed a safe, efficient, or reproducible path. An agent can also fail because of search availability, file parsing, answer formatting, or a tool timeout rather than because the model lacked reasoning ability.

Store trajectories and sources, not just answers. Measure unsupported claims, source quality, tool failures, latency, and cost alongside final-answer accuracy.

3. WebArena: browser agents in reproducible websites

WebArena provides self-hosted, functional websites modeled on common domains such as e-commerce, discussion forums, collaborative software development, and content management. Agents receive natural-language goals and operate through a browser; evaluators check the answer or resulting site state.

Use it when: you need repeatable browser-agent experiments involving navigation, forms, search, state tracking, and multi-step actions.

Do not infer: success in WebArena does not guarantee robustness on the live web. Production sites introduce changing layouts, authentication, consent dialogs, localization, rate limits, anti-bot controls, accessibility differences, and irreversible actions.

Separate failures into perception, planning, navigation, tool execution, and validation. Also log whether the agent recognized that the task was complete; unnecessary actions after success can create real-world risk.

4. The τ-bench family: conversation, tools, and policy

The τ-bench project evaluates agents in dynamic conversations with a simulated user, domain-specific tools, and explicit policies. Retail and airline-style tasks test whether an agent gathers required information, makes valid tool calls, and reaches a policy-compliant state. The maintained family has progressed beyond the original τ-bench, so a result must name the exact generation and dataset version.

Use it when: you are building a service agent that must balance a user's request with rules governing refunds, exchanges, bookings, account changes, or other tool-mediated operations.

Do not infer: a simulated user's behavior perfectly represents real customers. Policy wording, hidden database state, tool schema, conversation protocol, and simulator model can all affect difficulty.

Reliability matters as much as one-run success. Repeat the same task with controlled variation and report pass-at-one plus a consistency measure. Track policy violations separately from ordinary task failures; combining them into one percentage can hide unacceptable behavior.

5. ARC-AGI: abstraction and adaptation, with an important distinction

ARC-AGI-2 presents novel grid-transformation tasks. A system studies a few input-output examples, infers an abstract rule, and applies it to unseen cases. This targets few-shot generalization and skill acquisition rather than learned domain knowledge.

ARC-AGI-2 is useful as a reasoning diagnostic, but it is not a conventional agent benchmark: it does not test browser use, customer interaction, external tools, long-running workflow state, or policy compliance. ARC-AGI-3 extends the family into interactive environments where an agent must explore and adapt, making the distinction between benchmark versions essential.

Use it when: you want a controlled signal for novel-task abstraction or, with the interactive version, adaptation under sparse instructions.

Do not infer: success proves production autonomy or broad “AGI.” Report the benchmark version, public or private split, compute and search budget, and cost per task. Large test-time search can change both performance and the meaning of the comparison.

6. OSWorld: operating real desktop applications

OSWorld evaluates multimodal computer-use agents in operating-system environments with real applications, files, and workflows. Tasks can require visual perception, mouse and keyboard control, file operations, and work spanning multiple applications. The current OSWorld 2.0 project uses versioned benchmark releases to improve reproducibility and reduce leakage.

Use it when: you need evidence that an agent can complete desktop work rather than merely describe the steps.

Do not infer: one OSWorld score covers every operating system, application version, display scale, locale, or accessibility configuration. Desktop benchmarks are sensitive to virtual-machine images, network services, screen resolution, timeouts, and the maximum action count.

Report the exact release manifest and provider images. In addition to completion, measure time, step count, unnecessary clicks, recovery from pop-ups, and whether the agent avoids destructive or unauthorized actions.

7. AgentBench: broad coverage across environments

AgentBench evaluates language models as agents across eight distinct environments, covering several categories of interactive tasks rather than one specialized domain.

Use it when: you need an initial breadth check or want to see whether an architecture's strength transfers across different interaction patterns.

Do not infer: an aggregate score means the agent is production-ready. Averages can conceal a catastrophic weakness in a single high-risk environment, and older benchmark components can become less representative as tools and agent interfaces evolve.

Inspect every environment separately. If your product performs database operations but never plays games, the database result is much more decision-relevant than the benchmark-wide average.

How to choose benchmarks for an agent

Start from the consequences of the intended workflow:

• Coding agent: use SWE-bench Verified or a newer repository benchmark, then add private tasks from your languages, frameworks, and CI system.
• Research assistant: use GAIA-style tasks plus source-quality, citation, freshness, and unsupported-claim checks.
• Browser operator: use WebArena, then test your actual sites with reversible sandbox accounts.
• Customer-service agent: use the τ-bench family plus your real policies, tools, escalation rules, and adversarial requests.
• Computer-use agent: use a pinned OSWorld release plus application-specific workflows and safety gates.
• General agent research: combine breadth tests with ARC-style adaptation and domain-specific execution benchmarks.

A general model comparison can provide context, but it should not replace task evaluation. This guide to current large language model families explains why the strongest option depends on the workload.

What every agent benchmark report should disclose

• Exact model, provider endpoint, and model date or version.
• Agent harness, system prompt, tools, and tool descriptions.
• Benchmark release, split, task exclusions, and environment image.
• Maximum actions, tokens, wall-clock time, and tool calls.
• Retries, self-consistency, voting, human intervention, and failure recovery.
• Temperature and other sampling settings.
• Success metric, evaluator version, invalid-task handling, and confidence intervals where possible.
• Average cost, latency, and resource use per attempted and successful task.
• Safety failures, policy violations, destructive actions, and unnecessary side effects.
• Contamination controls and whether benchmark examples may have appeared in training or tuning data.

Build production evaluations around real failure costs

Public benchmarks are useful for comparison and regression testing, but they cannot encode your permissions, data, users, or consequences. A production evaluation set should include normal requests, ambiguous requests, missing information, tool outages, stale data, adversarial instructions, permission boundaries, rollback, and escalation to a human.

Track several outcomes separately: task completion, factual correctness, policy compliance, side effects, recovery, latency, and cost. An agent that succeeds 90% of the time but makes an unauthorized change in 1% of runs may be less deployable than a slower system that safely asks for confirmation.

The right question is therefore not “Which agent has the highest benchmark score?” It is “Which evidence shows that this exact agent configuration can complete our tasks repeatedly, within our safety, quality, latency, and cost limits?”