Humanity’s Last Exam: What the AI Benchmark Measures

Humanity's Last Exam (HLE) is a benchmark of difficult, expert-written academic questions used to evaluate AI models. It was created to provide a more demanding test as leading models achieved high scores on established benchmarks. Its results measure performance on a defined question set, not intelligence as a whole.

What is in the exam?

The finalized HLE set contains 2,500 questions across mathematics, science, humanities, and other specialized fields. It includes multiple-choice and short-answer tasks, with some questions requiring visual understanding. The project was organized by the Center for AI Safety and Scale AI with contributions from subject specialists.

The official project page links to the dataset and research paper and records the finalization of the 2,500-question version. It also identifies a later dynamic fork, HLE-Rolling. These distinctions matter when comparing published scores.

Why make a harder benchmark?

When many systems score near the top of a test, small differences may reveal little about their remaining weaknesses. A harder test gives researchers more room to measure progress on demanding academic questions.

HLE questions are intended to have known, checkable answers while being difficult to solve through a quick web lookup. The research paper explains the evaluation design. Early versions and finalized releases differ, so quote the dataset version whenever reporting a result.

Using model performance to screen candidate questions during construction is different from continuously removing every question a future model answers correctly. If an evaluation changed whenever a model succeeded, scores would no longer measure progress against the same test.

How to read an HLE score

A percentage is meaningful only with its evaluation conditions. Before comparing two results, check:

• Dataset: finalized HLE, a preview, a rolling version, or a subset?
• Model: which exact release and configuration was evaluated?
• Tools: could the system browse, run code, or use other assistance?
• Budget: how much reasoning time or how many attempts were allowed?
• Grading: how were responses matched to reference answers, and were disputed items addressed?

The Scale HLE leaderboard distinguishes the finalized benchmark from its legacy preview. Use a dated, like-for-like result instead of combining older launch scores with newer tool-assisted runs.

What the exam does not establish

A low score shows difficulty with the evaluated questions. It does not prove that a model is useless for ordinary writing, coding, or document work. Likewise, a high score does not prove dependable performance in an unfamiliar workplace, long-running task, or consequential decision.

Human experts typically specialize in a field, while HLE spans many fields. It is therefore misleading to interpret a model's overall percentage as a direct score against an average person or as a complete measure of the gap between humans and AI.

Public benchmarks also need ongoing scrutiny for answer quality and possible exposure during training. Public availability supports inspection and reproducibility; held-out evaluation can provide a separate check against memorization. Neither removes the need to examine methodology.

Use HLE alongside task-specific evidence

For researchers, HLE offers a common reference for difficult academic reasoning and knowledge. For users choosing an AI tool, it is one signal among several. Compare performance on examples from your own work and account for cost, speed, source checking, and review effort.

The TipsMake LLM comparison discusses selection by workload. If you are comparing assistants, also examine their practical tools and access through the Claude and ChatGPT comparison. A benchmark score and a usable product workflow answer different questions.