Methodology · v1.0 · 2026-10-30

The methodology in detail.

One score summarizes demonstrated capability across published reasoning settings and seven equally weighted domains. Related benchmark variants share a family budget, and influence from one evaluator is capped.

Evidence release tli-2026-v1.0-2026-10-30-refresh-47. Browse dated releases.

99 admitted benchmarks across 7 domains. The scale is anchored once to mean 100 and SD 15 across 18 reference models. The reference average stays fixed; 100 is neither human IQ nor the average of current models.

Evidence you can inspect

Results, source citations, review decisions and compatible reasoning modes are stored in a versioned relational database. Each publication uses a sealed, checksum-verified export; CSV files are optional exports, not editable scoring inputs. Previous revisions remain available for audit. Collected means a numeric result exists in the database. Used means its source, exact model checkpoint and benchmark protocol have been reviewed and its benchmark has a frozen calibration supported by multiple providers. For each benchmark we select the highest published aggregate across comparable reasoning modes, not necessarily maximum effort, and count it once. More tested settings provide more chances to record a high score; this selection advantage is not corrected by the conditional interval. Developer-reported and sole/unspecified settings remain eligible and disclosed. Unresolved does not mean false or useless: conflicting metrics, unverified citations and unmatched versions stay collected until reconciled.

Seven capability domains

Each of seven capability domains has an equal, fixed importance budget. These weights define the breadth valued by the index; they are not objectively optimal or learned importance weights. Missing domain abilities are conditional estimates, explicitly marked as unmeasured; individual missing benchmark results are never invented. Alternative weights and domain groupings are sensitivity checks, not replacements chosen to favor a model.

Weight and grouping sensitivity

This release checks 35 alternatives: halving or doubling each domain’s weight, and combining every pair of domains into one equally budgeted group. The largest rank movement relative to the equal-weight diagnostic is 5 places. These checks hold the published domain estimates fixed; they do not reassign benchmarks or refit the model. Rounded domain values can differ from published tie ordering. These are sensitivity scenarios, not confidence ranges or a search for preferred weights. Published scores and ranks are unchanged.

One score and hard rank

Models are ranked by the displayed point score once they have four contributing benchmarks across two domains. Ties use stable model-name ordering. Ranks are a current ordering, not a confidence test. Scores are published immediately without a provisional label; early scores can move substantially as evidence arrives. Contributing benchmark counts and measured domain coverage appear beside each score.

Score stability range

The score stability range is a 95% bootstrap score interval conditional on frozen calibration and comparable evidence availability. It resamples benchmark families and evaluators and adds a frozen missing-evidence correction. It excludes selection across reasoning settings, selective publication, calibration changes and new domains. It is not a 95% guarantee about true capability or where a future score will land. Sparse or conflicting evidence can widen the range.

Sensitivity to selective reporting

The separate reporting sensitivity range tests what happens when only the strongest or weakest standardized benchmark results are retained. It has no probability interpretation and is not combined with the score stability range. Stress-test coverage describes recovery of fuller-data scores under these deliberate masks, not real-world coverage or true intelligence. This test does not quantify the separate advantage from trying more reasoning settings.

Scores you can cite

A model’s score and interval depend only on its own accepted evidence and the frozen methodology. Other models’ additions and updates cannot change them. A methodology revision receives a new version and can change every score.

Validation and its limits

Validation includes provider and benchmark-family holdouts, retrospective sparse-to-full score recovery and selective-reporting stress tests. These test prediction and score stability within the benchmark evidence; they do not establish capability on independent real-world tasks. Prospective tracking records dated evidence snapshots for later comparison. Existing models at tracking start are baselines, not observed launches. Independent-task evaluation requires a registered protocol and outcomes excluded from scoring; no completed independent-task validation is claimed. This release checked 1291 withheld benchmark results; normalized prediction error was 4.4% lower than a refitted legacy baseline on the same reviewed evidence. Only 0 first-recorded model arrivals across 0 providers had usable before/after evidence. Those repository dates are not verified launch dates. They do not validate launch-day coverage for every model, including Astra. The main score interval recovered fuller-data scores in 90 of 92 random held-out evidence cases. Selective favorable and unfavorable publication had only 32% and 20% recovery of fuller-data scores in the main bar under these deliberate stress masks; this is not an estimate of real-world coverage. Reporting sensitivity is shown separately without a probability claim. 0 historical arrivals required a conservative fallback because their earlier reference data were insufficient to train the correction. Those replays do not validate the tighter empirical bands. Calibration needs reference models from at least three providers; a narrow bar is never forced by a width cap.

Value comparisons

The buying ladder compares current point scores with documented costs. Small score differences do not establish a reliable capability advantage, and interval overlap is not used as a significance test. The Value Pick is a shortlist starting point under the declared score floor, not a universal best buy. Capability combines the best published reasoning settings across benchmarks; observed cost describes one measured configuration and workload. Their combination is not a promise of that capability at that cost. Probability-of-superiority claims are not assessed.

Collected benchmarks and scoring evidence

The Contributing column on the score page shows the number of results used to calculate each score. The disclosures below show collected totals, independent family counts and reporting sensitivity. Unresolved source records, excluded composites and incompatible evaluations remain in the collected total. Related benchmarks can belong to the same family, so counts are not independent sample sizes.

The disclosures below list every collected result, whether it contributes, and its review status. An unresolved result awaits review; it has not necessarily been found incorrect or unhelpful. Collected totals alone do not establish scoring coverage or certainty.

Coverage and configuration

Use the highest published benchmark score across reasoning modes for the same model checkpoint and evaluation setup. A lower effort can win; each benchmark still contributes only once. A sole published setting with unspecified effort stays labelled as such. Retry budgets, tools, fallback models, task versions and harness differences are separate comparison questions and are not treated as reasoning modes.

This measures demonstrated capability across settings, not performance at one fixed effort or cost. We select each setting’s published aggregate, not its luckiest individual run. Models tested at more settings have more opportunities to record a high result. The main interval does not correct for that selection advantage; the chosen configuration and source are disclosed below.

A model receives a hard rank once it has 4 contributing benchmarks across 2 domains. Below that floor its score and uncertainty remain visible. No score is labelled provisional.

Cost and speed do not enter the capability score. Benchmark counts describe available evidence, not an equal number of independent experiments. More results usually improve certainty; conflicting results can increase uncertainty. A score assembled from the best mode on each benchmark is not a promise of that performance at the cost of one fixed mode.

View the leaderboard →

Model evidence and uncertainty

Opus 5.5 · 28 used / 48 collected

25 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±5.6 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

SWE-bench Pro: 89.9% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 175 §8.2; retain the reported metric/version and reasoning setting.
SWE-bench Multilingual ; Opus 5.5 release: 93.9% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 175 §8.2; retain the reported metric/version and reasoning setting.
SWE-bench Multimodal ; Opus 5.5 release: 61.4% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 175 §8.2; retain the reported metric/version and reasoning setting.
DeepSWE v1.1: 74.2% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 175 §8.3; retain the reported metric/version and reasoning setting.
FrontierCode v1.1 Main ; Anthropic H2H: 54.6% · Not used. Source · Evaluator-reported · Opus 5.5 (medium); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 176 §8.4; retain the reported metric/version and reasoning setting.
FrontierCode v1.1 Extended: 65.3% · Contributes. Source · Evaluator-reported · Opus 5.5 (medium); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 176 §8.4; retain the reported metric/version and reasoning setting.
Terminal-Bench 4.0: 66.36% · Contributes. Source · Evaluator-reported · Opus 5.5 (xhigh); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 178 §8.5; retain the reported metric/version and reasoning setting. xhigh=66.36 (rounded 66.4 in launch); fallback affects 10% of trials.
Terminal-Bench Science 0.1: 58.7% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 178 §8.6; retain the reported metric/version and reasoning setting.
FrontierSWE v2: 62.3% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 179 §8.7; retain the reported metric/version and reasoning setting.
CursorBench 4.0: 57.8% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 179 §8.8; retain the reported metric/version and reasoning setting.
MathArena ; ArXivMath Aug 2026 ; Anthropic no tools: 91.2% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; safeguard classifiers disabled; no fallback
System card p. 181–182 §8.9; retain the reported metric/version and reasoning setting.
MathArena ; ArXivMath Aug 2026 ; Anthropic with tools: 96.9% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; safeguard classifiers disabled; no fallback
System card p. 181–183 §8.9; retain the reported metric/version and reasoning setting.
ProgramBench ; Opus 5.5 filtered test-pass rate: 91.2% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 183 §8.10.1; retain the reported metric/version and reasoning setting. Hidden-test pass rate on 166 filtered tasks, without six-hour timeout; not Vals raw task pass rate.
Humanity's Last Exam (no tools): 64.4% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 174 table8.1.A; 184 §8.11.1; retain the reported metric/version and reasoning setting.
Humanity's Last Exam w/ tools: 67.7% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 174 table8.1.A; 184 §8.11.1; retain the reported metric/version and reasoning setting.
Chartography ; Opus 5.5 no tools: 64.4% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 200 §8.13.1; retain the reported metric/version and reasoning setting.
Chartography ; Opus 5.5 with tools: 89% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 200 §8.13.1; retain the reported metric/version and reasoning setting.
OSWorld 2.0 ; Sep 10 tasks, Anthropic partial credit: 81.8% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 205–206 §8.13.3; retain the reported metric/version and reasoning setting. September 10 task assets plus server-side context management; explicitly incomparable to earlier releases.
OSWorld 2.0 ; Sep 10 tasks, Anthropic strict pass: 48.7% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 205–206 §8.13.3; retain the reported metric/version and reasoning setting. September 10 task assets plus server-side context management; explicitly incomparable to earlier releases.
OfficeQA ; Opus 5.5 release: 78.9% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 208 §8.14.1; retain the reported metric/version and reasoning setting.
OfficeQA Pro †: 67.7% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 208 §8.14.1; retain the reported metric/version and reasoning setting.
Toolathlon-Verified: 77.8% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; none; safeguard-stopped trials counted as failures
System card p. 210–211 §8.14.5; retain the reported metric/version and reasoning setting.
AutomationBench ; Anthropic H2H: 40% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; none; safeguard-stopped trials counted as failures
System card p. 174 table8.1.A; 211–212 §8.14.6; retain the reported metric/version and reasoning setting.
HealthBench Professional ; Opus 5.5 length-adjusted: 65.6% · Not used. Source · Evaluator-reported · Opus 5.5 (max); Anthropic system-card evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. System card p. 213–214 §8.15.2; retain the reported metric/version and reasoning setting. Opus 4.8 grader and length-adjusted metric; 77.1 raw is not substituted.
AA-Briefcase v1.1 (Elo): 1802.99 · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Artificial Analysis independent evaluation, 2026-10-17; fallback not reported
Daily evaluator refresh; replaces 1804.52 recorded 2026-10-15.
GDPval-AA v2.1 (Elo): 1866.27 · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Artificial Analysis independent evaluation, 2026-10-08; fallback not reported
Daily evaluator refresh; replaces 1866.24 recorded 2026-10-06.
AutomationBench-AA: 69.5386812864% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Artificial Analysis independent evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. Exact model and effort in the evaluator chart payload. HLE is text-only/no tools. AutomationBench-AA is guardrail-adjusted objective credit, not raw objective completion or native whole-workflow pass rate.
Terminal-Bench 4.0 ; AA harness: 59.595959596% · Contributes. Source · Evaluator-reported · Opus 5.5 (xhigh); Artificial Analysis independent evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. Exact model and effort in the evaluator chart payload. HLE is text-only/no tools. AutomationBench-AA is guardrail-adjusted objective credit, not raw objective completion or native whole-workflow pass rate.
SciCode: 66.8981481481% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Artificial Analysis independent evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. Exact model and effort in the evaluator chart payload. HLE is text-only/no tools. AutomationBench-AA is guardrail-adjusted objective credit, not raw objective completion or native whole-workflow pass rate.
HLE text-only ; AA harness: 61.3531047266% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Artificial Analysis independent evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. Exact model and effort in the evaluator chart payload. HLE is text-only/no tools. AutomationBench-AA is guardrail-adjusted objective credit, not raw objective completion or native whole-workflow pass rate.
GDP.pdf ; AA harness: 28.8% · Contributes. Source · Evaluator-reported · Opus 5.5 (high); Artificial Analysis independent evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. Exact model and effort in the evaluator chart payload. HLE is text-only/no tools. AutomationBench-AA is guardrail-adjusted objective credit, not raw objective completion or native whole-workflow pass rate.
CritPt: 31.7142857143% · Contributes. Source · Evaluator-reported · Opus 5.5 (xhigh); Artificial Analysis independent evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. Exact model and effort in the evaluator chart payload. HLE is text-only/no tools. AutomationBench-AA is guardrail-adjusted objective credit, not raw objective completion or native whole-workflow pass rate.
AA-Omniscience: 46.4166666667 · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Artificial Analysis independent evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. Exact model and effort in the evaluator chart payload. HLE is text-only/no tools. AutomationBench-AA is guardrail-adjusted objective credit, not raw objective completion or native whole-workflow pass rate.
AA-LCR v1.1: 84.6666666667% · Contributes. Source · Evaluator-reported · Opus 5.5 (xhigh); Artificial Analysis independent evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. Exact model and effort in the evaluator chart payload. HLE is text-only/no tools. AutomationBench-AA is guardrail-adjusted objective credit, not raw objective completion or native whole-workflow pass rate.
MMMU-Pro: 87.6878612717% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Artificial Analysis independent evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. Independent MMMU-Pro aggregate for this effort.
Harvey LAB-AA: 91.243305833% · Not used. Source · Evaluator-reported · Opus 5.5 (xhigh); Artificial Analysis independent evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. Harvey is mean criterion-pass rate, not all-criteria task success.
AA Intelligence Index v4.3.2 (score): 57.6223698103 · Not used. Source · Evaluator-reported · Opus 5.5 (max); Artificial Analysis independent evaluation, September 22 2026; production default fallback to older Claude models
Scored as the shipped Opus 5.5 system. Default fallback can route cyber tasks to Opus 4.8 and biology/frontier-model-development tasks to Opus 5; this is not a pure-model claim. Reference composite only; not scored alongside its components.
Box ; Opus 5.5 Consumer products: 82% · Not used. Source · Evaluator-reported · Opus 5.5 (undisclosed); Box customer evaluation, September 22 2026; undisclosed
Customer-run industry accuracy; sample counts and uncertainty undisclosed. Separate from existing Box overall benchmark; uncalibrated.
Box ; Opus 5.5 Financial services: 75% · Not used. Source · Evaluator-reported · Opus 5.5 (undisclosed); Box customer evaluation, September 22 2026; undisclosed
Customer-run industry accuracy; sample counts and uncertainty undisclosed. Separate from existing Box overall benchmark; uncalibrated.
Box ; Opus 5.5 Technology: 74% · Not used. Source · Evaluator-reported · Opus 5.5 (undisclosed); Box customer evaluation, September 22 2026; undisclosed
Customer-run industry accuracy; sample counts and uncertainty undisclosed. Separate from existing Box overall benchmark; uncalibrated.
BioMysteryBench v1 ; Vals: 79.26% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Vals AI BioMysteryBench harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 79.26 ± 0.98, rank 4/27. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 50.48% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Vals AI Legal Research Bench harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 50.48 ± 3.48, rank 7/82. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 49.8% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Vals AI MedCode harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 49.8 ± 2.27, rank 19/107. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 91.43% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Vals AI MedScribe harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 91.43 ± 1.93, rank 1/109. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 100% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Vals AI ProofBench v1.1 harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 100.0 ± 0.0, rank 2/50. Vals notes some benchmarks may use a different provider or parameters.
Public Benefits Bench ; Vals: 70.64% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Vals AI Public Benefits Bench v1.1 harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 70.64 ± 1.19, rank 4/50. Vals notes some benchmarks may use a different provider or parameters.
SAGE ; Vals: 45.83% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Vals AI SAGE harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 45.83 ± 3.36, rank 37/91. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 90.29% · Contributes. Source · Evaluator-reported · Opus 5.5 (max); Vals AI Vibe Code Bench v1.1 harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 90.29 ± 1.53, rank 5/116. Vals notes some benchmarks may use a different provider or parameters.

Sonnet 5.5 · 19 used / 42 collected

17 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±14.2 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

SWE-bench Pro: 81.3% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row.
SWE-bench Multilingual ; Opus 5.5 release: 90.3% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row.
SWE-bench Multimodal ; Opus 5.5 release: 54.3% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row.
DeepSWE v1.1: 71% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row.
FrontierCode v1.1 Main ; Anthropic H2H: 52.1% · Not used. Source · Evaluator-reported · Sonnet 5.5 (xhigh); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. Cognition ran v1.1 in Claude Code; official mergeability score. xhigh exceeds max. Operator data archive did not yet contain this model.
FrontierCode v1.1 Extended: 64.4% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (xhigh); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. Cognition ran v1.1 in Claude Code; official mergeability score. xhigh exceeds max. Operator data archive did not yet contain this model.
Terminal-Bench 4.0: 70.6% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. Claude Code --bare, no internet; fallback affected 1.5% of trials (1.2% of requests). Mean of five trials per task.
Terminal-Bench Science 0.1: 59.9% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; safeguards enabled; no fallback fired
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row.
FrontierSWE v2: 61.9% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. Measured by Proximal, 34 tasks, mean of five trials per task, as reported by Anthropic.
CursorBench 4.0: 55.5% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. Cursor measured results in its production harness and supplied them to Anthropic; developer-hosted report, not a separately retrieved operator result.
MathArena ; ArXivMath Aug 2026 ; Anthropic no tools: 86.8% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; safeguard classifiers disabled; no fallback
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. August 57-problem set, four attempts per problem; not MathArena agentic leaderboard protocol.
MathArena ; ArXivMath Aug 2026 ; Anthropic with tools: 95.2% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; safeguard classifiers disabled; no fallback
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. August 57-problem set, four attempts per problem; not MathArena agentic leaderboard protocol.
ProgramBench ; Opus 5.5 filtered test-pass rate: 79.7% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. 166 filtered tasks, hidden-test pass rate, no six-hour timeout; not Vals raw task pass rate.
Humanity's Last Exam (no tools): 56.9% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row.
Humanity's Last Exam w/ tools: 64.5% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row.
Chartography ; Opus 5.5 no tools: 61.6% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. 100 tasks, Gemini 3.5 Flash grader, five runs; same Opus 5.5 release protocol.
Chartography ; Opus 5.5 with tools: 90.2% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. 100 tasks, Gemini 3.5 Flash grader, five runs; same Opus 5.5 release protocol.
OSWorld 2.0 ; Sep 10 tasks, Anthropic partial credit: 80.1% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. v2.1 September 10 assets, 108 tasks, server-side compaction; card explicitly says unchanged from Opus 5.5 configuration. Partial credit and strict pass stay separate.
OSWorld 2.0 ; Sep 10 tasks, Anthropic strict pass: 43.5% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. v2.1 September 10 assets, 108 tasks, server-side compaction; card explicitly says unchanged from Opus 5.5 configuration. Partial credit and strict pass stay separate.
OfficeQA ; Opus 5.5 release: 76.9% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row.
OfficeQA Pro †: 65.6% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row.
Harvey LAB-AA: 93.1% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. AA harness, 120 held-out tasks, mean criterion pass. Do not substitute all-criteria task success.
GDPval-AA v2.1 (Elo): 1844 · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. Artificial Analysis independent evaluation as reported by Anthropic. Pre-release serving of launch checkpoint had a structured-output bug, since fixed; do not infer a post-fix score.
AA-Briefcase v1.1 (Elo): 1811 · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. Artificial Analysis independent evaluation as reported by Anthropic. Pre-release serving of launch checkpoint had a structured-output bug, since fixed; do not infer a post-fix score.
Toolathlon-Verified: 77.8% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; none; safeguard-stopped trials counted as failures
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. Pass@1 across 3 trials × 108 tasks; not Pass@3 or Pass³.
AutomationBench ; Anthropic H2H: 44.7% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. Zapier private evaluation set v1.0.6, whole-workflow success with default fallback; separate from public set and AA partial credit. Do not update Opus from the selective refusal-only rerun.
HealthBench Professional ; Opus 5.5 length-adjusted: 69.2% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Same evaluation protocol as the corresponding catalog row. Opus 4.8 grader, length-adjusted metric; 77.1 raw is separate. Same Opus 5.5 baseline and adjustment formula.
BenchCAD ; Anthropic 1000-file Vision2Code no tools: 0.747 · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Mean voxel IoU over five runs; raw 0–1 units, not percent task success. Kept separate from the older modified-settings comparison.
BenchCAD ; Anthropic 1000-file Vision2Code with tools: 0.963 · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Mean voxel IoU over five runs; raw 0–1 units, not percent task success. Kept separate from the older modified-settings comparison.
Harvey LAB ; AA held-out all-pass: 11.7% · Not used. Source · Evaluator-reported · Sonnet 5.5 (high); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. 120 held-out tasks, AA harness, all-criteria success; separate from mean criterion pass and older Harvey harnesses.
HealthBench ; Anthropic Opus 4.8 grader length-adjusted: 65.4% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Exact labels in figure; distinct from raw and OpenAI-graded results.
HealthBench ; Anthropic Opus 4.8 grader raw: 69.4% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Raw score before response-length adjustment; retained separately and uncalibrated.
HealthBench Professional ; Anthropic Opus 4.8 grader raw: 77.1% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. Raw score before response-length adjustment; retained separately and uncalibrated.
PhysicianBench ; Anthropic Opus 5 grader: 63.2% · Not used. Source · Evaluator-reported · Sonnet 5.5 (max); Anthropic September 28 system card; production cyber fallback to Sonnet 5; biology requests may stop
Exact Sonnet 5.5 published aggregate. Shipped-system results can include disclosed Sonnet 5 fallback; no predecessor score is copied. 100 EHR tasks, pass@1, same agent harness and Opus 5 grading; exact figure labels.
BioMysteryBench v1 ; Vals: 81.11% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Vals AI BioMysteryBench harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 81.11 ± 0.64, rank 1/27. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 48.08% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Vals AI Legal Research Bench harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 48.08 ± 3.47, rank 12/82. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 52.92% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Vals AI MedCode harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 52.92 ± 2.12, rank 12/107. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 91.1% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Vals AI MedScribe harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 91.1 ± 1.96, rank 3/109. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 100% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Vals AI ProofBench v1.1 harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 100.0 ± 0.0, rank 1/50. Vals notes some benchmarks may use a different provider or parameters.
Public Benefits Bench ; Vals: 67.19% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Vals AI Public Benefits Bench v1.1 harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 67.19 ± 1.22, rank 15/50. Vals notes some benchmarks may use a different provider or parameters.
SAGE ; Vals: 51.77% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Vals AI SAGE harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 51.77 ± 3.41, rank 11/91. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 92.39% · Contributes. Source · Evaluator-reported · Sonnet 5.5 (max); Vals AI Vibe Code Bench v1.1 harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 92.39 ± 1.26, rank 1/116. Vals notes some benchmarks may use a different provider or parameters.

GPT-6 Astra · 57 used / 87 collected

46 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.2 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1580.2 · Not used. Source · Evaluator-reported · GPT-6 Astra (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 89.88764% · Contributes. Source · Evaluator-reported · GPT-6 Astra (high); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 74.115044% · Contributes. Source · Evaluator-reported · gpt-6-astra; xhigh; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
AA Intelligence Index v4.1.1 (score): 61.2 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 96.262626% · Contributes. Source · Evaluator-reported · GPT-6 Astra (xhigh); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 74.3% · Not used. Source · Evaluator-reported · GPT-6 Astra (max)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 53.54% · Contributes. Source · Evaluator-reported · openai_gpt-6-astra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 71.7% · Contributes. Source · Evaluator-reported · openai_gpt-6-astra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 56.481481% · Contributes. Source · Evaluator-reported · GPT-6 Astra (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 54.680259% · Contributes. Source · Evaluator-reported · GPT-6 Astra (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 39.42% · Contributes. Source · Evaluator-reported · openai_gpt-6-astra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 5.42% · Contributes. Source · Evaluator-reported · openai_gpt-6-astra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1562.01 · Not used. Source · Evaluator-reported · GPT-6 Astra (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 87.91% · Contributes. Source · Evaluator-reported · openai_gpt-6-astra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 48.49% · Contributes. Source · Evaluator-reported · openai_gpt-6-astra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Omniscience: 43.733333 · Contributes. Source · Evaluator-reported · GPT-6 Astra (high); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
SAGE ; Vals: 46.37% · Contributes. Source · Evaluator-reported · openai_gpt-6-astra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Code Migration †: 67.74% · Contributes. Source · Evaluator-reported · openai_gpt-6-astra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Humanity's Last Exam w/ tools: 57.2% · Contributes. Source · Developer-reported · GPT-6 Astra; OpenAI comparison table; maximum across reasoning modes
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProgramBench v1 (Raw Pass Rate) ; Vals: 85.417% · Contributes. Source · Archived source review · openai/gpt-6-astra · reasoning_effort=max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
FrontierMath v2 Tier 4: 97.560976% · Contributes. Source · Evaluator-reported · gpt-6-astra_medium; task version 2.0.0
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
APEX-Agents Mean Criteria Passed: 75.1% · Contributes. Source · Evaluator-reported · gpt-6-astra-max; max; loop_truncated_tools_agent
Same 240-task, 31-world APEX benchmark and mean-score metric. Current published repeated-run aggregate (959 samples); pass@1 is not substituted.
BrowseComp: 91.5% · Contributes. Source · Developer-reported · GPT-6 Astra; OpenAI comparison table; maximum across reasoning modes
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MMMU-Pro: 86.878613% · Contributes. Source · Evaluator-reported · GPT-6 Astra (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
ARC-AGI-2: 95% · Contributes. Source · Evaluator-reported · GPT-6 Astra; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
ExploitBench: 100% · Not used.
Inherited comparator metric/scaffold equivalence to Astra Cap Percent is not established; the source flags possible contamination. Excluded pending protocol audit.
FrontierCode v1.1 Extended: 64.48% · Contributes. Source · Evaluator-reported · GPT-6 Astra; max; codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierCode v1.1 Main ; Anthropic H2H: 53.3% · Not used.
Not admitted after the September 6 model review: the collected 53.3 for GPT-6 Astra on FrontierCode v1.1 Main ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AutomationBench ; Anthropic H2H: 41.4% · Not used.
Not admitted after the September 6 model review: the collected 41.4 for GPT-6 Astra on AutomationBench ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 43.092784% · Contributes. Source · Evaluator-reported · GPT-6 Astra (xhigh); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 31.714286% · Contributes. Source · Evaluator-reported · GPT-6 Astra (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
Terminal-Bench 4.0: 58.2% · Contributes. Source · Evaluator-reported · GPT-6 Astra; max; Codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 99% · Contributes. Source · Evaluator-reported · openai_gpt-6-astra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 89.59% · Contributes. Source · Evaluator-reported · openai_gpt-6-astra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ARC-AGI-1: 98.5% · Contributes. Source · Evaluator-reported · GPT-6 Astra; XHigh; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
ARC-AGI-3 (standard harness): 62.71% · Contributes. Source · Evaluator-reported · GPT-6 Astra; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
BenchCAD (Vision2Code subset): 95.9% · Not used. Source · Developer-reported · OpenAI full Vision2Code set; not Anthropic subset
System card identifies the OpenAI comparison as the full 17,900-file set, while this row is a modified 1000-file subset. Requires a separate protocol row.
Terminal-Bench Science 0.1: 64.6% · Contributes. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
SRE-Bench (pass@1): 88% · Contributes. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
OSWorld 2.0 ; OpenAI H2H (offline subset): 72.6% · Contributes. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Agents' Last Exam ; OpenAI H2H: 59.3% · Contributes. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
ScreenSpot-Pro: 92.7% · Not used. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
HealthBench: 59.7% · Not used. Source · Developer-reported · GPT-6 Astra; OpenAI published HealthBench research evaluation; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
HealthBench Hard: 37.8% · Not used. Source · Developer-reported · GPT-6 Astra; OpenAI published HealthBench research evaluation; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
HealthBench Professional (length-adjusted): 63.4% · Contributes. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
GeneBench Pro: 37.1% · Not used. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
LifeSciBench: 60.3% · Not used. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
SEC-Bench Pro: 85.4% · Not used. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
ExploitGym: 42.4% · Not used. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
AA Coding Agent Index v1.4 (score): 67 · Not used.
Held-out composite, never a scoring input
Internal Database Migration Tasks ; OpenAI: 63.9% · Contributes. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Internal Design Tasks ; OpenAI: 50% · Contributes. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Internal Data Science Tasks ; OpenAI: 40.9% · Contributes. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
OpenScore String Quartets (1 - OMR-NED): 0.84 · Not used. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
MedChemBench (internal): 49.3% · Not used. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
MRCR v2 (8 needles, 512K-1M): 96.3% · Not used. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
ExploitBench (Jun-Aug 2026, 300-turn limit): 39% · Not used. Source · Developer-reported · GPT-6 Astra; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Mystery Game Puzzles v1.0.4 ; Epoch: 84% · Contributes. Source · Evaluator-reported · gpt-6-astra_max; task version 1.0.4
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
WANDR ; Perplexity: 0.682 · Not used.
Exact reported metric and evaluation configuration unresolved.
FrontierMath Erdos (68 conjectures) ; Epoch: 2.941176 · Not used. Source · Evaluator-reported · gpt-6-astra_max; task version
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
Box Complex Work Eval (overall): 77% · Not used.
Not admitted after the September 6 model review: the collected 77 for GPT-6 Astra on Box Complex Work Eval (overall) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCyber (Sep 2026) ; Irregular: 86 · Not used.
Not admitted after the September 6 model review: the collected 86 for GPT-6 Astra on FrontierCyber (Sep 2026) ; Irregular still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CyScenarioBench (Sep 2026) ; Irregular: 59% · Not used.
Not admitted after the September 6 model review: the collected 59 for GPT-6 Astra on CyScenarioBench (Sep 2026) ; Irregular still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
BioMysteryBench v1 ; Vals: 79.26% · Contributes. Source · Evaluator-reported · openai_gpt-6-astra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Tax Agent Bench v1 ; Vals: 63.34% · Contributes. Source · Evaluator-reported · openai_gpt-6-astra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
RuneBench (30m, mean ln(1 + XP/min)): 7.258225 · Contributes. Source · Archived source review · gpt6astra-high
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 2714 · Contributes. Source · Archived source review · GPT-6 Astra (Max)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Roboflow Vision Evals (six-task mean, high tier): 88.533333% · Contributes. Source · Archived source review · Require a complete six-task high-tier panel, the highest published benchmark tier. Retain high-tier scores even when lower than low-tier. Do not substitute the best of three repetitions: use the published mean. Each panel enters the fit once; do not also fit its components.
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1797 · Contributes. Source · Archived source review · gpt-6-astra-max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA-AnalystAgent: 51.25% · Not used. Source · Evaluator-reported · GPT-6 Astra (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-AnalystAgent, so every model is scored on its current set.
AA-LCR v1.1: 80.666667% · Contributes. Source · Evaluator-reported · GPT-6 Astra (max)
AA v4.3 evaluation, kept separate from prior benchmark versions and other agent harnesses.
Agent Arena overall IPS: 0.125451 · Not used. Source · Evaluator-reported · GPT-6 Astra (Max); Agent Arena 2026-09-08
Native inverse-propensity-score metric. Retained as a separate uncalibrated benchmark, not converted to an accuracy percentage or Text Arena Elo.
Astra-26 LiveBrowseComp ; native web: 42.307692% · Not used. Source · Evaluator-reported · GPT-6 Astra medium; Codex CLI 0.153.4; built-in web
Native web reference configuration only. Exa, Keenable, Parallel, and Valyu are different tool stacks, not interchangeable reasoning modes. One run per task; not the existing BrowseComp benchmark.
AutomationBench-AA: 68.491748% · Contributes. Source · Evaluator-reported · GPT-6 Astra (max)
AA v4.3 evaluation, kept separate from prior benchmark versions and other agent harnesses.
Creative Writing v3: 2163.9 · Contributes. Source · Archived source review · gpt-6-astra; published evaluation, effort unspecified
Star denotes community-submitted evaluation hosted on the official benchmark board; not independently rerun by the benchmark author.
GDP.pdf ; AA harness: 32.2% · Contributes. Source · Evaluator-reported · GPT-6 Astra (xhigh)
AA v4.3 evaluation, kept separate from prior benchmark versions and other agent harnesses.
LiveBench: 82.160643% · Contributes. Source · Evaluator-reported · gpt-6-astra-max; LiveBench 2026-06-25 task set
Derived exactly using the existing category mapping and evaluator-published task scores; not a new benchmark version.
MathArena ; ArXivLean Jun 2026: 63.83% · Contributes. Source · Evaluator-reported · GPT-6 Astra (max); MathArena June 2026 competition harness
Exact checkpoint, metric and published aggregate matched to the evaluator source. Date is retrieval date; first publication date is not inferred.
MathArena ; ArXivMath Jun 2026: 94.44% · Contributes. Source · Evaluator-reported · GPT-6 Astra (max); MathArena named monthly competition
Exact published two-decimal accuracy from the direct model comparison. Month is kept explicit; overall values are not mixed with month-level rows.
MathArena ; ArXivMath May 2026: 95% · Contributes. Source · Evaluator-reported · GPT-6 Astra (max); MathArena named monthly competition
Exact published two-decimal accuracy from the direct model comparison. Month is kept explicit; overall values are not mixed with month-level rows.
MathArena ; BrokenArXiv Jun 2026: 99.07% · Contributes. Source · Evaluator-reported · GPT-6 Astra (max); MathArena named monthly competition
Exact published two-decimal accuracy from the direct model comparison. Month is kept explicit; overall values are not mixed with month-level rows.
MLCR-AA (Medical Long Context Reasoning): 35% · Not used. Source · Evaluator-reported · GPT-6 Astra (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
Terminal-Bench 4.0 ; AA harness: 59.59596% · Contributes. Source · Evaluator-reported · GPT-6 Astra (xhigh)
AA v4.3 evaluation, kept separate from prior benchmark versions and other agent harnesses.
AA Intelligence Index v4.3.2 (score): 52.673669 · Not used. Source · Evaluator-reported · GPT-6 Astra (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Restored on 2026-10-28: 56.853856 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1568.98 · Contributes. Source · Evaluator-reported · GPT-6 Astra (max); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1568.14 recorded 2026-10-27.
GDPval-AA v2.1 (Elo): 1584.4 · Contributes. Source · Evaluator-reported · GPT-6 Astra (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1571.76 recorded 2026-10-21.
HLE text-only ; AA harness: 54.680259% · Contributes. Source · Evaluator-reported · GPT-6 Astra (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

Gemini 4 Carbon · 7 used / 7 collected

7 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±23.6 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1610.19 · Contributes. Source · Evaluator-reported · Gemini 4 Carbon (high); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1609.49 recorded 2026-10-28.
AutomationBench-AA: 77.8166057262% · Contributes. Source · Evaluator-reported · Gemini 4 Carbon (high); Artificial Analysis independent evaluation, 2026-10-28; fallback not reported
Exact model and effort in the evaluator chart payload.
Terminal-Bench 4.0 ; AA harness: 65.1515151515% · Contributes. Source · Evaluator-reported · Gemini 4 Carbon (high); Artificial Analysis independent evaluation, 2026-10-28; fallback not reported
Exact model and effort in the evaluator chart payload.
GDP.pdf ; AA harness: 28.4% · Contributes. Source · Evaluator-reported · Gemini 4 Carbon (high); Artificial Analysis independent evaluation, 2026-10-28; fallback not reported
Exact model and effort in the evaluator chart payload.
Vending-Bench 2: 16484 · Contributes. Source · Evaluator-reported · Gemini 4 Carbon (effort not stated); Gemini 4 Carbon developer-reported result (google); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (fable-5-1 5422). Value read from an image of the lab table. Printed as $16,484 (final bank balance in USD). Table on the Google launch post; methodology at deepmind.google/models/evals-methodology/gemini-4-carbon. Effort not stated.
Legal Research Bench ; Vals: 55.77% · Contributes. Source · Evaluator-reported · Gemini 4 Carbon (high); Vals AI Legal Research Bench harness; provider Google; fallback not reported
Vals AI model page, headline accuracy 55.77 ± 3.45, rank 1/87. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 89.06% · Contributes. Source · Evaluator-reported · Gemini 4 Carbon (high); Vals AI Vibe Code Bench v1.1 harness; provider Google; fallback not reported
Vals AI model page, headline accuracy 89.06 ± 2.29, rank 8/121. Vals notes some benchmarks may use a different provider or parameters.

Gemini 4 Argon · 26 used / 27 collected

25 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±5.6 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1484.44 · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1483.28 recorded 2026-10-27.
GDPval-AA v2.1 (Elo): 1631.09 · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1623.11 recorded 2026-10-20.
AutomationBench-AA: 77.513469459% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
Terminal-Bench 4.0 ; AA harness: 57.0707070707% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
SciCode: 61.8055555556% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
HLE text-only ; AA harness: 57.0898980538% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
GDP.pdf ; AA harness: 21.8% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
CritPt: 27.1428571429% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
AA-Omniscience: 42.35 · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Restored on 2026-10-28: 46.1 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. Exact model and effort in the evaluator chart payload.
AA-LCR v1.1: 79.6666666667% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
AA Intelligence Index v4.3.2 (score): 52.5605655746 · Not used. Source · Evaluator-reported · Gemini 4 Argon (high); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Restored on 2026-10-28: 50.412194 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Reference composite only, never scored alongside its component evaluations.
DeepSWE v1.1: 77.9% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (effort not stated); Google Gemini 4 Argon launch benchmark table; methodology at deepmind.google/models/evals-methodology/gemini-4-argon; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (gpt-6-astra 74.1, opus-5-5 74.2, fable-5-1 67.4). Effort not stated.
Vibe Code Bench v1.1 ; Vals: 91.9% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (effort not stated); Google Gemini 4 Argon launch benchmark table; methodology at deepmind.google/models/evals-methodology/gemini-4-argon; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (gpt-6-astra 89.6, fable-5-1 90.3). Effort not stated. Table does not name the version; comparison columns match v1.1.
Terminal-Bench 4.0: 57.4% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (effort not stated); Google Gemini 4 Argon launch benchmark table; methodology at deepmind.google/models/evals-methodology/gemini-4-argon; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (gpt-6-astra 58.2, fable-5-1 57.9, opus-5-5 66.4). Effort not stated.
Finance Agent v2 ; Vals: 65.4% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (effort not stated); Google Gemini 4 Argon launch benchmark table; methodology at deepmind.google/models/evals-methodology/gemini-4-argon; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (gpt-6-astra 53.5, fable-5-1 58.9). Effort not stated.
Harvey LAB (Vals): 19.6% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (effort not stated); Google Gemini 4 Argon launch benchmark table; methodology at deepmind.google/models/evals-methodology/gemini-4-argon; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (gpt-6-astra 5.4, fable-5-1 6.7). Effort not stated.
OSWorld 2.0 ; OpenAI H2H (offline subset): 69.2% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (effort not stated); Google Gemini 4 Argon launch benchmark table; methodology at deepmind.google/models/evals-methodology/gemini-4-argon; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (gpt-6-astra 72.6). Effort not stated. Offline subset, partial score, as labelled in the table.
BioMysteryBench v1 ; Vals: 76.3% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Vals AI BioMysteryBench harness; provider Google; fallback not reported
Vals AI model page, headline accuracy 76.3 ± 0.37, rank 6/27. Vals notes some benchmarks may use a different provider or parameters.
Code Migration †: 68.17% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Vals AI Code Migration harness; provider Google; fallback not reported
Vals AI model page, headline accuracy 68.17 ± 4.35, rank 2/77. Vals notes some benchmarks may use a different provider or parameters.
Excel Modeling Benchmark ; Vals: 75.24% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Vals AI EMB harness; provider Google; fallback not reported
Vals AI model page, headline accuracy 75.24 ± 2.46, rank 4/74. Vals notes some benchmarks may use a different provider or parameters.
LegalBench ; Vals: 88.3% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Vals AI LegalBench harness; provider Google; fallback not reported
Vals AI model page, headline accuracy 88.3 ± 0.37, rank 3/151. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 54.81% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Vals AI Legal Research Bench harness; provider Google; fallback not reported
Vals AI model page, headline accuracy 54.81 ± 3.46, rank 4/78. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 58.8% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Vals AI MedCode harness; provider Google; fallback not reported
Vals AI model page, headline accuracy 58.8 ± 2.1, rank 3/107. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 87.43% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Vals AI MedScribe harness; provider Google; fallback not reported
Vals AI model page, headline accuracy 87.43 ± 1.96, rank 17/109. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 99% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Vals AI ProofBench v1.1 harness; provider Google; fallback not reported
Vals AI model page, headline accuracy 99.0 ± 1.0, rank 6/50. Vals notes some benchmarks may use a different provider or parameters.
Public Benefits Bench ; Vals: 69.76% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Vals AI Public Benefits Bench v1.1 harness; provider Google; fallback not reported
Vals AI model page, headline accuracy 69.76 ± 0.0, rank 7/50. Vals notes some benchmarks may use a different provider or parameters.
SAGE ; Vals: 53.65% · Contributes. Source · Evaluator-reported · Gemini 4 Argon (high); Vals AI SAGE harness; provider Google; fallback not reported
Vals AI model page, headline accuracy 53.65 ± 3.38, rank 4/91. Vals notes some benchmarks may use a different provider or parameters.

Muse Wonder Full · 23 used / 24 collected

23 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±7.4 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1729.32 · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
GDPval-AA v2.1 (Elo): 1857.09 · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
AutomationBench-AA: 71.3307497322% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
Terminal-Bench 4.0 ; AA harness: 57.0707070707% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
SciCode: 62.6157407407% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
HLE text-only ; AA harness: 53.6144578313% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
GDP.pdf ; AA harness: 32.6% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
CritPt: 32.8571428571% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
AA-Omniscience: 30.15 · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Restored on 2026-10-28: 30.6 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. Exact model and effort in the evaluator chart payload.
AA-LCR v1.1: 83.6666666667% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
AA Intelligence Index v4.3.2 (score): 55.8021199856 · Not used. Source · Evaluator-reported · Muse Wonder Full (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Restored on 2026-10-28: 49.727212 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Reference composite only, never scored alongside its component evaluations.
DeepSWE v1.1: 79% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Muse Wonder Full developer-reported result (meta); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (opus-5-5 74.2, fable-5-1 67.4). Value read from an image of the lab table. Value read from the scorecard image embedded on page 3 of the PDF model card. Per section 3, the Muse Wonder Full DeepSWE v1.1 result is Datacurve third-party (pass@1).
Finance Agent v2 ; Vals: 59.1% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Muse Wonder Full developer-reported result (meta); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (muse-wonder 56.6, fable-5-1 58.9). Value read from an image of the lab table. Value read from the scorecard image embedded on page 3 of the PDF model card. Per section 3, the result source is Vals AI and the metric is private-test-set accuracy.
BioMysteryBench v1 ; Vals: 73.33% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Vals AI BioMysteryBench harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 73.33 ± 0.64, rank 8/27. Vals notes some benchmarks may use a different provider or parameters.
Code Migration †: 61.03% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Vals AI Code Migration harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 61.03 ± 4.42, rank 7/77. Vals notes some benchmarks may use a different provider or parameters.
Excel Modeling Benchmark ; Vals: 71.92% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Vals AI EMB harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 71.92 ± 2.66, rank 9/74. Vals notes some benchmarks may use a different provider or parameters.
Harvey LAB (Vals): 21.67% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Vals AI Harvey's Legal Agent Benchmark harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 21.67 ± 3.54, rank 4/79. Vals notes some benchmarks may use a different provider or parameters.
LegalBench ; Vals: 87.42% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Vals AI LegalBench harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 87.42 ± 0.41, rank 4/151. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 53.37% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Vals AI Legal Research Bench harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 53.37 ± 3.47, rank 5/78. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 49.93% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Vals AI MedCode harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 49.93 ± 2.18, rank 18/107. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 89.33% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Vals AI MedScribe harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 89.33 ± 2.0, rank 8/109. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 94% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Vals AI ProofBench v1.1 harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 94.0 ± 2.39, rank 9/50. Vals notes some benchmarks may use a different provider or parameters.
Public Benefits Bench ; Vals: 73.75% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Vals AI Public Benefits Bench v1.1 harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 73.75 ± 1.15, rank 3/50. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 86% · Contributes. Source · Evaluator-reported · Muse Wonder Full (max); Vals AI Vibe Code Bench v1.1 harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 86.0 ± 1.93, rank 12/112. Vals notes some benchmarks may use a different provider or parameters.

Kimi K3.1 Pro · 18 used / 19 collected

17 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±14.2 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1677.79 · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (none); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
GDPval-AA v2.1 (Elo): 1845.72 · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (none); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
AutomationBench-AA: 73.2977109092% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (none); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
Terminal-Bench 4.0 ; AA harness: 67.1717171717% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (none); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
SciCode: 60.7638888889% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (none); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
HLE text-only ; AA harness: 52.4559777572% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (none); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
GDP.pdf ; AA harness: 31.4% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (none); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
CritPt: 24.5714285714% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (none); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
AA-Omniscience: 29.5166666667 · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (none); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Restored on 2026-10-28: 29.866667 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. Exact model and effort in the evaluator chart payload.
AA-LCR v1.1: 86.3333333333% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (none); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
MMMU-Pro: 83.8728323699% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (none); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Independent MMMU-Pro aggregate for this effort.
AA Intelligence Index v4.3.2 (score): 55.0869210959 · Not used. Source · Evaluator-reported · Kimi K3.1 Pro (none); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Restored on 2026-10-28: 46.002624 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Reference composite only, never scored alongside its component evaluations.
Terminal-Bench 4.0: 65.7% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (max); Kimi K3.1 Pro developer-reported result (moonshotai); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (gpt-6-astra 58.2, opus-5-5 66.4). Claude-Code harness (Footnotes, Terminal-Bench 4.0).
BrowseComp: 92.1% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (max); Kimi K3.1 Pro developer-reported result (moonshotai); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (gpt-6-astra 91.5). Context compaction at 200K tokens, max 500 steps, temperature 1.0, top-p 1.0 (Footnotes, BrowseComp).
BabyVision (with CI): 95.4% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (max); Kimi K3.1 Pro developer-reported result (moonshotai); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (deepseek-v4-1-flash-1017 92.5). Lab labels the row BabyVision w/ python; averaged over four runs (Footnotes, multimodal benchmarks w/ python).
BioMysteryBench v1 ; Vals: 75.56% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (max); Vals AI BioMysteryBench harness; provider Moonshot AI; fallback not reported
Vals AI model page, headline accuracy 75.56 ± 0.64, rank 7/29. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 37.5% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (max); Vals AI Legal Research Bench harness; provider Moonshot AI; fallback not reported
Vals AI model page, headline accuracy 37.5 ± 3.37, rank 42/86. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 91% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (max); Vals AI ProofBench v1.1 harness; provider Moonshot AI; fallback not reported
Vals AI model page, headline accuracy 91.0 ± 2.88, rank 10/52. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 84.89% · Contributes. Source · Evaluator-reported · Kimi K3.1 Pro (max); Vals AI Vibe Code Bench v1.1 harness; provider Moonshot AI; fallback not reported
Vals AI model page, headline accuracy 84.89 ± 2.95, rank 17/120. Vals notes some benchmarks may use a different provider or parameters.

Fable 5.1 · 40 used / 74 collected

36 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.4 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1763.64 · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
Source still labels this model Default Fallback. Retained for completeness, excluded from single-model scoring.
Terminal-Bench 2.1: 91.4% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
DeepSWE v1.1: 67.4% · Contributes. Source · Developer-reported · Fable 5.1 final snapshot; adaptive max; five trials
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LMArena (Chatbot Arena) ; Text: 1504.213301 · Contributes. Source · Evaluator-reported · claude-fable-5.1-max; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 66 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 93.74% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
AA-LCR: 80% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 58.88% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 76.67% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 63.078704% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
Source still labels this model Default Fallback. Retained for completeness, excluded from single-model scoring.
Humanity's Last Exam (no tools): 59.128823% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
Source still labels this model Default Fallback. Retained for completeness, excluded from single-model scoring.
Legal Research Bench ; Vals: 55.29% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 6.67% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1694 · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
MedScribe ; Vals: 91.29% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 88.51% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 75.96% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Public Benefits Bench ; Vals: 74.9% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 53.51% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Omniscience: 43.45 · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
Source still labels this model Default Fallback. Retained for completeness, excluded from single-model scoring.
MMLU Pro ; Vals: 92.38% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 83.414155% · Contributes. Source · Evaluator-reported · claude-fable-5-1-max-effort; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
MortgageTax ; Vals: 70.79% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SAGE ; Vals: 48.53% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Toolathlon-Verified: 77.8% · Not used.
Not admitted after the September 6 model review: the collected 77.8 for Fable 5.1 on Toolathlon-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
LiveCodeBench: 90.52% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Humanity's Last Exam w/ tools: 65% · Contributes. Source · Developer-reported · Fable 5.1; OpenAI comparison table; maximum across reasoning modes
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProgramBench v1 (Raw Pass Rate) ; Vals: 82.688% · Contributes. Source · Archived source review · anthropic/claude-fable-5-1 · compute=max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
FrontierSWE: 57% · Not used.
Not admitted after the September 6 model review: the collected 57 for Fable 5.1 on FrontierSWE still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierMath v2 Tier 4: 87.804878% · Contributes. Source · Evaluator-reported · claude-fable-5-1_max; task version 2.0.0
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
CursorBench v3.2: 73.4% · Contributes. Source · Evaluator-reported · Fable 5.1 Max; Cursor harness
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
APEX-Agents Mean Criteria Passed: 77.1% · Contributes. Source · Evaluator-reported · claude-fable-5.1; max; loop_truncated_tools_agent
Same 240-task, 31-world APEX benchmark and mean-score metric. Current published repeated-run aggregate (959 samples); pass@1 is not substituted.
SWE-bench Pro: 81.2% · Not used.
Not admitted after the September 6 model review: the collected 81.2 for Fable 5.1 on SWE-bench Pro still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
APEX-SWE (Pass@1, Terminus-2): 63.6% · Contributes. Source · Evaluator-reported · claude-fable-5.1; max; terminus-2
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MMMU-Pro: 90.64% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
ARC-AGI-2: 90% · Contributes. Source · Evaluator-reported · Fable 5.1; XHigh; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
OSWorld 2.0 ; Anthropic H2H: 77.9% · Not used.
Not admitted after the September 6 model review: the collected 77.9 for Fable 5.1 on OSWorld 2.0 ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Extended: 63.6% · Contributes. Source · Evaluator-reported · Claude Fable 5.1; medium; claude-code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MathArena ; ArXivMath Jun 2026: 90.97% · Contributes. Source · Evaluator-reported · Fable 5.1 (max); MathArena named monthly competition
Exact published two-decimal accuracy from the direct model comparison. Month is kept explicit; overall values are not mixed with month-level rows.
OfficeQA Pro †: 69% · Not used.
Not admitted after the September 6 model review: the collected 69 for Fable 5.1 on OfficeQA Pro † still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Main ; Anthropic H2H: 50.9% · Not used.
Not admitted after the September 6 model review: the collected 50.9 for Fable 5.1 on FrontierCode v1.1 Main ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AutomationBench ; Anthropic H2H: 31.4% · Not used.
Not admitted after the September 6 model review: the collected 31.4 for Fable 5.1 on AutomationBench ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Harvey Legal Agent Benchmark ; held-out: 16.7% · Not used.
Not admitted after the September 6 model review: the collected 16.7 for Fable 5.1 on Harvey Legal Agent Benchmark ; held-out still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 47.2% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
CritPt: 29.7% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
Terminal-Bench 4.0: 57.9% · Contributes. Source · Evaluator-reported · Fable 5.1; max; Claude Code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 100% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 90.26% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5-1; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MineBench: 2050 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
ARC-AGI-1: 97.5% · Contributes. Source · Evaluator-reported · Fable 5.1; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
BenchCAD (Vision2Code subset): 84.3% · Not used. Source · Developer-reported · Fable 5.1; max; with tools; 1000-file subset; three disclosed reference modifications
Matched 1000-file Vision2Code subset and disclosed modified grading. Full-set OpenAI results cannot be silently compared as this subset.
Terminal-Bench Science 0.1: 52.6% · Contributes. Source · Developer-reported · Fable 5.1; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Harvey LAB-AA: 93.02% · Not used.
Not admitted after the September 6 model review: the collected 93.02 for Fable 5.1 on Harvey LAB-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vending-Bench 2: 5421.556667 · Contributes. Source · Evaluator-reported · Claude Fable 5.1; Andon Labs Vending-Bench 2
Compare reasoning configurations using their mean over runs. Geometric mean, individual best runs, Fireworks Kimi endpoint, Vending Arena, and April DeepSeek Pro are not substituted.
Internal Database Migration Tasks ; OpenAI: 57.8% · Contributes. Source · Developer-reported · Fable 5.1; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Mystery Game Puzzles v1.0.4 ; Epoch: 58% · Contributes. Source · Evaluator-reported · claude-fable-5-1_max; task version 1.0.4
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
FrontierMath Erdos (68 conjectures) ; Epoch: 0 · Not used. Source · Evaluator-reported · claude-fable-5-1_max; task version
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
RuneBench (30m, mean ln(1 + XP/min)): 6.143277 · Contributes. Source · Archived source review · fable51-xhigh
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 2245 · Contributes. Source · Archived source review · Claude Fable 5.1
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Roboflow Vision Evals (six-task mean, high tier): 82.516667% · Contributes. Source · Archived source review · Require a complete six-task high-tier panel, the highest published benchmark tier. Retain high-tier scores even when lower than low-tier. Do not substitute the best of three repetitions: use the published mean. Each panel enters the fit once; do not also fit its components.
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1762 · Contributes. Source · Archived source review · claude-fable-5.1-max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA-AnalystAgent: 57.5% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
Source still labels this model Default Fallback. Retained for completeness, excluded from single-model scoring.
AA-LCR v1.1: 85.333333% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
Distinct AA protocol/version; not pooled with other harnesses. Fable Default Fallback scores are retained but not attributed to a single checkpoint.
Agent Arena overall IPS: 0.145069 · Not used. Source · Evaluator-reported · Fable 5.1 (Max); Agent Arena 2026-09-08
Native inverse-propensity-score metric. Retained as a separate uncalibrated benchmark, not converted to an accuracy percentage or Text Arena Elo.
AutomationBench-AA: 59.375917% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
Distinct AA protocol/version; not pooled with other harnesses. Fable Default Fallback scores are retained but not attributed to a single checkpoint.
Code Migration †: 54.607% · Not used. Source · Evaluator-reported · Claude Fable 5.1; max; Vals published harness
Newly located exact Vals overall result. Retained as unresolved because Fable model page discloses fallback routing and benchmark-level impact has not been established.
Creative Writing v3: 2152.7 · Contributes. Source · Archived source review · claude-fable-5-1; published evaluation, effort unspecified
Star denotes community-submitted evaluation hosted on the official benchmark board; not independently rerun by the benchmark author.
GDP.pdf ; AA harness: 28% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Low Effort, Default Fallback)
Distinct AA protocol/version; not pooled with other harnesses. Fable Default Fallback scores are retained but not attributed to a single checkpoint.
MathArena ; ArXivMath May 2026: 87.5% · Contributes. Source · Evaluator-reported · Fable 5.1 (max); MathArena named monthly competition
Exact published two-decimal accuracy from the direct model comparison. Month is kept explicit; overall values are not mixed with month-level rows.
MathArena ; BrokenArXiv Jun 2026: 84.72% · Contributes. Source · Evaluator-reported · Fable 5.1 (max); MathArena named monthly competition
Exact published two-decimal accuracy from the direct model comparison. Month is kept explicit; overall values are not mixed with month-level rows.
MLCR-AA (Medical Long Context Reasoning): 71.111111% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)
Source still labels this model Default Fallback. Retained for completeness, excluded from single-model scoring.
SkillsBench †: 61.552% · Not used. Source · Evaluator-reported · Claude Fable 5.1; max; Vals published harness
Newly located exact Vals overall result. Retained as unresolved because Fable model page discloses fallback routing and benchmark-level impact has not been established.
Tax Agent Bench v1 ; Vals: 77.642% · Not used. Source · Evaluator-reported · Claude Fable 5.1; max; Vals published harness
Newly located exact Vals overall result. Retained as unresolved because Fable model page discloses fallback routing and benchmark-level impact has not been established.
Terminal-Bench 4.0 ; AA harness: 55.050505% · Not used. Source · Evaluator-reported · Claude Fable 5.1 (Adaptive Reasoning, Xhigh Effort, Default Fallback)
Distinct AA protocol/version; not pooled with other harnesses. Fable Default Fallback scores are retained but not attributed to a single checkpoint.

Opus 5 · 73 used / 103 collected

61 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.1 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1738.08 · Not used. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 89.138577% · Contributes. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 5 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 73.648649% · Contributes. Source · Evaluator-reported · claude-opus-5; max; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1493.32766 · Contributes. Source · Evaluator-reported · claude-opus-5-high; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 63 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 93.737374% · Contributes. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, Xhigh Effort); Artificial Analysis published evaluation harness
Best published result among 5 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 75.7% · Not used. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, Max Effort)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 58.63% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 73.56% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 56.365741% · Contributes. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 5 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 54.865616% · Contributes. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 5 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 55.29% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 6.67% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1647.02 · Not used. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 90.98% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 97% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 86.97% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 75.14% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
KingBench 3: 77.5% · Not used.
Not admitted after the September 6 model review: the collected 77.5 for Opus 5 on KingBench 3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vals Index: 74.8% · Not used. Source · Evaluator-reported · claude-opus-5; max effort; Vals AI index methodology
Held-out composite, never a scoring input
Public Benefits Bench ; Vals: 76.93% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 63.57% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AutomationBench v1.0.6: 50.3% · Not used.
Not admitted after the September 6 model review: the collected 50.3 for Opus 5 on AutomationBench v1.0.6 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Omniscience: 37.066667 · Contributes. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 5 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 91.59% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 80.084976% · Contributes. Source · Evaluator-reported · claude-opus-5-max-effort; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
CorpFin v2 ; Vals: 73.2% · Contributes. Source · Archived source review · claude-opus-5; max effort; Vals AI CorpFin v2 evaluation with Sonnet 4.5 judge
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
MortgageTax ; Vals: 72.06% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SAGE ; Vals: 49.43% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Code Migration †: 57.47% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Toolathlon-Verified: 80.6% · Not used.
Not admitted after the September 6 model review: the collected 80.6 for Opus 5 on Toolathlon-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Agents' Last Exam: 31.6% · Contributes. Source · Evaluator-reported · Opus 5; High; Claude Code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Terminal-Bench v3.0: 42.7% · Contributes. Source · Evaluator-reported · Claude Opus 5 (max); mini-SWE-agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveCodeBench: 89.03% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Humanity's Last Exam w/ tools: 63.6% · Contributes. Source · Developer-reported · Opus 5; OpenAI comparison table; maximum across reasoning modes
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Design Arena (Elo): 1407 · Contributes. Source · Archived source review · claude-opus-5; max effort; Design Arena 2026-08-11 rating snapshot
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Vals Multimodal Index: 73.9% · Not used. Source · Evaluator-reported · claude-opus-5; max effort; Vals AI multimodal index methodology
Held-out composite, never a scoring input
SWE-Marathon v1.1: 50% · Not used.
Not admitted after the September 6 model review: the collected 50 for Opus 5 on SWE-Marathon v1.1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ProgramBench v1 (Raw Pass Rate) ; Vals: 82.273% · Contributes. Source · Archived source review · anthropic/claude-opus-5 · compute=max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
SkillsBench †: 60.44% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierMath v2 Tier 4: 73.170732% · Contributes. Source · Evaluator-reported · claude-opus-5_max; task version 2.0.0
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
CursorBench v3.2: 70% · Contributes. Source · Evaluator-reported · Opus 5 Max; Cursor harness
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
APEX-Agents Mean Criteria Passed: 60.6% · Contributes. Source · Evaluator-reported · claude-opus-5; max; loop_truncated_tools_agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Pro: 79.2% · Not used.
Not admitted after the September 6 model review: the collected 79.2 for Opus 5 on SWE-bench Pro still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CharXiv (with CI / RQ): 89.3% · Not used.
Not admitted after the September 6 model review: the collected 89.3 for Opus 5 on CharXiv (with CI / RQ) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
APEX-SWE (Pass@1, Terminus-2): 63.7% · Contributes. Source · Evaluator-reported · claude-opus-5; max; terminus-2
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
BrowseComp: 90.8% · Contributes. Source · Developer-reported · Opus 5; OpenAI comparison table; maximum across reasoning modes
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MMMU-Pro: 84.739884% · Contributes. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 5 reasoning configurations of the exact AA model release; fallback configurations excluded.
ARC-AGI-2: 90.4% · Contributes. Source · Evaluator-reported · Opus 5; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
OSWorld 2.0 ; Anthropic H2H: 70.6% · Contributes. Source · Archived source review · claude-opus-5; max effort; Anthropic Opus 5 published head-to-head setup
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
ExploitBench: 70% · Not used.
Inherited comparator metric/scaffold equivalence to Astra Cap Percent is not established; the source flags possible contamination. Excluded pending protocol audit.
PostTrainBench: 35.04% · Not used.
Not admitted after the September 6 model review: the collected 35.04 for Opus 5 on PostTrainBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
SimpleQA: 56.7% · Not used.
Not admitted after the September 6 model review: the collected 56.7 for Opus 5 on SimpleQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Extended: 63.63% · Contributes. Source · Evaluator-reported · Claude Opus 5; medium; claude-code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
JobBench: 65.7% · Not used.
Not admitted after the September 6 model review: the collected 65.7 for Opus 5 on JobBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
OSWorld-Verified: 90.69% · Not used.
Not admitted after the September 6 model review: the collected 90.69 for Opus 5 on OSWorld-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Creative Writing v3: 2116.1 · Contributes. Source · Evaluator-reported · claude-opus-5; published evaluation, effort unspecified
Exact checkpoint on the evaluator-hosted board. A starred model is externally submitted; it is not claimed as independently rerun by the benchmark owner.
MathArena ; ArXivMath Jun 2026: 80.95% · Contributes. Source · Archived source review · claude-opus-5; max effort; MathArena June 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
MathArena ; BrokenArXiv Jun 2026: 90.74% · Contributes. Source · Archived source review · claude-opus-5; max effort; MathArena June 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
AIIQ Composite IQ: 134 · Not used. Source · Evaluator-reported · claude-opus-5; max effort; AIIQ composite snapshot
Held-out composite, never a scoring input
MCP Atlas †: 85.8% · Not used.
Not admitted after the September 6 model review: the collected 85.8 for Opus 5 on MCP Atlas † still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
OfficeQA Pro †: 66.9% · Not used.
Not admitted after the September 6 model review: the collected 66.9 for Opus 5 on OfficeQA Pro † still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EQ-Bench 4: 1385 · Contributes. Source · Evaluator-reported · claude-opus-5; evaluator published configuration, effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MathArena ; ArXivLean Jun 2026: 31.25% · Contributes. Source · Archived source review · claude-opus-5; max effort; MathArena June 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Frontier-Bench v0.1 ; Anthropic H2H: 43.3% · Contributes. Source · Archived source review · claude-opus-5; max effort; Anthropic Opus 5 published head-to-head setup
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
FrontierCode v1.1 Main ; Anthropic H2H: 53.4% · Not used.
Not admitted after the September 6 model review: the collected 53.4 for Opus 5 on FrontierCode v1.1 Main ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AutomationBench ; Anthropic H2H: 26% · Contributes. Source · Archived source review · claude-opus-5; max effort; Anthropic Opus 5 published head-to-head setup
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Harvey Legal Agent Benchmark ; held-out: 11.7% · Not used.
Not admitted after the September 6 model review: the collected 11.7 for Opus 5 on Harvey Legal Agent Benchmark ; held-out still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 44.742268% · Contributes. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, High Effort); Artificial Analysis published evaluation harness
Best published result among 5 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 29.142857% · Contributes. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 5 reasoning configurations of the exact AA model release; fallback configurations excluded.
Terminal-Bench 4.0: 51.8% · Contributes. Source · Evaluator-reported · Opus 5; max; Claude Code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
GDP.pdf: 24% · Contributes. Source · Evaluator-reported · Claude Opus 5 (Adaptive/Max); Surge original evaluation
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 99% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 88.4% · Contributes. Source · Evaluator-reported · anthropic_claude-opus-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MineBench: 2106 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
OSWorld 2.0: 31.43% · Not used.
Not admitted after the September 6 model review: the collected 31.43 for Opus 5 on OSWorld 2.0 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ARC-AGI-1: 97.5% · Contributes. Source · Evaluator-reported · Opus 5; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
ARC-AGI-3 (standard harness): 30.16% · Contributes. Source · Evaluator-reported · Opus 5; High; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
Terminal-Bench Science 0.1: 30% · Contributes. Source · Developer-reported · Opus 5; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
SRE-Bench (pass@1): 12.5% · Contributes. Source · Developer-reported · Opus 5; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
OSWorld 2.0 ; OpenAI H2H (offline subset): 70.2% · Contributes. Source · Developer-reported · Opus 5; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Agents' Last Exam ; OpenAI H2H: 55.5% · Contributes. Source · Developer-reported · Opus 5; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
HealthBench Professional (length-adjusted): 56.4% · Contributes. Source · Developer-reported · Opus 5; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Harvey LAB-AA: 93.46% · Not used.
Not admitted after the September 6 model review: the collected 93.46 for Opus 5 on Harvey LAB-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EnterpriseOps-Gym-AA: 47.48% · Not used.
Not admitted after the September 6 model review: the collected 47.48 for Opus 5 on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 59.444444% · Not used. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, High Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
AA-AnalystAgent: 53.75% · Not used. Source · Evaluator-reported · Claude Opus 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-AnalystAgent, so every model is scored on its current set.
DeepSearchQA: 95% · Not used.
Not admitted after the September 6 model review: the collected 95 for Opus 5 on DeepSearchQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vending-Bench 2: 11181.873333 · Contributes. Source · Evaluator-reported · Claude Opus 5; Andon Labs Vending-Bench 2
Compare reasoning configurations using their mean over runs. Geometric mean, individual best runs, Fireworks Kimi endpoint, Vending Arena, and April DeepSeek Pro are not substituted.
ZeroBench: 26% · Contributes. Source · Evaluator-reported · Claude Opus 5 (max); official main questions
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Mystery Game Puzzles v1.0.4 ; Epoch: 59% · Contributes. Source · Evaluator-reported · claude-opus-5_max; task version 1.0.4
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
RuneBench (30m, mean ln(1 + XP/min)): 5.705729 · Contributes. Source · Archived source review · opus5-xhigh
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 2315 · Contributes. Source · Archived source review · Claude Opus 5 (Max)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1688 · Contributes. Source · Archived source review · claude-opus-5-max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 50.777112 · Not used. Source · Evaluator-reported · Opus 5 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1656.53 · Contributes. Source · Evaluator-reported · Opus 5 (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1657.59 recorded 2026-10-22.
AA-LCR v1.1: 82% · Contributes. Source · Evaluator-reported · Opus 5 (medium); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 56.573556% · Contributes. Source · Evaluator-reported · Opus 5 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 21.6% · Contributes. Source · Evaluator-reported · Opus 5 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1721.67 · Contributes. Source · Evaluator-reported · Opus 5 (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1720.14 recorded 2026-10-22.
HLE text-only ; AA harness: 54.865616% · Contributes. Source · Evaluator-reported · Opus 5 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 48.989899% · Contributes. Source · Evaluator-reported · Opus 5 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
BioMysteryBench v1 ; Vals: 79.26% · Contributes. Source · Evaluator-reported · Opus 5 (max); Vals AI BioMysteryBench harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 79.26 ± 2.06, rank 5/27. Vals notes some benchmarks may use a different provider or parameters.

Muse Wonder · 20 used / 21 collected

20 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±10.5 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1658.65 · Contributes. Source · Evaluator-reported · Muse Wonder (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Daily evaluator refresh; replaces 1729.32 recorded 2026-10-12.
GDPval-AA v2.1 (Elo): 1770.34 · Contributes. Source · Evaluator-reported · Muse Wonder (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Daily evaluator refresh; replaces 1857.09 recorded 2026-10-12.
AutomationBench-AA: 68.094551% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Daily evaluator refresh; replaces 71.3307497322 recorded 2026-10-12.
Terminal-Bench 4.0 ; AA harness: 52.020202% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Daily evaluator refresh; replaces 57.0707070707 recorded 2026-10-12.
SciCode: 60.300926% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Daily evaluator refresh; replaces 62.6157407407 recorded 2026-10-12.
HLE text-only ; AA harness: 50.092678% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Daily evaluator refresh; replaces 53.6144578313 recorded 2026-10-12.
GDP.pdf ; AA harness: 30.6% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Daily evaluator refresh; replaces 32.6 recorded 2026-10-12.
CritPt: 28.285714% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Daily evaluator refresh; replaces 32.8571428571 recorded 2026-10-12.
AA-Omniscience: 25.483333 · Contributes. Source · Evaluator-reported · Muse Wonder (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Restored on 2026-10-28: 25.1 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. Daily evaluator refresh; replaces 30.15 recorded 2026-10-12.
AA-LCR v1.1: 80.666667% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Daily evaluator refresh; replaces 83.6666666667 recorded 2026-10-12.
AA Intelligence Index v4.3.2 (score): 52.393895 · Not used. Source · Evaluator-reported · Muse Wonder (max); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Restored on 2026-10-28: 43.324823 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Daily evaluator refresh; replaces 55.8021199856 recorded 2026-10-12.
DeepSWE v1.1: 78.8% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Muse Wonder developer-reported result (meta); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (opus-5-5 74.2, fable-5-1 67.4). Value read from an image of the lab table. Meta's methodology says the DeepSWE scorecard uses Datacurve's third-party results for Muse Wonder and Claude Opus 5.5, OpenAI's provider-reported result for GPT-6 Astra, and Anthropic's system card for Claude Fable 5.1. Metric pass@1.
Finance Agent v2 ; Vals: 56.6% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Muse Wonder developer-reported result (meta); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (fable-5-1 58.9, gpt-6-astra 53.5). Value read from an image of the lab table. Meta's methodology says the result source is Vals AI and the primary metric is private-test-set accuracy.
BioMysteryBench v1 ; Vals: 67.04% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Vals AI BioMysteryBench harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 67.04 ± 0.37, rank 19/27. Vals notes some benchmarks may use a different provider or parameters.
Code Migration †: 62.55% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Vals AI Code Migration harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 62.55 ± 4.35, rank 6/77. Vals notes some benchmarks may use a different provider or parameters.
Excel Modeling Benchmark ; Vals: 67.08% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Vals AI EMB harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 67.08 ± 2.89, rank 18/74. Vals notes some benchmarks may use a different provider or parameters.
Harvey LAB (Vals): 17.08% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Vals AI Harvey's Legal Agent Benchmark harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 17.08 ± 3.1, rank 8/79. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 50.48% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Vals AI Legal Research Bench harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 50.48 ± 3.48, rank 6/78. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 84% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Vals AI ProofBench v1.1 harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 84.0 ± 3.69, rank 12/50. Vals notes some benchmarks may use a different provider or parameters.
Public Benefits Bench ; Vals: 70.23% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Vals AI Public Benefits Bench v1.1 harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 70.23 ± 1.19, rank 6/50. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 83.09% · Contributes. Source · Evaluator-reported · Muse Wonder (max); Vals AI Vibe Code Bench v1.1 harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 83.09 ± 2.49, rank 18/112. Vals notes some benchmarks may use a different provider or parameters.

GPT-6.1 Sol · 19 used / 24 collected

19 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±11.7 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1555.26 · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1555.1 recorded 2026-10-27.
GDPval-AA v2.1 (Elo): 1607.61 · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1592.1 recorded 2026-10-21.
AutomationBench-AA: 66.5737214136% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (xhigh); Artificial Analysis independent evaluation, September 30 2026; fallback not reported
Exact model and effort in the evaluator chart payload. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
Terminal-Bench 4.0 ; AA harness: 56.0606060606% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Artificial Analysis independent evaluation, September 30 2026; fallback not reported
Exact model and effort in the evaluator chart payload. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
SciCode: 55.787037037% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (high); Artificial Analysis independent evaluation, September 30 2026; fallback not reported
Exact model and effort in the evaluator chart payload. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
HLE text-only ; AA harness: 52.9193697868% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Artificial Analysis independent evaluation, September 30 2026; fallback not reported
Exact model and effort in the evaluator chart payload. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
GDP.pdf ; AA harness: 32% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (high); Artificial Analysis independent evaluation, September 30 2026; fallback not reported
Exact model and effort in the evaluator chart payload. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
CritPt: 31.7142857143% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (xhigh); Artificial Analysis independent evaluation, September 30 2026; fallback not reported
Exact model and effort in the evaluator chart payload. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
AA-Omniscience: 41.5333333333 · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Artificial Analysis independent evaluation, September 30 2026; fallback not reported
Restored on 2026-10-28: 42.033333 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. Exact model and effort in the evaluator chart payload. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
AA-LCR v1.1: 84% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (low); Artificial Analysis independent evaluation, September 30 2026; fallback not reported
Exact model and effort in the evaluator chart payload. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
MMMU-Pro: 85.9537572254% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Artificial Analysis independent evaluation, September 30 2026; fallback not reported
Independent MMMU-Pro aggregate for this effort.
AA Intelligence Index v4.3.2 (score): 51.8332597012 · Not used. Source · Evaluator-reported · GPT-6.1 Sol (max); Artificial Analysis independent evaluation, September 30 2026; fallback not reported
Restored on 2026-10-28: 54.508734 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Reference composite only, never scored alongside its component evaluations.
HealthBench Professional ; Sep 22 length-adjusted: 64.2% · Not used. Source · Evaluator-reported · GPT-6.1 Sol (undisclosed); OpenAI September 30 system-card addendum; fallback not reported
Table 7 length-adjusted column, not the parenthesised unadjusted score. Same length-adjusted protocol as the September 22 release rows; remains uncalibrated.
HealthBench Overall ; Sep 22 length-adjusted: 58.5% · Not used. Source · Evaluator-reported · GPT-6.1 Sol (undisclosed); OpenAI September 30 system-card addendum; fallback not reported
Table 7 length-adjusted column, not the parenthesised unadjusted score. Same length-adjusted protocol as the September 22 release rows; remains uncalibrated.
HealthBench Hard ; Sep 22 length-adjusted: 36.2% · Not used. Source · Evaluator-reported · GPT-6.1 Sol (undisclosed); OpenAI September 30 system-card addendum; fallback not reported
Table 7 length-adjusted column, not the parenthesised unadjusted score. Same length-adjusted protocol as the September 22 release rows; remains uncalibrated.
HealthBench Consensus ; Sep 22 length-adjusted: 96% · Not used. Source · Evaluator-reported · GPT-6.1 Sol (undisclosed); OpenAI September 30 system-card addendum; fallback not reported
Table 7 length-adjusted column, not the parenthesised unadjusted score. Same length-adjusted protocol as the September 22 release rows; remains uncalibrated.
BioMysteryBench v1 ; Vals: 79.63% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Vals AI BioMysteryBench harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 79.63 ± 0.74, rank 2/27. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 38.46% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Vals AI Legal Research Bench harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 38.46 ± 3.38, rank 37/82. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 48.84% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Vals AI MedCode harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 48.84 ± 2.12, rank 28/107. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 86.45% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Vals AI MedScribe harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 86.45 ± 1.9, rank 23/109. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 99% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Vals AI ProofBench v1.1 harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 99.0 ± 1.0, rank 4/50. Vals notes some benchmarks may use a different provider or parameters.
Public Benefits Bench ; Vals: 59.34% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Vals AI Public Benefits Bench v1.1 harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 59.34 ± 1.28, rank 35/50. Vals notes some benchmarks may use a different provider or parameters.
SAGE ; Vals: 46.54% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Vals AI SAGE harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 46.54 ± 3.46, rank 35/91. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 88.93% · Contributes. Source · Evaluator-reported · GPT-6.1 Sol (max); Vals AI Vibe Code Bench v1.1 harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 88.93 ± 1.89, rank 8/116. Vals notes some benchmarks may use a different provider or parameters.

MiMo-V2.6-Pro · 26 used / 37 collected

24 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±6.5 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

DeepSWE v1.1: 71.9% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Released benchmark table 71.9; training-progress endpoint 72.57 is not the released checkpoint result and is not substituted.
ProgramBench ; MiMo V2.6 release: 26.5% · Not used. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
MiMo Code Bench: 63.2% · Not used. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
AutomationBench v1.0.6: 53.1% · Not used. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
Toolathlon-Verified: 76.9% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
Agents' Last Exam: 31.6% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
Terminal-Bench 4.0: 34.9% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
Terminal-Bench 2.1: 89.9% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
OSWorld-Verified: 82% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
JobBench: 62% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
CyberGym: 94% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
MiMo Cyber Bench: 80.2% · Not used. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Unresolved: model card reports 80.2, while the official release bench.js reports 81.7. Neither is admitted to the score.
ExploitGym: 17.8% · Not used. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
ExploitBench ; MiMo V2.6 release: 47.9% · Not used. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Release table reports 47.9; supplied source review identifies a conflicting product chart value of 43.8. Protocol/version equivalence and discrepancy remain unresolved; not scored.
SEC-Bench Pro ; MiMo V2.6 release: 66.3% · Not used. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
MiMo Visual Coding: 72.3% · Not used. Source · Evaluator-reported · MiMo-V2.6-Pro; Xiaomi release evaluation; harness undisclosed; effort, retries and fallback undisclosed
Reviewed published release aggregate; no effort or harness equivalence inferred for versioned/ambiguous protocols.
AA-Briefcase v1.1 (Elo): 1521.82 · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Artificial Analysis independent evaluation, September 22 2026; effort, retries and fallback undisclosed
Directly reviewed exact-model chart payload from the independent evaluator; not a reseller attribution.
GDPval-AA v2.1 (Elo): 1673.19 · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Artificial Analysis independent evaluation, September 22 2026; effort, retries and fallback undisclosed
Directly reviewed exact-model chart payload from the independent evaluator; not a reseller attribution. Native 1673.19 Elo corroborates rounded 1673 from Xiaomi; normalized 58.6595% is not an additional observation.
AutomationBench-AA: 58.6234094342% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Artificial Analysis independent evaluation, September 22 2026; effort, retries and fallback undisclosed
Directly reviewed exact-model chart payload from the independent evaluator; not a reseller attribution.
Terminal-Bench 4.0 ; AA harness: 34.8484848485% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Artificial Analysis independent evaluation, September 22 2026; effort, retries and fallback undisclosed
Directly reviewed exact-model chart payload from the independent evaluator; not a reseller attribution.
SciCode: 60.8796296296% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Artificial Analysis independent evaluation, September 22 2026; effort, retries and fallback undisclosed
Directly reviewed exact-model chart payload from the independent evaluator; not a reseller attribution.
Humanity's Last Exam (no tools): 49.3512511585% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Artificial Analysis independent evaluation, September 22 2026; effort, retries and fallback undisclosed
Directly reviewed exact-model chart payload from the independent evaluator; not a reseller attribution.
GDP.pdf ; AA harness: 19.2% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Artificial Analysis independent evaluation, September 22 2026; effort, retries and fallback undisclosed
Directly reviewed exact-model chart payload from the independent evaluator; not a reseller attribution.
CritPt: 26.5714285714% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Artificial Analysis independent evaluation, September 22 2026; effort, retries and fallback undisclosed
Directly reviewed exact-model chart payload from the independent evaluator; not a reseller attribution.
AA-Omniscience: 8.3833333333 · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Artificial Analysis independent evaluation, September 22 2026; effort, retries and fallback undisclosed
Directly reviewed exact-model chart payload from the independent evaluator; not a reseller attribution. Index 8.3833333333; accuracy 34.85% and non-hallucination 59.3757994372% are different metrics, not the index and not additional scored benchmarks.
AA-LCR v1.1: 86.3333333333% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro; Artificial Analysis independent evaluation, September 22 2026; effort, retries and fallback undisclosed
Directly reviewed exact-model chart payload from the independent evaluator; not a reseller attribution.
AA Intelligence Index v4.3.2 (score): 46.324206531 · Not used. Source · Evaluator-reported · MiMo-V2.6-Pro; Artificial Analysis independent evaluation, September 22 2026; effort, retries and fallback undisclosed
Composite retained for reference only; not substituted into v4.1.1 and not scored alongside its components.
OpenLM Arena+ ; Overall (Elo): 1507 · Not used. Source · Evaluator-reported · MiMo-V2.6-Pro; OpenLM Arena+ LLM-as-a-judge; overall track; effort, retries and fallback undisclosed
Direct operator listing. LLM-as-a-judge, not human-vote LMArena; shares a suite with the coding track.
OpenLM Arena+ ; Coding (Elo): 1560 · Not used. Source · Evaluator-reported · MiMo-V2.6-Pro; OpenLM Arena+ LLM-as-a-judge; coding track; effort, retries and fallback undisclosed
Direct operator coding-track listing; LLM-as-a-judge. Shares a suite with the overall track; no calibrated score contribution.
HLE text-only ; AA harness: 49.351251% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro (none); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Legal Research Bench ; Vals: 47.12% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro (undisclosed); Vals AI Legal Research Bench harness; provider Xiaomi; fallback not reported
Vals AI model page, headline accuracy 47.12 ± 3.47, rank 15/82. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 44.97% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro (undisclosed); Vals AI MedCode harness; provider Xiaomi; fallback not reported
Vals AI model page, headline accuracy 44.97 ± 2.1, rank 39/107. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 88.31% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro (undisclosed); Vals AI MedScribe harness; provider Xiaomi; fallback not reported
Vals AI model page, headline accuracy 88.31 ± 1.94, rank 13/109. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 70% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro (undisclosed); Vals AI ProofBench v1.1 harness; provider Xiaomi; fallback not reported
Vals AI model page, headline accuracy 70.0 ± 4.61, rank 18/50. Vals notes some benchmarks may use a different provider or parameters.
Public Benefits Bench ; Vals: 68.94% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro (undisclosed); Vals AI Public Benefits Bench v1.1 harness; provider Xiaomi; fallback not reported
Vals AI model page, headline accuracy 68.94 ± 1.2, rank 8/50. Vals notes some benchmarks may use a different provider or parameters.
SAGE ; Vals: 45.05% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro (undisclosed); Vals AI SAGE harness; provider Xiaomi; fallback not reported
Vals AI model page, headline accuracy 45.05 ± 3.4, rank 41/91. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 85.22% · Contributes. Source · Evaluator-reported · MiMo-V2.6-Pro (undisclosed); Vals AI Vibe Code Bench v1.1 harness; provider Xiaomi; fallback not reported
Vals AI model page, headline accuracy 85.22 ± 3.39, rank 15/116. Vals notes some benchmarks may use a different provider or parameters.

Fable 5 · 57 used / 114 collected

48 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.2 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1741 · Not used. Source · Evaluator-reported · Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
Terminal-Bench 2.1: 84.6% · Not used. Source · Evaluator-reported · Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
DeepSWE v1.1: 69.911504% · Contributes. Source · Evaluator-reported · claude-fable-5; xhigh; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1507.164172 · Contributes. Source · Evaluator-reported · claude-fable-5; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 62 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 92.6% · Contributes. Source · Archived source review · claude-fable-5; max effort; No tools
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
AA-LCR: 70% · Not used. Source · Evaluator-reported · Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 56.31% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 73.67% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 60% · Not used. Source · Evaluator-reported · Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
Humanity's Last Exam (no tools): 53.3% · Not used. Source · Evaluator-reported · Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
Legal Research Bench ; Vals: 49.52% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 11.25% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1574 · Not used. Source · Evaluator-reported · Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
MedScribe ; Vals: 88.52% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 95% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 88.56% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 76.94% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
KingBench 3: 82.5% · Not used.
Not admitted after the September 6 model review: the collected 82.5 for Fable 5 on KingBench 3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vals Index: 75.1% · Not used. Source · Evaluator-reported · claude-fable-5; max effort; Vals AI index methodology
Held-out composite, never a scoring input
Public Benefits Bench ; Vals: 70.43% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 56.07% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AutomationBench v1.0.6: 46.2% · Not used.
Not admitted after the September 6 model review: the collected 46.2 for Fable 5 on AutomationBench v1.0.6 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Omniscience: 40.15 · Not used. Source · Evaluator-reported · Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
MMLU Pro ; Vals: 91.5% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 82.970798% · Contributes. Source · Evaluator-reported · claude-fable-5-max-effort; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
CorpFin v2 ; Vals: 71.8% · Contributes. Source · Archived source review · claude-fable-5; max effort; Vals AI CorpFin v2 evaluation with Sonnet 4.5 judge
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
MortgageTax ; Vals: 68.92% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SAGE ; Vals: 51.89% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Code Migration †: 55.06% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Toolathlon-Verified: 77.9% · Not used.
Not admitted after the September 6 model review: the collected 77.9 for Fable 5 on Toolathlon-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Agents' Last Exam: 25.7% · Contributes. Source · Evaluator-reported · Fable 5; XHigh; Claude Code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Terminal-Bench v3.0: 34.1% · Contributes. Source · Evaluator-reported · Claude Fable 5 (max); Claude Code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveCodeBench: 89.78% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CyberGym: 83.8% · Not used.
Not admitted after the September 6 model review: the collected 83.8 for Fable 5 on CyberGym still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Humanity's Last Exam w/ tools: 63.8% · Contributes. Source · Developer-reported · Fable 5; OpenAI comparison table; maximum across reasoning modes
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Design Arena (Elo): 1400 · Contributes. Source · Archived source review · claude-fable-5; max effort; Design Arena 2026-08-11 rating snapshot
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Vals Multimodal Index: 74.2% · Not used. Source · Evaluator-reported · claude-fable-5; max effort; Vals AI multimodal index methodology
Held-out composite, never a scoring input
SWE-Marathon v1.1: 33.1% · Not used.
Not admitted after the September 6 model review: the collected 33.1 for Fable 5 on SWE-Marathon v1.1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
SkillsBench †: 70.9% · Not used.
Not admitted after the September 6 model review: the collected 70.9 for Fable 5 on SkillsBench † still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierSWE: 86.6% · Not used.
Not admitted after the September 6 model review: the collected 86.6 for Fable 5 on FrontierSWE still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierMath v2 Tier 4: 87.804878% · Contributes. Source · Evaluator-reported · claude-fable-5_max; task version 2.0.0
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
CursorBench v3.2: 70.5% · Contributes. Source · Evaluator-reported · Fable 5 Max; Cursor harness
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
APEX-Agents Mean Criteria Passed: 59.2% · Contributes. Source · Evaluator-reported · claude-fable-5; max; loop_truncated_tools_agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Pro: 80% · Not used.
Not admitted after the September 6 model review: the collected 80 for Fable 5 on SWE-bench Pro still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CharXiv (with CI / RQ): 93.5% · Not used.
Not admitted after the September 6 model review: the collected 93.5 for Fable 5 on CharXiv (with CI / RQ) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
APEX-SWE (Pass@1, Terminus-2): 58.8% · Contributes. Source · Evaluator-reported · claude-fable-5; max; terminus-2
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
BrowseComp: 87.4% · Contributes. Source · Developer-reported · Fable 5; OpenAI comparison table; maximum across reasoning modes
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MMMU-Pro: 81.2% · Not used. Source · Evaluator-reported · Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
ARC-AGI-2: 89.2% · Contributes. Source · Evaluator-reported · Fable 5; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
OSWorld 2.0 ; Anthropic H2H: 66.1% · Contributes. Source · Archived source review · claude-fable-5; max effort; Anthropic Opus 5 published head-to-head setup
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
ExploitBench: 78% · Not used.
Inherited comparator metric/scaffold equivalence to Astra Cap Percent is not established; the source flags possible contamination. Excluded pending protocol audit.
PostTrainBench: 41.8% · Not used.
Not admitted after the September 6 model review: the collected 41.8 for Fable 5 on PostTrainBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
SimpleQA: 68.3% · Not used.
Not admitted after the September 6 model review: the collected 68.3 for Fable 5 on SimpleQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Extended: 64.94% · Contributes. Source · Evaluator-reported · Claude Fable 5; xhigh; claude-code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
JobBench: 57.4% · Not used.
Not admitted after the September 6 model review: the collected 57.4 for Fable 5 on JobBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
BabyVision (with CI): 90.5% · Not used.
Not admitted after the September 6 model review: the collected 90.5 for Fable 5 on BabyVision (with CI) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
PerceptionBench: 57.2% · Not used.
Not admitted after the September 6 model review: the collected 57.2 for Fable 5 on PerceptionBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
LVBench (with Memory): 90.1% · Not used.
Not admitted after the September 6 model review: the collected 90.1 for Fable 5 on LVBench (with Memory) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
OSWorld-Verified: 85% · Not used.
Not admitted after the September 6 model review: the collected 85 for Fable 5 on OSWorld-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Creative Writing v3: 1933.2 · Contributes. Source · Evaluator-reported · claude-fable-5; published evaluation, effort unspecified
Exact checkpoint on the evaluator-hosted board. A starred model is externally submitted; it is not claimed as independently rerun by the benchmark owner.
MathArena ; ArXivMath Jun 2026: 83.67% · Contributes. Source · Archived source review · claude-fable-5; max effort; MathArena June 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
MathArena ; BrokenArXiv Jun 2026: 47.84% · Contributes. Source · Archived source review · claude-fable-5; max effort; MathArena June 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
AIIQ Composite IQ: 134 · Not used. Source · Evaluator-reported · claude-fable-5; max effort; AIIQ composite snapshot
Held-out composite, never a scoring input
MCP Atlas †: 84.7% · Not used.
Not admitted after the September 6 model review: the collected 84.7 for Fable 5 on MCP Atlas † still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
OfficeQA Pro †: 69.9% · Not used.
Not admitted after the September 6 model review: the collected 69.9 for Fable 5 on OfficeQA Pro † still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EQ-Bench 4: 1340.4 · Contributes. Source · Evaluator-reported · claude-fable-5; evaluator published configuration, effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CoWorkBench: 75.9% · Not used.
Not admitted after the September 6 model review: the collected 75.9 for Fable 5 on CoWorkBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ²-Bench Telecom: 98.5% · Not used.
Not admitted after the September 6 model review: the collected 98.5 for Fable 5 on τ²-Bench Telecom still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
PaperBench (Replication Score): 88.8% · Not used.
Not admitted after the September 6 model review: the collected 88.8 for Fable 5 on PaperBench (Replication Score) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
QwenReactBench (Elo): 1770 · Not used.
Not admitted after the September 6 model review: the collected 1770 for Fable 5 on QwenReactBench (Elo) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ERQA: 70% · Not used.
Not admitted after the September 6 model review: the collected 70 for Fable 5 on ERQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vision2Web (Avg. Frontend/Webpage/etc.): 70.5% · Not used.
Not admitted after the September 6 model review: the collected 70.5 for Fable 5 on Vision2Web (Avg. Frontend/Webpage/etc.) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MobileWorld: 85.5% · Not used.
Not admitted after the September 6 model review: the collected 85.5 for Fable 5 on MobileWorld still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
WebArena-Verified: 71.3% · Not used.
Not admitted after the September 6 model review: the collected 71.3 for Fable 5 on WebArena-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathArena ; ArXivLean Jun 2026: 14.58% · Contributes. Source · Archived source review · claude-fable-5; max effort; MathArena June 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Frontier-Bench v0.1 ; Anthropic H2H: 33.7% · Contributes. Source · Archived source review · claude-fable-5; max effort; Anthropic Opus 5 published head-to-head setup
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
FrontierCode v1.1 Main ; Anthropic H2H: 53.5% · Not used.
Not admitted after the September 6 model review: the collected 53.5 for Fable 5 on FrontierCode v1.1 Main ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AutomationBench ; Anthropic H2H: 17.4% · Contributes. Source · Archived source review · claude-fable-5; max effort; Anthropic Opus 5 published head-to-head setup
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Harvey Legal Agent Benchmark ; held-out: 13.3% · Not used.
Not admitted after the September 6 model review: the collected 13.3 for Fable 5 on Harvey Legal Agent Benchmark ; held-out still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathArena ; ArXivMath May 2026: 87.5% · Contributes. Source · Archived source review · claude-fable-5; max effort; MathArena May 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
EQ-Bench 3: 2049.7 · Not used. Source · Evaluator-reported · *claude-fable-5; published evaluation, effort unspecified
Exact checkpoint on the evaluator-hosted board. A starred model is externally submitted; it is not claimed as independently rerun by the benchmark owner.
τ³-Banking: 38.1% · Not used. Source · Evaluator-reported · Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
CritPt: 28.6% · Not used. Source · Evaluator-reported · Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
Terminal-Bench 4.0: 44.5% · Contributes. Source · Evaluator-reported · Fable 5; max; Claude Code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
GDP.pdf: 29.8% · Contributes. Source · Evaluator-reported · Claude Fable 5 (Adaptive/Max); Surge original evaluation
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 95% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 90.35% · Contributes. Source · Evaluator-reported · anthropic_claude-fable-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MineBench: 1972 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
ARC-AGI-1: 98.5% · Contributes. Source · Evaluator-reported · Fable 5; XHigh; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
Terminal-Bench Science 0.1: 21.4% · Contributes. Source · Developer-reported · Fable 5; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Agents' Last Exam ; OpenAI H2H: 48.7% · Contributes. Source · Developer-reported · Fable 5; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
HealthBench Professional (length-adjusted): 60.9% · Contributes. Source · Developer-reported · Fable 5; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Harvey LAB-AA: 93.56% · Not used.
Not admitted after the September 6 model review: the collected 93.56 for Fable 5 on Harvey LAB-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EnterpriseOps-Gym-AA: 51.12% · Not used.
Not admitted after the September 6 model review: the collected 51.12 for Fable 5 on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 64.44% · Not used. Source · Evaluator-reported · Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
AA-AnalystAgent: 48.75% · Not used. Source · Evaluator-reported · Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)
AA labels this configuration Default Fallback; per-evaluation fallback use is not established. Do not attribute a multi-model result to the named model alone.
DeepSearchQA: 94.2% · Not used.
Not admitted after the September 6 model review: the collected 94.2 for Fable 5 on DeepSearchQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vending-Bench 2: 5680.258 · Contributes. Source · Evaluator-reported · Claude Fable 5 - High; Andon Labs Vending-Bench 2
Compare reasoning configurations using their mean over runs. Geometric mean, individual best runs, Fireworks Kimi endpoint, Vending Arena, and April DeepSeek Pro are not substituted.
IFBench: 63.5% · Not used.
Not admitted after the September 6 model review: the collected 63.5 for Fable 5 on IFBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AndroidWorld: 88.8% · Not used.
Not admitted after the September 6 model review: the collected 88.8 for Fable 5 on AndroidWorld still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CWE-bench: 47.8% · Not used.
Not admitted after the September 6 model review: the collected 47.8 for Fable 5 on CWE-bench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ZeroBench: 24% · Contributes. Source · Evaluator-reported · Claude Fable 5 (max); official main questions
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SimpleVQA: 73.4% · Not used.
Not admitted after the September 6 model review: the collected 73.4 for Fable 5 on SimpleVQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathVision (with CI): 98.6% · Not used.
Not admitted after the September 6 model review: the collected 98.6 for Fable 5 on MathVision (with CI) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
OmniDocBench 1.5: 89.5% · Not used.
Not admitted after the September 6 model review: the collected 89.5 for Fable 5 on OmniDocBench 1.5 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Internal Database Migration Tasks ; OpenAI: 50.3% · Contributes. Source · Developer-reported · Fable 5; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Internal Design Tasks ; OpenAI: 35.8% · Contributes. Source · Developer-reported · Fable 5; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Internal Data Science Tasks ; OpenAI: 34.7% · Contributes. Source · Developer-reported · Fable 5; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Mystery Game Puzzles v1.0.4 ; Epoch: 52% · Contributes. Source · Evaluator-reported · claude-fable-5_max; task version 1.0.4
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
FrontierMath Erdos (68 conjectures) ; Epoch: 0 · Not used. Source · Evaluator-reported · claude-fable-5_max; task version
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
RuneBench (30m, mean ln(1 + XP/min)): 5.67775 · Contributes. Source · Archived source review · fable-5-xhigh
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 2149 · Contributes. Source · Archived source review · Claude Fable 5 (Max)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1629 · Contributes. Source · Archived source review · claude-fable-5
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.

DeepSeek V4.1 Flash 1017 · 15 used / 16 collected

15 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±16.9 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1570.42 · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
GDPval-AA v2.1 (Elo): 1745.77 · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
AutomationBench-AA: 66.6409536495% · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
Terminal-Bench 4.0 ; AA harness: 33.8383838384% · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
SciCode: 55.9027777778% · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
HLE text-only ; AA harness: 43.3271547729% · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
GDP.pdf ; AA harness: 17.2% · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
CritPt: 24% · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
AA-Omniscience: 4.7 · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Restored on 2026-10-28: 0.9 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. Exact model and effort in the evaluator chart payload.
AA-LCR v1.1: 82.6666666667% · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
MMMU-Pro: 80.6358381503% · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Independent MMMU-Pro aggregate for this effort.
AA Intelligence Index v4.3.2 (score): 45.0965048682 · Not used. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Restored on 2026-10-28: 32.489639 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Reference composite only, never scored alongside its component evaluations.
NL2Repo: 68.2% · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (effort not stated); DeepSeek V4.1 Flash 1017 developer-reported result (deepseek); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (deepseek-v4-1-flash 65.4, kimi-k3 58).
BabyVision (with CI): 92.5% · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (effort not stated); DeepSeek V4.1 Flash 1017 developer-reported result (deepseek); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (gpt-5-6-sol 88.9, kimi-k3 85.7).
Legal Research Bench ; Vals: 45.19% · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Vals AI Legal Research Bench harness; provider DeepSeek; fallback not reported
Vals AI model page, headline accuracy 45.19 ± 3.46, rank 17/81. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 87.73% · Contributes. Source · Evaluator-reported · DeepSeek V4.1 Flash 1017 (max); Vals AI Vibe Code Bench v1.1 harness; provider DeepSeek; fallback not reported
Vals AI model page, headline accuracy 87.73 ± 2.75, rank 11/115. Vals notes some benchmarks may use a different provider or parameters.

GPT-6 Sol · 27 used / 40 collected

27 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±3.9 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1477.11 · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1478.37 recorded 2026-10-22.
GDPval-AA v2.1 (Elo): 1508.58 · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1507.57 recorded 2026-10-22.
AutomationBench-AA: 61.6942351159% · Contributes. Source · Evaluator-reported · GPT-6 Sol (xhigh); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-sol-xhigh / automationbench-aa. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
Terminal-Bench 4.0 ; AA harness: 43.9393939394% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-sol / terminalbench-4-0. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
SciCode: 57.6388888889% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-sol / scicode. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
HLE text-only ; AA harness: 47.9147358665% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-sol / humanitys-last-exam. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
GDP.pdf ; AA harness: 25.2% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; replaces 24.8 recorded 2026-09-22.
CritPt: 30.8571428571% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-sol / critpt. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
AA-Omniscience: 27.1166666667 · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-sol / omniscience. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
AA-LCR v1.1: 83.6666666667% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-sol / artificial-analysis-long-context-reasoning. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
MMMU-Pro: 82.947977% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; replaces 83.2947976879 recorded 2026-09-22.
AA Intelligence Index v4.3.2 (score): 47.630518 · Not used. Source · Evaluator-reported · GPT-6 Sol (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; replaces 47.5276426438 recorded 2026-09-22.
FrontierCode v1.1 Main ; Anthropic H2H: 49.27% · Not used. Source · Evaluator-reported · GPT-6 Sol (max); Cognition FrontierCode 1.1; Codex harness; fallback not reported
Cognition v1_1 data, main. new_score is mergeability; correct is a different metric. Unfair internet use is zeroed. Existing catalog name Main ; Anthropic H2H tracks this same public Cognition Main protocol.
FrontierCode v1.1 Extended: 60.69% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Cognition FrontierCode 1.1; Codex harness; fallback not reported
Cognition v1_1 data, extended. new_score is mergeability; correct is a different metric. Unfair internet use is zeroed. Existing catalog name Main ; Anthropic H2H tracks this same public Cognition Main protocol.
DeepSWE v1.1: 68.8% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); OpenAI launch research/API evaluation; fallback not reported
Explicit OpenAI launch paragraph. The archived operator leaderboard predates these additions; not inferred from GPT-5.6 results.
HealthBench Professional ; Sep 22 length-adjusted: 60.8% · Not used. Source · Evaluator-reported · GPT-6 Sol (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
Table 29 length-adjusted column, not parenthesized raw score. Revised comparison baselines differ from September 3; kept separate and uncalibrated.
HealthBench Overall ; Sep 22 length-adjusted: 53.2% · Not used. Source · Evaluator-reported · GPT-6 Sol (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
Table 29 length-adjusted column, not parenthesized raw score. Revised comparison baselines differ from September 3; kept separate and uncalibrated.
HealthBench Hard ; Sep 22 length-adjusted: 30.1% · Not used. Source · Evaluator-reported · GPT-6 Sol (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
Table 29 length-adjusted column, not parenthesized raw score. Revised comparison baselines differ from September 3; kept separate and uncalibrated.
HealthBench Consensus ; Sep 22 length-adjusted: 96.2% · Not used. Source · Evaluator-reported · GPT-6 Sol (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
Table 29 length-adjusted column, not parenthesized raw score. Revised comparison baselines differ from September 3; kept separate and uncalibrated.
ExploitBench: 81.7% · Not used. Source · Evaluator-reported · GPT-6 Sol (max); OpenAI September 22 system-card appendix; fallback not reported
§11.8.1.2.1. Publisher warns of possible historical vulnerability contamination; not the recent internal port.
SEC-Bench Pro: 66.3% · Not used. Source · Evaluator-reported · GPT-6 Sol (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
§11.8.1.2, same OpenAI V8/SpiderMonkey vulnerability-discovery evaluation; named effort not specified in prose.
ExploitBench Internal Port ; Sep 22 release: 5.5% · Not used. Source · Evaluator-reported · GPT-6 Sol (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
§11.8.1.2.2 June–August vulnerabilities. Astra comparison is 31.5 versus 39 in earlier release; no substitution into the old calibrated row.
ExploitGym ; Sep 22 release: 22.1% · Not used. Source · Evaluator-reported · GPT-6 Sol (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
Working-exploit success, §11.8.1.2. Distinct from honeypot compliance and safety behavior.
Sandbox Bench ; 22 targets: 4.5454545455% · Not used. Source · Evaluator-reported · GPT-6 Sol (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
§11.8.1.2.3 one of 22 targets; converted from count to percent. Not a best-trial or per-category observation.
Internal Research Debugging ; Sep 22: 64.2% · Not used. Source · Evaluator-reported · GPT-6 Sol (undisclosed); OpenAI system-card figure 84; fallback not reported
Figure 84: exact printed mean rubric reward (%), not task success.
KernelGen 1P ; Sep 22: 38.12% · Not used. Source · Evaluator-reported · GPT-6 Sol (undisclosed); OpenAI system-card figure 85; fallback not reported
Figure 85: exact printed rubric reward (%), not speedup ratio.
Agents' Last Exam ; OpenAI H2H: 56.4% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); OpenAI launch research/API evaluation; fallback not reported
OpenAI launch, ALE V1; not the separately archived public 147-task board.
OSWorld 2.0 ; OpenAI H2H (offline subset): 60.5% · Contributes. Source · Evaluator-reported · GPT-6 Sol (xhigh); OpenAI launch research/API evaluation; fallback not reported
OpenAI launch explicitly identifies v2026.08.08 offline partial reward; not full-set binary pass or revised Anthropic assets.
AutomationBench ; Anthropic H2H: 33.2% · Contributes. Source · Evaluator-reported · GPT-6 Sol (xhigh); Zapier operator leaderboard; native full-workflow success; fallback not reported
Zapier native operator leaderboard, September 22. Same held-out evaluation as H2H row (Opus5 max 26.94), not public 600-task v1.0.6 or AA partial credit.
BioMysteryBench v1 ; Vals: 74.81% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Vals AI BioMysteryBench harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 74.81 ± 2.43, rank 7/27. Vals notes some benchmarks may use a different provider or parameters.
Code Migration †: 57.2% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Vals AI Code Migration harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 57.2 ± 4.21, rank 9/79. Vals notes some benchmarks may use a different provider or parameters.
Excel Modeling Benchmark ; Vals: 71.53% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Vals AI EMB harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 71.53 ± 2.35, rank 11/76. Vals notes some benchmarks may use a different provider or parameters.
Finance Agent v2 ; Vals: 49.05% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Vals AI Finance Agent (v2) harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 49.05 ± 0.58, rank 48/80. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 28.85% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Vals AI Legal Research Bench harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 28.85 ± 3.15, rank 52/82. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 47.07% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Vals AI MedCode harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 47.07 ± 2.12, rank 36/107. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 82.03% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Vals AI MedScribe harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 82.03 ± 1.94, rank 52/109. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 83% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Vals AI ProofBench v1.1 harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 83.0 ± 3.77, rank 13/50. Vals notes some benchmarks may use a different provider or parameters.
Public Benefits Bench ; Vals: 56.63% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Vals AI Public Benefits Bench v1.1 harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 56.63 ± 1.29, rank 40/50. Vals notes some benchmarks may use a different provider or parameters.
SAGE ; Vals: 44.79% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Vals AI SAGE harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 44.79 ± 3.36, rank 44/91. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 87.82% · Contributes. Source · Evaluator-reported · GPT-6 Sol (max); Vals AI Vibe Code Bench v1.1 harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 87.82 ± 2.53, rank 10/116. Vals notes some benchmarks may use a different provider or parameters.

GPT-5.6 Sol · 90 used / 143 collected

75 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±0.9 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1627.48 · Not used. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 89.513109% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (xhigh); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 72.666667% · Contributes. Source · Evaluator-reported · gpt-5-6-sol; max; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1483.151082 · Contributes. Source · Evaluator-reported · gpt-5.6-sol-xhigh; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 61 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 94.141414% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 73.7% · Not used. Source · Evaluator-reported · GPT-5.6 Sol (max)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 53.76% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 72.34% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 57.75463% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (high); Artificial Analysis published evaluation harness
Best published result among 5 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 49.490269% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 48.08% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 2.5% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1478.66 · Not used. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 85.23% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 96.2% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 86.97% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 74.78% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
KingBench 3: 71.25% · Not used.
Not admitted after the September 6 model review: the collected 71.25 for GPT-5.6 Sol on KingBench 3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vals Index: 73.1% · Not used. Source · Evaluator-reported · gpt-5.6-sol; max effort; Vals AI index methodology
Held-out composite, never a scoring input
Public Benefits Bench ; Vals: 66.51% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 43.97% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AutomationBench v1.0.6: 45.8% · Not used.
Not admitted after the September 6 model review: the collected 45.8 for GPT-5.6 Sol on AutomationBench v1.0.6 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Omniscience: 21.966667 · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 89.1% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 81.053548% · Contributes. Source · Evaluator-reported · gpt-5.6-sol-max; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
CorpFin v2 ; Vals: 64.4% · Contributes. Source · Archived source review · gpt-5.6-sol; max effort; Vals AI CorpFin v2 evaluation with Sonnet 4.5 judge
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
MortgageTax ; Vals: 67.29% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SAGE ; Vals: 52.56% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Code Migration †: 52.92% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Toolathlon-Verified: 74.9% · Not used.
Not admitted after the September 6 model review: the collected 74.9 for GPT-5.6 Sol on Toolathlon-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Agents' Last Exam: 30.6% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol; XHigh; Codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Terminal-Bench v3.0: 34.6% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveCodeBench: 82.6% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CyberGym: 83.6% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol; Z.ai GLM-5.3 model-card comparison table; GLM max, other comparator efforts not fully disclosed
CyberGym release-table percentage. Z.ai documents single-run pass@1 on 1,507 tasks, unlimited task timeout, Claude Code 2.1.207 and restricted network for GLM-5.3. Do not transfer those GLM-specific settings to every comparator. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
Humanity's Last Exam w/ tools: 64.5% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol; Z.ai GLM-5.3 model-card comparison table; GLM max, other comparator efforts not fully disclosed
HLE with tools release-table percentage. Z.ai documents temperature=1, top_p=.95, 163,840 output tokens, 300K context and GPT-5.6 Luna medium judge for its evaluation. Comparator-specific tools and judging settings are not fully disclosed. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
Design Arena (Elo): 1379 · Contributes. Source · Archived source review · gpt-5.6-sol; max effort; Design Arena 2026-08-11 rating snapshot
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Vals Multimodal Index: 72.2% · Not used. Source · Evaluator-reported · gpt-5.6-sol; max effort; Vals AI multimodal index methodology
Held-out composite, never a scoring input
SWE-Marathon v1.1: 42.5% · Not used.
Not admitted after the September 6 model review: the collected 42.5 for GPT-5.6 Sol on SWE-Marathon v1.1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ProgramBench v1 (Raw Pass Rate) ; Vals: 77.644% · Contributes. Source · Archived source review · openai/gpt-5.6-sol · reasoning_effort=max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
SkillsBench †: 54.1% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierSWE: 71.3% · Contributes. Source · Developer-reported · GPT-5.6 Sol; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierMath v2 Tier 4: 82.926829% · Contributes. Source · Evaluator-reported · gpt-5.6-sol_max; task version 2.0.0
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
CursorBench v3.2: 67.2% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol Max; Cursor harness
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
APEX-Agents Mean Criteria Passed: 56.7% · Contributes. Source · Evaluator-reported · gpt-5-6-sol-max; max; loop_truncated_tools_agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Pro: 64.6% · Not used.
Not admitted after the September 6 model review: the collected 64.6 for GPT-5.6 Sol on SWE-bench Pro still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CharXiv (with CI / RQ): 89.1% · Contributes. Source · Developer-reported · GPT-5.6 Sol; max; Kimi developer comparison table; with Python tools
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
APEX-SWE (Pass@1, Terminus-2): 45.8% · Contributes. Source · Evaluator-reported · gpt-5-6-sol-xhigh; xhigh; terminus-2
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
BrowseComp: 90.4% · Contributes. Source · Developer-reported · GPT-5.6 Sol; OpenAI comparison table; maximum across reasoning modes
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MMMU-Pro: 83.410405% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
ARC-AGI-2: 92.5% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
OSWorld 2.0 ; Anthropic H2H: 62.6% · Contributes. Source · Archived source review · gpt-5.6-sol; max effort; Anthropic Opus 5 published head-to-head setup
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
ExploitBench: 76.5% · Not used.
Inherited comparator metric/scaffold equivalence to Astra Cap Percent is not established; the source flags possible contamination. Excluded pending protocol audit.
PostTrainBench: 34.6% · Contributes. Source · Developer-reported · GPT-5.6 Sol; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SimpleQA: 71.6% · Not used.
Not admitted after the September 6 model review: the collected 71.6 for GPT-5.6 Sol on SimpleQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Extended: 60.55% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol; max; codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
JobBench: 45.4% · Contributes. Source · Developer-reported · GPT-5.6 Sol; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
BabyVision (with CI): 88.9% · Contributes. Source · Developer-reported · GPT-5.6 Sol; max; Kimi developer comparison table; with Python tools
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
PerceptionBench: 59.7% · Contributes. Source · Developer-reported · GPT-5.6 Sol; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LVBench (with Memory): 84.2% · Not used.
Not admitted after the September 6 model review: the collected 84.2 for GPT-5.6 Sol on LVBench (with Memory) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
OSWorld-Verified: 83% · Contributes. Source · Developer-reported · GPT-5.6 Sol; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Creative Writing v3: 1964.1 · Contributes. Source · Evaluator-reported · gpt-5.6-sol; published evaluation, effort unspecified
Exact checkpoint on the evaluator-hosted board. A starred model is externally submitted; it is not claimed as independently rerun by the benchmark owner.
MathArena ; ArXivMath Jun 2026: 86.73% · Contributes. Source · Archived source review · gpt-5.6-sol; max effort; MathArena June 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
MathArena ; BrokenArXiv Jun 2026: 67.28% · Contributes. Source · Archived source review · gpt-5.6-sol; max effort; MathArena June 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
AIIQ Composite IQ: 136 · Not used. Source · Evaluator-reported · gpt-5.6-sol; max effort; AIIQ composite snapshot
Held-out composite, never a scoring input
MCP Atlas †: 83.6% · Contributes. Source · Developer-reported · GPT-5.6 Sol; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
OfficeQA Pro †: 63.2% · Contributes. Source · Developer-reported · GPT-5.6 Sol; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
EQ-Bench 4: 1250.3 · Contributes. Source · Evaluator-reported · openai/gpt-5.6-sol; evaluator published configuration, effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CoWorkBench: 71.5% · Not used.
Not admitted after the September 6 model review: the collected 71.5 for GPT-5.6 Sol on CoWorkBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ²-Bench Telecom: 85.1% · Not used.
Not admitted after the September 6 model review: the collected 85.1 for GPT-5.6 Sol on τ²-Bench Telecom still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
PaperBench (Replication Score): 90.5% · Not used.
Not admitted after the September 6 model review: the collected 90.5 for GPT-5.6 Sol on PaperBench (Replication Score) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
QwenReactBench (Elo): 1564 · Not used.
Not admitted after the September 6 model review: the collected 1564 for GPT-5.6 Sol on QwenReactBench (Elo) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ERQA: 70% · Not used.
Not admitted after the September 6 model review: the collected 70 for GPT-5.6 Sol on ERQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vision2Web (Avg. Frontend/Webpage/etc.): 62.1% · Not used.
Not admitted after the September 6 model review: the collected 62.1 for GPT-5.6 Sol on Vision2Web (Avg. Frontend/Webpage/etc.) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MobileWorld: 76.9% · Not used.
Not admitted after the September 6 model review: the collected 76.9 for GPT-5.6 Sol on MobileWorld still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
WebArena-Verified: 69.7% · Not used.
Not admitted after the September 6 model review: the collected 69.7 for GPT-5.6 Sol on WebArena-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathArena ; ArXivLean Jun 2026: 37.5% · Contributes. Source · Archived source review · gpt-5.6-sol; max effort; MathArena June 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Frontier-Bench v0.1 ; Anthropic H2H: 34.4% · Contributes. Source · Archived source review · gpt-5.6-sol; max effort; Anthropic Opus 5 published head-to-head setup
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
FrontierCode v1.1 Main ; Anthropic H2H: 47.5% · Not used.
Not admitted after the September 6 model review: the collected 47.5 for GPT-5.6 Sol on FrontierCode v1.1 Main ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AutomationBench ; Anthropic H2H: 18.1% · Contributes. Source · Archived source review · gpt-5.6-sol; max effort; Anthropic Opus 5 published head-to-head setup
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Harvey Legal Agent Benchmark ; held-out: 2.5% · Not used.
Not admitted after the September 6 model review: the collected 2.5 for GPT-5.6 Sol on Harvey Legal Agent Benchmark ; held-out still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 44.329897% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 32.285714% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
Terminal-Bench 4.0: 37.3% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol; max; Codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
GDP.pdf: 30.7% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (Max reasoning); Surge original evaluation
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 83% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 80.5% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MineBench: 2108 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
OSWorld 2.0: 62.6% · Not used. Source · Developer-reported · GPT-5.6 Sol; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ARC-AGI-1: 97.5% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol; XHigh; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
ARC-AGI-3 (standard harness): 7.78% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
BenchCAD (Vision2Code subset): 83.3% · Not used. Source · Developer-reported · OpenAI full Vision2Code set; not Anthropic subset
System card identifies the OpenAI comparison as the full 17,900-file set, while this row is a modified 1000-file subset. Requires a separate protocol row.
Terminal-Bench Science 0.1: 22.4% · Contributes. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
SRE-Bench (pass@1): 55.9% · Contributes. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
OSWorld 2.0 ; OpenAI H2H (offline subset): 65.7% · Contributes. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Agents' Last Exam ; OpenAI H2H: 53.6% · Contributes. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
ScreenSpot-Pro: 76.9% · Not used. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
HealthBench: 55.6% · Not used. Source · Developer-reported · GPT-5.6 Sol; OpenAI published HealthBench research evaluation; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
HealthBench Hard: 31.1% · Not used. Source · Developer-reported · GPT-5.6 Sol; OpenAI published HealthBench research evaluation; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
HealthBench Professional (length-adjusted): 60.5% · Contributes. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
GeneBench Pro: 32.3% · Not used. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
LifeSciBench: 59.9% · Not used. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
SEC-Bench Pro: 79.1% · Not used. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
ExploitGym: 30.3% · Not used. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Harvey LAB-AA: 87.18% · Not used.
Not admitted after the September 6 model review: the collected 87.18 for GPT-5.6 Sol on Harvey LAB-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EnterpriseOps-Gym-AA: 42.91% · Not used.
Not admitted after the September 6 model review: the collected 42.91 for GPT-5.6 Sol on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 26.111111% · Not used. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
AA-AnalystAgent: 47.5% · Not used. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-AnalystAgent, so every model is scored on its current set.
ITBench-AA: 56.214689% · Not used. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes ITBench-AA, so every model is scored on its current set.
Vending-Bench 2: 9619.368 · Contributes. Source · Evaluator-reported · GPT-5.6 Sol; Andon Labs Vending-Bench 2
Compare reasoning configurations using their mean over runs. Geometric mean, individual best runs, Fireworks Kimi endpoint, Vending Arena, and April DeepSeek Pro are not substituted.
IFBench: 72.7% · Not used.
Not admitted after the September 6 model review: the collected 72.7 for GPT-5.6 Sol on IFBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AndroidWorld: 77.6% · Not used.
Not admitted after the September 6 model review: the collected 77.6 for GPT-5.6 Sol on AndroidWorld still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CWE-bench: 44.2% · Not used.
Not admitted after the September 6 model review: the collected 44.2 for GPT-5.6 Sol on CWE-bench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ZeroBench: 30% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); official main questions
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SimpleVQA: 66.6% · Not used.
Not admitted after the September 6 model review: the collected 66.6 for GPT-5.6 Sol on SimpleVQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathVision (with CI): 97.8% · Contributes. Source · Developer-reported · GPT-5.6 Sol; max; Kimi developer comparison table; with Python tools
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
OmniDocBench 1.5: 85.8% · Contributes. Source · Developer-reported · GPT-5.6 Sol; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Internal Database Migration Tasks ; OpenAI: 42.7% · Contributes. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Internal Design Tasks ; OpenAI: 47.4% · Contributes. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Internal Data Science Tasks ; OpenAI: 30.5% · Contributes. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
OpenScore String Quartets (1 - OMR-NED): 0.19 · Not used. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
MedChemBench (internal): 47.4% · Not used. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
MRCR v2 (8 needles, 512K-1M): 73.8% · Not used. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
ExploitBench (Jun-Aug 2026, 300-turn limit): 5.5% · Not used. Source · Developer-reported · GPT-5.6 Sol; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
Mystery Game Puzzles v1.0.4 ; Epoch: 58% · Contributes. Source · Evaluator-reported · gpt-5.6-sol_max; task version 1.0.4
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
FrontierMath Erdos (68 conjectures) ; Epoch: 0 · Not used. Source · Evaluator-reported · gpt-5.6-sol_max; task version
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
Box Complex Work Eval (overall): 74% · Not used.
Not admitted after the September 6 model review: the collected 74 for GPT-5.6 Sol on Box Complex Work Eval (overall) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCyber (Sep 2026) ; Irregular: 34 · Not used.
Not admitted after the September 6 model review: the collected 34 for GPT-5.6 Sol on FrontierCyber (Sep 2026) ; Irregular still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CyScenarioBench (Sep 2026) ; Irregular: 27% · Not used.
Not admitted after the September 6 model review: the collected 27 for GPT-5.6 Sol on CyScenarioBench (Sep 2026) ; Irregular still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
BioMysteryBench v1 ; Vals: 71.11% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Tax Agent Bench v1 ; Vals: 67.95% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-sol; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
RuneBench (30m, mean ln(1 + XP/min)): 5.896324 · Contributes. Source · Archived source review · gpt56-xhigh
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 2284 · Contributes. Source · Archived source review · GPT-5.6 Sol (Max)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Roboflow Vision Evals (six-task mean, high tier): 81% · Contributes. Source · Archived source review · Require a complete six-task high-tier panel, the highest published benchmark tier. Retain high-tier scores even when lower than low-tier. Do not substitute the best of three repetitions: use the published mean. Each panel enters the fit once; do not also fit its components.
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1617 · Contributes. Source · Archived source review · gpt-5.6-sol-xhigh (codex-harness)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 46.972707 · Not used. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1478.85 · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1478.53 recorded 2026-10-22.
AA-LCR v1.1: 84% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 60.08115% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 27.8% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1609.27 · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1607.18 recorded 2026-10-22.
HLE text-only ; AA harness: 49.490269% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 39.89899% · Contributes. Source · Evaluator-reported · GPT-5.6 Sol (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

GPT-6.1 Terra · 15 used / 16 collected

14 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±17.6 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1402.26 · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1401.68 recorded 2026-10-27.
GDPval-AA v2.1 (Elo): 1476.05 · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1467.61 recorded 2026-10-26.
AutomationBench-AA: 60.1676232726% · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
Terminal-Bench 4.0 ; AA harness: 55.5555555556% · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
SciCode: 55.0925925926% · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
HLE text-only ; AA harness: 46.2465245598% · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
GDP.pdf ; AA harness: 28.6% · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
CritPt: 28% · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
AA-Omniscience: 29.7833333333 · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Restored on 2026-10-28: 29.566667 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. Exact model and effort in the evaluator chart payload.
AA-LCR v1.1: 82.6666666667% · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Exact model and effort in the evaluator chart payload.
MMMU-Pro: 84.6820809249% · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Independent MMMU-Pro aggregate for this effort.
AA Intelligence Index v4.3.2 (score): 48.1061836463 · Not used. Source · Evaluator-reported · GPT-6.1 Terra (max); Artificial Analysis independent evaluation, 2026-10-26; fallback not reported
Restored on 2026-10-28: 50.976519 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Reference composite only, never scored alongside its component evaluations.
Terminal-Bench 4.0: 50.3% · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); GPT-6.1 Terra developer-reported result (openai); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (gpt-6-astra 58.2, gpt-5-6-terra 21.5).
Agents' Last Exam ; OpenAI H2H: 60% · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); GPT-6.1 Terra developer-reported result (openai); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (gpt-6-astra 59.3).
Legal Research Bench ; Vals: 25.96% · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Vals AI Legal Research Bench harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 25.96 ± 3.05, rank 58/86. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 84% · Contributes. Source · Evaluator-reported · GPT-6.1 Terra (max); Vals AI Vibe Code Bench v1.1 harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 84.0 ± 2.87, rank 19/120. Vals notes some benchmarks may use a different provider or parameters.

Muse Spark 1.3 · 25 used / 37 collected

24 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±6.5 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1719.65 · Not used. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 85.76779% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 75.4% · Not used.
Not admitted after the September 6 model review: the collected 75.4 for Muse Spark 1.3 on DeepSWE v1.1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA Intelligence Index v4.1.1 (score): 61 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 94.141414% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (xhigh); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 79% · Not used. Source · Evaluator-reported · Muse Spark 1.3 (xhigh)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
SciCode: 59.722222% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (xhigh); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 49.073216% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
AutomationBench v1.0.6: 43.8% · Not used.
Not admitted after the September 6 model review: the collected 43.8 for Muse Spark 1.3 on AutomationBench v1.0.6 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Omniscience: 24.933333 · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
LiveBench: 81.59% · Not used.
Not admitted after the September 6 model review: the collected 81.59 for Muse Spark 1.3 on LiveBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MMMU-Pro: 82.023121% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (xhigh); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
OSWorld 2.0 ; Anthropic H2H: 66.9% · Not used.
Not admitted after the September 6 model review: the collected 66.9 for Muse Spark 1.3 on OSWorld 2.0 ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
JobBench: 61.2% · Not used.
Not admitted after the September 6 model review: the collected 61.2 for Muse Spark 1.3 on JobBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 52.371134% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 26% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (xhigh); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
MineBench: 1836 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
OSWorld 2.0: 32% · Not used.
Not admitted after the September 6 model review: the collected 32 for Muse Spark 1.3 on OSWorld 2.0 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Harvey LAB-AA: 95.48% · Not used.
Not admitted after the September 6 model review: the collected 95.48 for Muse Spark 1.3 on Harvey LAB-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Tax Agent Bench v1 ; Vals: 71.93% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_3; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
RuneBench (30m, mean ln(1 + XP/min)): 4.58575 · Contributes. Source · Archived source review · muse13
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Roboflow Vision Evals (six-task mean, high tier): 77.733333% · Contributes. Source · Archived source review · Require a complete six-task high-tier panel, the highest published benchmark tier. Retain high-tier scores even when lower than low-tier. Do not substitute the best of three repetitions: use the published mean. Each panel enters the fit once; do not also fit its components.
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1622 · Contributes. Source · Archived source review · muse-spark-1.3 (xHigh)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 48.092311 · Not used. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Restored on 2026-10-28: 36.417204 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1581.34 · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-LCR v1.1: 83% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 57.853438% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 26.6% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1680.99 · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
HLE text-only ; AA harness: 48.702502% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 33.333333% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Code Migration †: 27.58% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (xhigh); Vals AI Code Migration harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 27.58 ± 4.07, rank 45/79. Vals notes some benchmarks may use a different provider or parameters.
Excel Modeling Benchmark ; Vals: 62.71% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (xhigh); Vals AI EMB harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 62.71 ± 2.96, rank 31/76. Vals notes some benchmarks may use a different provider or parameters.
Finance Agent v2 ; Vals: 58.9% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (xhigh); Vals AI Finance Agent (v2) harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 58.9 ± 0.44, rank 8/80. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 40.87% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (xhigh); Vals AI Legal Research Bench harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 40.87 ± 3.42, rank 28/82. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 55% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (xhigh); Vals AI ProofBench v1.1 harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 55.0 ± 5.0, rank 27/50. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 82.86% · Contributes. Source · Evaluator-reported · Muse Spark 1.3 (xhigh); Vals AI Vibe Code Bench v1.1 harness; provider Meta; fallback not reported
Vals AI model page, headline accuracy 82.86 ± 2.9, rank 20/116. Vals notes some benchmarks may use a different provider or parameters.

Grok 4.7 · 22 used / 27 collected

21 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±9.4 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

CursorBench 4.0: 46.3% · Not used. Source · Evaluator-reported · Grok 4.7 (xhigh); CursorBench 4.0 published evaluation; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted.
DeepSWE v1.1: 71% · Contributes. Source · Evaluator-reported · Grok 4.7 (high); xAI launch evaluation; harness undisclosed; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted. The launch table explicitly marks this result high effort, not xhigh.
EEBench: 64% · Not used. Source · Evaluator-reported · Grok 4.7 (xhigh); xAI launch evaluation; harness undisclosed; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted.
Terminal-Bench 4.0: 38% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); xAI launch evaluation; harness undisclosed; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted.
Harvey Legal Agent Benchmark ; xAI release: 19.6% · Not used. Source · Evaluator-reported · Grok 4.7 (xhigh); xAI launch evaluation; harness undisclosed; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted.
HealthBench Professional ; xAI release: 56.7% · Not used. Source · Evaluator-reported · Grok 4.7 (xhigh); xAI launch evaluation; harness undisclosed; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted. Length adjustment is undisclosed; this does not populate the calibrated length-adjusted row.
AA-Briefcase v1.1 (Elo): 1637.72 · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1637.84 recorded 2026-10-27.
GDPval-AA v2.1 (Elo): 1721.59 · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1710.54 recorded 2026-10-21.
AutomationBench-AA: 65.5583514897% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Artificial Analysis independent evaluation, September 21 2026; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted.
Terminal-Bench 4.0 ; AA harness: 25.7575757576% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Artificial Analysis independent evaluation, September 21 2026; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted.
SciCode: 57.7546296296% · Contributes. Source · Evaluator-reported · Grok 4.7 (high); Artificial Analysis independent evaluation, September 21 2026; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted.
Humanity's Last Exam (no tools): 43.1417979611% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Artificial Analysis independent evaluation, September 21 2026; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted.
GDP.pdf ; AA harness: 23.2% · Contributes. Source · Evaluator-reported · Grok 4.7 (high); Artificial Analysis independent evaluation, September 21 2026; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted.
CritPt: 18% · Contributes. Source · Evaluator-reported · Grok 4.7 (high); Artificial Analysis independent evaluation, September 21 2026; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted.
AA-Omniscience: 32.0333333333 · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Artificial Analysis independent evaluation, September 21 2026; retrieved 2026-09-21
Restored on 2026-10-28: 31.233333 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted.
AA-LCR v1.1: 77% · Contributes. Source · Evaluator-reported · Grok 4.7 (high); Artificial Analysis independent evaluation, September 21 2026; retrieved 2026-09-21
Reviewed published aggregate for the exact checkpoint and effort. Retry budget and fallback accounting are undisclosed. Separate benchmark versions and evaluator harnesses are not substituted.
AA Intelligence Index v4.3.2 (score): 46.446551 · Not used. Source · Evaluator-reported · Grok 4.7 (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Restored on 2026-10-28: 34.335012 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Daily evaluator refresh; the evaluator published this benchmark after admission.
HLE text-only ; AA harness: 43.141798% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
BioMysteryBench v1 ; Vals: 69.26% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Vals AI BioMysteryBench harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 69.26 ± 1.33, rank 13/27. Vals notes some benchmarks may use a different provider or parameters.
LegalBench ; Vals: 84.39% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Vals AI LegalBench harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 84.39 ± 0.46, rank 34/151. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 47.12% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Vals AI Legal Research Bench harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 47.12 ± 3.47, rank 14/82. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 49.55% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Vals AI MedCode harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 49.55 ± 2.17, rank 22/107. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 89.38% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Vals AI MedScribe harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 89.38 ± 1.89, rank 7/109. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 26% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Vals AI ProofBench v1.1 harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 26.0 ± 4.41, rank 37/50. Vals notes some benchmarks may use a different provider or parameters.
Public Benefits Bench ; Vals: 65.63% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Vals AI Public Benefits Bench v1.1 harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 65.63 ± 1.24, rank 21/50. Vals notes some benchmarks may use a different provider or parameters.
SAGE ; Vals: 40.79% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Vals AI SAGE harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 40.79 ± 3.35, rank 59/91. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 86.17% · Contributes. Source · Evaluator-reported · Grok 4.7 (xhigh); Vals AI Vibe Code Bench v1.1 harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 86.17 ± 2.18, rank 12/116. Vals notes some benchmarks may use a different provider or parameters.

Kimi K3 · 72 used / 101 collected

65 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.0 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1587.5 · Not used. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 85.018727% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 68.514412% · Contributes. Source · Evaluator-reported · kimi-k3; max; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1488.665886 · Contributes. Source · Evaluator-reported · kimi-k3-max; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 60 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 93.535354% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 74.7% · Not used. Source · Evaluator-reported · Kimi K3 (max)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 54.4% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI Finance Agent v2 harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Excel Modeling Benchmark ; Vals: 66.4% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI Excel Modeling overall evaluation
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
SciCode: 59.490741% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 46.895273% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 44.23% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI Legal Research overall harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Harvey LAB (Vals): 10.83% · Not used.
Not admitted after the September 6 model review: the collected 10.83 for Kimi K3 on Harvey LAB (Vals) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Briefcase Overall Elo: 1499.88 · Not used. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 87.96% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI health evaluation
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
SWE-bench Verified ; Vals: 93.4% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI mini-SWE-agent bash harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
LegalBench ; Vals: 86.02% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI LegalBench evaluation
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
TaxEval v2 ; Vals: 75.72% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI max-compute evaluation
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
KingBench 3: 77.5% · Not used.
Not admitted after the September 6 model review: the collected 77.5 for Kimi K3 on KingBench 3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vals Index: 74.7% · Not used. Source · Evaluator-reported · kimi-k3; max effort; Vals AI index methodology
Held-out composite, never a scoring input
Public Benefits Bench ; Vals: 68.27% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI public-benefits evaluation
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
MedCode ; Vals: 48.88% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI health evaluation
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
AutomationBench v1.0.6: 46.7% · Not used.
Not admitted after the September 6 model review: the collected 46.7 for Kimi K3 on AutomationBench v1.0.6 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Omniscience: 19.7 · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 87.97% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI five-shot MMLU-Pro harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
LiveBench: 79.193048% · Contributes. Source · Evaluator-reported · kimi-k3; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
CorpFin v2 ; Vals: 71.56% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI CorpFin v2 evaluation with Sonnet 4.5 judge
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
MortgageTax ; Vals: 66.34% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI max-compute evaluation
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
SAGE ; Vals: 54.26% · Contributes. Source · Archived source review · kimi-k3; max effort; Vals AI max-compute evaluation
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Code Migration †: 16.1% · Not used.
Not admitted after the September 6 model review: the collected 16.1 for Kimi K3 on Code Migration † still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Toolathlon-Verified: 76.5% · Contributes. Source · Evaluator-reported · Kimi Kimi K3 (max) ✓; Default agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Agents' Last Exam: 28.3% · Contributes. Source · Evaluator-reported · Kimi K3; Max; Kimi Code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Terminal-Bench v3.0: 17.4% · Not used.
Not admitted after the September 6 model review: the collected 17.4 for Kimi K3 on Terminal-Bench v3.0 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
LiveCodeBench: 87.19% · Not used.
Not admitted after the September 6 model review: the collected 87.19 for Kimi K3 on LiveCodeBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CyberGym: 80% · Contributes. Source · Evaluator-reported · Kimi K3; Z.ai GLM-5.3 model-card comparison table; GLM max, other comparator efforts not fully disclosed
CyberGym release-table percentage. Z.ai documents single-run pass@1 on 1,507 tasks, unlimited task timeout, Claude Code 2.1.207 and restricted network for GLM-5.3. Do not transfer those GLM-specific settings to every comparator. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
Humanity's Last Exam w/ tools: 59.8% · Contributes. Source · Evaluator-reported · Kimi K3; Z.ai GLM-5.3 model-card comparison table; GLM max, other comparator efforts not fully disclosed
HLE with tools release-table percentage. Z.ai documents temperature=1, top_p=.95, 163,840 output tokens, 300K context and GPT-5.6 Luna medium judge for its evaluation. Comparator-specific tools and judging settings are not fully disclosed. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
Design Arena (Elo): 1453 · Contributes. Source · Archived source review · kimi-k3; max effort; Design Arena 2026-08-11 rating snapshot
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Vals Multimodal Index: 73.42% · Not used. Source · Evaluator-reported · kimi-k3; max effort; Vals AI multimodal index methodology
Held-out composite, never a scoring input
SWE-Marathon v1.1: 48.1% · Not used.
Not admitted after the September 6 model review: the collected 48.1 for Kimi K3 on SWE-Marathon v1.1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ProgramBench v1 (Raw Pass Rate) ; Vals: 62.768% · Contributes. Source · Archived source review · kimi/kimi-k3 · Vals published row (effort not stated)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
FrontierSWE: 81.2% · Contributes. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierMath v2 Tier 4: 39.02439% · Contributes. Source · Evaluator-reported · kimi-k3_max; task version 2.0.0
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
CursorBench v3.2: 60.8% · Contributes. Source · Evaluator-reported · Kimi K3 Max; Cursor harness
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
APEX-Agents Mean Criteria Passed: 55.4% · Contributes. Source · Evaluator-reported · kimi-k3; max; loop_truncated_tools_agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CharXiv (with CI / RQ): 91.3% · Contributes. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; with Python tools
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
APEX-SWE (Pass@1, Terminus-2): 48% · Contributes. Source · Evaluator-reported · kimi-k3; max; terminus-2
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
BrowseComp: 91.2% · Contributes. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MMMU-Pro: 80.520231% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
ARC-AGI-2: 60.4% · Contributes. Source · Evaluator-reported · Kimi K3; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
OSWorld 2.0 ; Anthropic H2H: 58.3% · Contributes. Source · Archived source review · kimi-k3; max effort; Anthropic Opus 5 published head-to-head setup
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
ExploitBench: 32.2% · Not used.
Inherited comparator metric/scaffold equivalence to Astra Cap Percent is not established; the source flags possible contamination. Excluded pending protocol audit.
PostTrainBench: 36.6% · Contributes. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierCode v1.1 Extended: 58.19% · Contributes. Source · Evaluator-reported · Kimi K3; none; mini-swe-agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
JobBench: 54.3% · Contributes. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
BabyVision (with CI): 85.7% · Contributes. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; with Python tools
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
PerceptionBench: 58.5% · Contributes. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
OSWorld-Verified: 84.8% · Contributes. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Creative Writing v3: 2070.8 · Contributes. Source · Evaluator-reported · kimi-k3; published evaluation, effort unspecified
Exact checkpoint on the evaluator-hosted board. A starred model is externally submitted; it is not claimed as independently rerun by the benchmark owner.
MathArena ; ArXivMath Jun 2026: 72.11% · Contributes. Source · Archived source review · kimi-k3; max effort; MathArena June 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
MathArena ; BrokenArXiv Jun 2026: 51.85% · Contributes. Source · Archived source review · kimi-k3; max effort; MathArena June 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
AIIQ Composite IQ: 122 · Not used. Source · Evaluator-reported · kimi-k3; max effort; AIIQ composite snapshot
Held-out composite, never a scoring input
MCP Atlas †: 84.2% · Contributes. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
OfficeQA Pro †: 63.3% · Contributes. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
EQ-Bench 4: 1339.3 · Contributes. Source · Evaluator-reported · moonshotai/kimi-k3; evaluator published configuration, effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
NL2Repo: 58% · Contributes. Source · Evaluator-reported · Kimi K3; Z.ai GLM-5.3 model-card comparison table; GLM max, other comparator efforts not fully disclosed
NL2Repo release-table percentage. Z.ai documents temperature=1, top_p=1, 64K output, 1M context and anti-cheating judges for its evaluation. Comparator-specific harness settings are not fully disclosed. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
τ²-Bench Telecom: 80.63% · Not used.
Not admitted after the September 6 model review: the collected 80.63 for Kimi K3 on τ²-Bench Telecom still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Main ; Anthropic H2H: 44.2% · Not used.
Not admitted after the September 6 model review: the collected 44.2 for Kimi K3 on FrontierCode v1.1 Main ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathArena ; ArXivMath May 2026: 61.67% · Contributes. Source · Archived source review · kimi-k3; max effort; MathArena May 2026 competition harness
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
τ³-Banking: 45.979381% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 23.428571% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
GDP.pdf: 19% · Contributes. Source · Evaluator-reported · Kimi K3 (Max reasoning); Surge original evaluation
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 87% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Vals AI ProofBench v1.1 harness; provider Moonshot AI; fallback not reported
Vals AI model page, headline accuracy 87.0 ± 3.38, rank 12/52. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 84.97% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Vals AI Vibe Code Bench v1.1 harness; provider Moonshot AI; fallback not reported
Vals AI model page, headline accuracy 84.97 ± 2.75, rank 16/120. Vals notes some benchmarks may use a different provider or parameters.
MineBench: 1812 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
ARC-AGI-1: 94.5% · Contributes. Source · Evaluator-reported · Kimi K3; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
Harvey LAB-AA: 94.64% · Not used.
Not admitted after the September 6 model review: the collected 94.64 for Kimi K3 on Harvey LAB-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EnterpriseOps-Gym-AA: 45.33% · Not used.
Not admitted after the September 6 model review: the collected 45.33 for Kimi K3 on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 38.333333% · Not used. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
AA-AnalystAgent: 38.75% · Not used. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-AnalystAgent, so every model is scored on its current set.
ITBench-AA: 47.693032% · Not used. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes ITBench-AA, so every model is scored on its current set.
DeepSearchQA: 95% · Not used. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vending-Bench 2: 5165.035 · Contributes. Source · Evaluator-reported · Kimi K3 (Moonshot); Andon Labs Vending-Bench 2
Compare reasoning configurations using their mean over runs. Geometric mean, individual best runs, Fireworks Kimi endpoint, Vending Arena, and April DeepSeek Pro are not substituted.
IFBench: 74.4% · Not used.
Not admitted after the September 6 model review: the collected 74.4 for Kimi K3 on IFBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CWE-bench: 22.5% · Not used.
Not admitted after the September 6 model review: the collected 22.5 for Kimi K3 on CWE-bench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ZeroBench: 23% · Not used.
Not admitted after the September 6 model review: the collected 23 for Kimi K3 on ZeroBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathVision (with CI): 97.8% · Contributes. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; with Python tools
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
OmniDocBench 1.5: 91.1% · Contributes. Source · Developer-reported · Kimi K3; max; Kimi developer comparison table; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Mystery Game Puzzles v1.0.4 ; Epoch: 26% · Contributes. Source · Evaluator-reported · kimi-k3_max; task version 1.0.4
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
BioMysteryBench v1 ; Vals: 72.59% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Vals AI BioMysteryBench harness; provider Moonshot AI; fallback not reported
Vals AI model page, headline accuracy 72.59 ± 1.61, rank 10/29. Vals notes some benchmarks may use a different provider or parameters.
Tax Agent Bench v1 ; Vals: 68.672% · Not used.
Not admitted after the September 6 model review: the collected 68.672 for Kimi K3 on Tax Agent Bench v1 ; Vals still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
RuneBench (30m, mean ln(1 + XP/min)): 3.753577 · Contributes. Source · Archived source review · kimi3
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 1932 · Contributes. Source · Archived source review · Kimi K3 (Max)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1674 · Contributes. Source · Archived source review · kimi-k3-max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 43.593823 · Not used. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1498.75 · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1499.38 recorded 2026-10-21.
AA-LCR v1.1: 88.666667% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 58.272978% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 22% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1534.14 · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1532.59 recorded 2026-10-21.
HLE text-only ; AA harness: 46.895273% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 12.626263% · Contributes. Source · Evaluator-reported · Kimi K3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

MiniMax-M3.1-Flash · 16 used / 17 collected

16 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±15.5 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1473.37 · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (none); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1473.9 recorded 2026-10-19.
GDPval-AA v2.1 (Elo): 1465.71 · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (none); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1457.69 recorded 2026-10-19.
AutomationBench-AA: 72.5218095753% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (none); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
Terminal-Bench 4.0 ; AA harness: 25.7575757576% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (none); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
SciCode: 53.3564814815% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (none); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
HLE text-only ; AA harness: 47.9610750695% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (none); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
GDP.pdf ; AA harness: 20.8% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (none); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
CritPt: 22.2857142857% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (none); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
AA-Omniscience: 15.55 · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (none); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Restored on 2026-10-28: -6.4 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. Exact model and effort in the evaluator chart payload.
AA-LCR v1.1: 79.6666666667% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (none); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Exact model and effort in the evaluator chart payload.
MMMU-Pro: 79.1329479769% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (none); Artificial Analysis independent evaluation, 2026-10-19; fallback not reported
Independent MMMU-Pro aggregate for this effort.
AA Intelligence Index v4.3.2 (score): 42.705748 · Not used. Source · Evaluator-reported · MiniMax-M3.1-Flash (none); Artificial Analysis independent evaluation, 2026-10-20; fallback not reported
Restored on 2026-10-28: 28.909521 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Daily evaluator refresh; replaces 42.3811052132 recorded 2026-10-19.
NL2Repo: 61.2% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (effort not stated); MiniMax-M3.1-Flash developer-reported result (minimax); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (deepseek-v4-1-flash-1017 68.2, kimi-k3 58). Value read from an image of the lab table. Lab footnote: Claude Code 2.1.233 harness, max 512K context; reward-hacking cases scored 0; two inconsistent tasks corrected.
CyberGym: 82.2% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (effort not stated); MiniMax-M3.1-Flash developer-reported result (minimax); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (glm-5-3 84.5, gpt-5-6-sol 83.6). Value read from an image of the lab table. Lab footnote: Claude Code 2.1.263 harness, max 1M context. Body text rounds this to 82%.
BabyVision (with CI): 91% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (effort not stated); MiniMax-M3.1-Flash developer-reported result (minimax); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (deepseek-v4-1-flash-1017 92.5, gpt-5-6-sol 88.9). Value read from an image of the lab table. Lab row label: BabyVision (w/tools). Lab footnote: Claude Code 2.1.285, minimum image side length 1,500 pixels.
Legal Research Bench ; Vals: 34.13% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (max); Vals AI Legal Research Bench harness; provider MiniMax; fallback not reported
Vals AI model page, headline accuracy 34.13 ± 3.3, rank 44/82. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 73.94% · Contributes. Source · Evaluator-reported · MiniMax-M3.1-Flash (max); Vals AI Vibe Code Bench v1.1 harness; provider MiniMax; fallback not reported
Vals AI model page, headline accuracy 73.94 ± 4.25, rank 38/116. Vals notes some benchmarks may use a different provider or parameters.

DeepSeek V4.1-Flash · 30 used / 41 collected

26 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±4.7 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GPQA Diamond: 90.9% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim.
Humanity's Last Exam (no tools): 36.8% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim. Full HLE 36.8; the separately reported 39.1 is text-only and is not substituted.
HLE text-only ; DeepSeek V4.1 release: 39.1% · Not used. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim. Retained as a separate protocol; does not contribute to the current frozen score.
Codeforces Rating ; DeepSeek V4.1 release: 3471 · Not used. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim. Retained as a separate protocol; does not contribute to the current frozen score.
MathArena Apex: 65.6% · Not used. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim. Retained as a separate protocol; does not contribute to the current frozen score.
Terminal-Bench 2.1: 90.6% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim.
Terminal-Bench v3.0: 30% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim.
Terminal-Bench 4.0: 31.2% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim.
DeepSWE v1.1: 74.2% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim.
ProgramBench ; DeepSeek V4.1 release: 20.3% · Not used. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim. Retained as a separate protocol; does not contribute to the current frozen score.
NL2Repo: 65.4% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim.
CyberGym: 88.1% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim.
SEC-Bench Pro ; DeepSeek V4.1 release: 62.8% · Not used. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim. Retained as a separate protocol; does not contribute to the current frozen score.
ExploitGym: 15.3% · Not used. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim.
Humanity's Last Exam w/ tools: 63.9% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim.
Automation-Bench ; DeepSeek V4.1 release: 54.8% · Not used. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim. Retained as a separate protocol; does not contribute to the current frozen score.
Agents' Last Exam: 31.8% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim.
Chartography (w/tools): 78.9% · Not used. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim. Retained as a separate protocol; does not contribute to the current frozen score.
BabyVision (w/tools): 89.6% · Not used. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim. Retained as a separate protocol; does not contribute to the current frozen score.
ZeroBench-main (w/tools): 49% · Not used. Source · Evaluator-reported · DeepSeek V4.1-Flash, September 10 release; sole published setting (effort undisclosed)
Developer-reported release-table aggregate, corroborated against the supplied screenshot. Effort, harness, retry budget and fallback accounting are undisclosed; no maximum-effort claim. Retained as a separate protocol; does not contribute to the current frozen score.
AA Intelligence Index v4.3.2 (score): 39.456167 · Not used. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1418.83 · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1419.38 recorded 2026-10-21.
AA-LCR v1.1: 84% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Omniscience: -5.3 · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 68.890978% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
CritPt: 14.285714% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 12.8% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1600 · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
HLE text-only ; AA harness: 39.249305% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
MMMU-Pro: 76.99422% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
SciCode: 51.851852% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 26.767677% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
BioMysteryBench v1 ; Vals: 67.78% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Vals AI BioMysteryBench harness; provider DeepSeek; fallback not reported
Vals AI model page, headline accuracy 67.78 ± 1.11, rank 16/27. Vals notes some benchmarks may use a different provider or parameters.
LegalBench ; Vals: 83.28% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Vals AI LegalBench harness; provider DeepSeek; fallback not reported
Vals AI model page, headline accuracy 83.28 ± 0.46, rank 56/151. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 41.35% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Vals AI Legal Research Bench harness; provider DeepSeek; fallback not reported
Vals AI model page, headline accuracy 41.35 ± 3.42, rank 26/82. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 41.17% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Vals AI MedCode harness; provider DeepSeek; fallback not reported
Vals AI model page, headline accuracy 41.17 ± 2.04, rank 57/107. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 85.5% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Vals AI MedScribe harness; provider DeepSeek; fallback not reported
Vals AI model page, headline accuracy 85.5 ± 1.92, rank 27/109. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 54% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Vals AI ProofBench v1.1 harness; provider DeepSeek; fallback not reported
Vals AI model page, headline accuracy 54.0 ± 5.01, rank 28/50. Vals notes some benchmarks may use a different provider or parameters.
Public Benefits Bench ; Vals: 64.28% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Vals AI Public Benefits Bench v1.1 harness; provider DeepSeek; fallback not reported
Vals AI model page, headline accuracy 64.28 ± 1.25, rank 23/50. Vals notes some benchmarks may use a different provider or parameters.
SAGE ; Vals: 47.88% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Vals AI SAGE harness; provider DeepSeek; fallback not reported
Vals AI model page, headline accuracy 47.88 ± 3.44, rank 29/91. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 84.74% · Contributes. Source · Evaluator-reported · DeepSeek V4.1-Flash (max); Vals AI Vibe Code Bench v1.1 harness; provider DeepSeek; fallback not reported
Vals AI model page, headline accuracy 84.74 ± 2.89, rank 17/116. Vals notes some benchmarks may use a different provider or parameters.

Gemini 3.8 Flash · 45 used / 60 collected

43 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.3 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1464.96 · Not used. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 87.640449% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 73.825503% · Contributes. Source · Evaluator-reported · gemini-3-8-flash; high; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1494 · Not used.
Not admitted after the September 6 model review: the collected 1494 for Gemini 3.8 Flash on LMArena (Chatbot Arena) ; Text still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA Intelligence Index v4.1.1 (score): 59 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 95.252525% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 82% · Not used. Source · Evaluator-reported · Gemini 3.8 Flash (high)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 61.44% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 72.2% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 56.597222% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 47.822057% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 38.94% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 10% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1201.13 · Not used. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 84.5% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 80% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 86.99% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 74.45% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vals Index: 62.25% · Not used.
Held-out composite, never a scoring input
Public Benefits Bench ; Vals: 65.29% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 48.13% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Omniscience: 29.55 · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 90.22% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 75.83% · Not used.
Not admitted after the September 6 model review: the collected 75.83 for Gemini 3.8 Flash on LiveBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MortgageTax ; Vals: 65.34% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SAGE ; Vals: 35.06% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Code Migration †: 36.55% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveCodeBench: 89.48% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProgramBench v1 (Raw Pass Rate) ; Vals: 71.899% · Contributes. Source · Archived source review · google/gemini-3.8-flash · reasoning_effort=high
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
SkillsBench †: 57.98% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CursorBench v3.2: 69.2% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash High; Cursor harness
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Pro: 61.6% · Not used.
Not admitted after the September 6 model review: the collected 61.6 for Gemini 3.8 Flash on SWE-bench Pro still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CharXiv (with CI / RQ): 86.2% · Not used.
Not admitted after the September 6 model review: the collected 86.2 for Gemini 3.8 Flash on CharXiv (with CI / RQ) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MMMU-Pro: 85.606936% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
FrontierCode v1.1 Extended: 53.45% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash; medium; chisel
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LVBench (with Memory): 87.8% · Not used.
Not admitted after the September 6 model review: the collected 87.8 for Gemini 3.8 Flash on LVBench (with Memory) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathArena ; ArXivMath Jun 2026: 68.03% · Not used.
Not admitted after the September 6 model review: the collected 68.03 for Gemini 3.8 Flash on MathArena ; ArXivMath Jun 2026 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathArena ; BrokenArXiv Jun 2026: 20.83% · Not used.
Not admitted after the September 6 model review: the collected 20.83 for Gemini 3.8 Flash on MathArena ; BrokenArXiv Jun 2026 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Main ; Anthropic H2H: 43.6% · Not used.
Not admitted after the September 6 model review: the collected 43.6 for Gemini 3.8 Flash on FrontierCode v1.1 Main ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 45.773196% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (medium); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 18.285714% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
Terminal-Bench 4.0: 19.1% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash; high; mini-SWE-agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 48% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 78.65% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MineBench: 1917 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
HealthBench Professional (length-adjusted): 52.1% · Contributes. Source · Developer-reported · Gemini 3.8 Flash; publisher-reported maximum across reasoning efforts; OpenAI research/API evaluation
Developer-reported maximum across reasoning modes. Exact selected mode is not individually disclosed. Research prompts/tools may differ from production; protocol-specific rows remain separate.
BioMysteryBench v1 ; Vals: 62.22% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Tax Agent Bench v1 ; Vals: 66.77% · Contributes. Source · Evaluator-reported · google_gemini-3.8-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
RuneBench (30m, mean ln(1 + XP/min)): 5.962732 · Contributes. Source · Archived source review · gemini38flash
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 1811 · Contributes. Source · Archived source review · Gemini 3.8 Flash
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Roboflow Vision Evals (six-task mean, high tier): 86.516667% · Contributes. Source · Archived source review · Require a complete six-task high-tier panel, the highest published benchmark tier. Retain high-tier scores even when lower than low-tier. Do not substitute the best of three repetitions: use the published mean. Each panel enters the fit once; do not also fit its components.
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1567 · Contributes. Source · Archived source review · gemini-3.8-flash-high
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 40.926232 · Not used. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Restored on 2026-10-28: 30.546812 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1202.78 · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1202.25 recorded 2026-10-27.
AA-LCR v1.1: 81.333333% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 59.930094% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 21% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1445.4 · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1435.28 recorded 2026-10-21.
HLE text-only ; AA harness: 47.822057% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 19.69697% · Contributes. Source · Evaluator-reported · Gemini 3.8 Flash (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

Grok 4.6 · 46 used / 67 collected

42 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.3 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1665.73 · Not used. Source · Evaluator-reported · Grok 4.6 (xhigh); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 88.389513% · Contributes. Source · Evaluator-reported · Grok 4.6 (high); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 67.477876% · Contributes. Source · Evaluator-reported · grok-4-6; medium; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1461.15072 · Contributes. Source · Evaluator-reported · grok-4.6-high; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 61 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 94.949495% · Contributes. Source · Evaluator-reported · Grok 4.6 (high); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 75% · Not used. Source · Evaluator-reported · Grok 4.6 (high)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 53.68% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI Finance Agent (v2) harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 53.68 ± 0.67, rank 33/81. Vals notes some benchmarks may use a different provider or parameters.
Excel Modeling Benchmark ; Vals: 62.73% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI EMB harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 62.73 ± 3.08, rank 30/76. Vals notes some benchmarks may use a different provider or parameters.
SciCode: 56.481481% · Contributes. Source · Evaluator-reported · Grok 4.6 (high); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 44.068582% · Contributes. Source · Evaluator-reported · Grok 4.6 (xhigh); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 48.08% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI Legal Research Bench harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 48.08 ± 3.47, rank 10/86. Vals notes some benchmarks may use a different provider or parameters.
Harvey LAB (Vals): 15.8% · Not used.
Not admitted after the September 6 model review: the collected 15.8 for Grok 4.6 High on Harvey LAB (Vals) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Briefcase Overall Elo: 1549.11 · Not used. Source · Evaluator-reported · Grok 4.6 (xhigh); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 86.53% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI MedScribe harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 86.53 ± 1.96, rank 22/109. Vals notes some benchmarks may use a different provider or parameters.
SWE-bench Verified ; Vals: 95.6% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI SWE-bench harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 95.6 ± 0.92, rank 4/88. Vals notes some benchmarks may use a different provider or parameters.
LegalBench ; Vals: 86.31% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI LegalBench harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 86.31 ± 0.42, rank 14/151. Vals notes some benchmarks may use a different provider or parameters.
TaxEval v2 ; Vals: 71.1% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI TaxEval v2 harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 71.1 ± 0.9, rank 87/145. Vals notes some benchmarks may use a different provider or parameters.
Vals Index: 71.1% · Not used.
Held-out composite, never a scoring input
Public Benefits Bench ; Vals: 66.85% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI Public Benefits Bench v1.1 harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 66.85 ± 1.23, rank 17/50. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 44.71% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI MedCode harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 44.71 ± 2.26, rank 40/107. Vals notes some benchmarks may use a different provider or parameters.
AA-Omniscience: 30.483333 · Contributes. Source · Evaluator-reported · Grok 4.6 (high); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 89.4% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI MMLU Pro harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 89.4 ± 0.3, rank 11/138. Vals notes some benchmarks may use a different provider or parameters.
LiveBench: 78.042% · Contributes. Source · Evaluator-reported · grok-4.6; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
MortgageTax ; Vals: 64.19% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI MortgageTax harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 64.19 ± 0.95, rank 47/98. Vals notes some benchmarks may use a different provider or parameters.
SAGE ; Vals: 28.9% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI SAGE harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 28.9 ± 3.08, rank 84/91. Vals notes some benchmarks may use a different provider or parameters.
Code Migration †: 44.57% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI Code Migration harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 44.57 ± 4.48, rank 20/79. Vals notes some benchmarks may use a different provider or parameters.
Terminal-Bench v3.0: 26.5% · Contributes. Source · Evaluator-reported · Grok 4.6 (high); Grok Build
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveCodeBench: 88.22% · Not used.
Not admitted after the September 6 model review: the collected 88.22 for Grok 4.6 High on LiveCodeBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CyberGym: 79.7% · Not used.
Not admitted after the September 6 model review: the collected 79.7 for Grok 4.6 High on CyberGym still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
SWE-Marathon v1.1: 31.9% · Not used.
Not admitted after the September 6 model review: the collected 31.9 for Grok 4.6 High on SWE-Marathon v1.1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CursorBench v3.2: 70.8% · Contributes. Source · Evaluator-reported · Grok 4.6 Extra High; Cursor harness
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
APEX-Agents Mean Criteria Passed: 57.5% · Contributes. Source · Evaluator-reported · grok-4-6; high; loop_truncated_tools_agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
APEX-SWE (Pass@1, Terminus-2): 56.4% · Contributes. Source · Evaluator-reported · grok-4-6; high; terminus-2
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ARC-AGI-2: 67.1% · Not used.
Not admitted after the September 6 model review: the collected 67.1 for Grok 4.6 High on ARC-AGI-2 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
SimpleQA: 53% · Not used.
Not admitted after the September 6 model review: the collected 53 for Grok 4.6 High on SimpleQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Extended: 61.31% · Contributes. Source · Evaluator-reported · Grok 4.6; high; grok-build
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
OfficeQA Pro †: 63.2% · Not used.
Not admitted after the September 6 model review: the collected 63.2 for Grok 4.6 High on OfficeQA Pro † still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Main ; Anthropic H2H: 48% · Not used.
Not admitted after the September 6 model review: the collected 48 for Grok 4.6 High on FrontierCode v1.1 Main ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 50.721649% · Contributes. Source · Evaluator-reported · Grok 4.6 (high); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 19.714286% · Contributes. Source · Evaluator-reported · Grok 4.6 (xhigh); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
Terminal-Bench 4.0: 20.3% · Contributes. Source · Evaluator-reported · Grok 4.6; high; Grok Build
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
GDP.pdf: 17.2% · Contributes. Source · Evaluator-reported · Grok 4.6 (xHigh reasoning); Surge original evaluation
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 51% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI ProofBench v1.1 harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 51.0 ± 5.02, rank 30/52. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 76.24% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI Vibe Code Bench v1.1 harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 76.24 ± 3.82, rank 37/120. Vals notes some benchmarks may use a different provider or parameters.
MineBench: 1931 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
ARC-AGI-1: 87.5% · Not used.
Not admitted after the September 6 model review: the collected 87.5 for Grok 4.6 High on ARC-AGI-1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ARC-AGI-3 (standard harness): 2.11% · Not used.
Not admitted after the September 6 model review: the collected 2.11 for Grok 4.6 High on ARC-AGI-3 (standard harness) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EnterpriseOps-Gym-AA: 48.34% · Not used.
Not admitted after the September 6 model review: the collected 48.34 for Grok 4.6 High on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 12.222222% · Not used. Source · Evaluator-reported · Grok 4.6 (high); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
AA-AnalystAgent: 41.25% · Not used. Source · Evaluator-reported · Grok 4.6 (high); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-AnalystAgent, so every model is scored on its current set.
Vending-Bench 2: 9047.0325 · Contributes. Source · Evaluator-reported · Grok 4.6; Andon Labs Vending-Bench 2
Compare reasoning configurations using their mean over runs. Geometric mean, individual best runs, Fireworks Kimi endpoint, Vending Arena, and April DeepSeek Pro are not substituted.
CWE-bench: 38.2% · Contributes. Source · Evaluator-reported · Grok 4.6 High; high reasoning; grok-cli
Published high effort matches this model configuration. Other models pinned to max/xhigh are not approved from this high-only board.
ZeroBench: 17% · Contributes. Source · Evaluator-reported · Grok 4.6 (xhigh); official main questions
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
BioMysteryBench v1 ; Vals: 72.22% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Vals AI BioMysteryBench harness; provider SpaceXAI; fallback not reported
Vals AI model page, headline accuracy 72.22 ± 0.0, rank 11/29. Vals notes some benchmarks may use a different provider or parameters.
Tax Agent Bench v1 ; Vals: 70.788% · Not used.
Not admitted after the September 6 model review: the collected 70.788 for Grok 4.6 High on Tax Agent Bench v1 ; Vals still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
RuneBench (30m, mean ln(1 + XP/min)): 5.621351 · Contributes. Source · Archived source review · grok46-xhigh
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 2099 · Contributes. Source · Archived source review · Grok 4.6 (xHigh)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1625 · Contributes. Source · Archived source review · grok-4.6-high
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 44.311307 · Not used. Source · Evaluator-reported · Grok 4.6 High (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1530.65 · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1532.04 recorded 2026-10-22.
AA-LCR v1.1: 80.333333% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 66.678496% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 17% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1618.44 · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1616.9 recorded 2026-10-22.
HLE text-only ; AA harness: 42.910102% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 21.212121% · Contributes. Source · Evaluator-reported · Grok 4.6 High (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

GLM-5.3 · 46 used / 64 collected

42 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.3 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1679.47 · Not used. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 83.895131% · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 68.957871% · Contributes. Source · Evaluator-reported · glm-5-3; max; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1482.03722 · Contributes. Source · Evaluator-reported · glm-5.3-max; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 60 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 91.717172% · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 76.33% · Not used. Source · Evaluator-reported · GLM-5.3 (max)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 55.84% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 56.34% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 59.027778% · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 42.261353% · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 49.04% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 8.33% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1518.58 · Not used. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 88.81% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 95.4% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 84.84% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 72.36% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
KingBench 3: 91.25% · Not used.
Not admitted after the September 6 model review: the collected 91.25 for GLM-5.3 on KingBench 3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Public Benefits Bench ; Vals: 68.54% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 42.86% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AutomationBench v1.0.6: 48.2% · Not used.
Not admitted after the September 6 model review: the collected 48.2 for GLM-5.3 on AutomationBench v1.0.6 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Omniscience: 14.3 · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 86.77% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 76.13781% · Contributes. Source · Evaluator-reported · glm-5.3; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
Code Migration †: 44.22% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Toolathlon-Verified: 73% · Not used.
Not admitted after the September 6 model review: the collected 73 for GLM-5.3 on Toolathlon-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Agents' Last Exam: 28.5% · Not used.
Not admitted after the September 6 model review: the collected 28.5 for GLM-5.3 on Agents' Last Exam still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Terminal-Bench v3.0: 32.4% · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Claude Code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveCodeBench: 80.53% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CyberGym: 84.5% · Contributes. Source · Evaluator-reported · GLM-5.3; Z.ai GLM-5.3 model-card comparison table; GLM max, other comparator efforts not fully disclosed
CyberGym release-table percentage. Z.ai documents single-run pass@1 on 1,507 tasks, unlimited task timeout, Claude Code 2.1.207 and restricted network for GLM-5.3. Do not transfer those GLM-specific settings to every comparator. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
Humanity's Last Exam w/ tools: 62.5% · Contributes. Source · Evaluator-reported · GLM-5.3; Z.ai GLM-5.3 model-card comparison table; GLM max, other comparator efforts not fully disclosed
HLE with tools release-table percentage. Z.ai documents temperature=1, top_p=.95, 163,840 output tokens, 300K context and GPT-5.6 Luna medium judge for its evaluation. Comparator-specific tools and judging settings are not fully disclosed. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
SWE-Marathon v1.1: 42.5% · Not used.
Not admitted after the September 6 model review: the collected 42.5 for GLM-5.3 on SWE-Marathon v1.1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ProgramBench v1 (Raw Pass Rate) ; Vals: 66.252% · Contributes. Source · Archived source review · zai/glm-5.3 · reasoning_effort=max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
SkillsBench †: 47.51% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierSWE: 78.1% · Not used.
Not admitted after the September 6 model review: the collected 78.1 for GLM-5.3 on FrontierSWE still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierMath v2 Tier 4: 29.27% · Not used.
Not admitted after the September 6 model review: the collected 29.27 for GLM-5.3 on FrontierMath v2 Tier 4 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ExploitBench: 54.4% · Not used.
Inherited comparator metric/scaffold equivalence to Astra Cap Percent is not established; the source flags possible contamination. Excluded pending protocol audit.
PostTrainBench: 39.8% · Not used.
Not admitted after the September 6 model review: the collected 39.8 for GLM-5.3 on PostTrainBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Creative Writing v3: 2062.4 · Contributes. Source · Evaluator-reported · *GLM-5.3; published evaluation, effort unspecified
Exact checkpoint on the evaluator-hosted board. A starred model is externally submitted; it is not claimed as independently rerun by the benchmark owner.
NL2Repo: 58% · Contributes. Source · Evaluator-reported · GLM-5.3; Z.ai GLM-5.3 model-card comparison table; GLM max, other comparator efforts not fully disclosed
NL2Repo release-table percentage. Z.ai documents temperature=1, top_p=1, 64K output, 1M context and anti-cheating judges for its evaluation. Comparator-specific harness settings are not fully disclosed. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
τ³-Banking: 50.309278% · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 19.142857% · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
Terminal-Bench 4.0: 41.8% · Contributes. Source · Evaluator-reported · GLM-5.3; max; Claude Code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 49% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 78.13% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MineBench: 1850 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
EnterpriseOps-Gym-AA: 36.44% · Not used.
Not admitted after the September 6 model review: the collected 36.44 for GLM-5.3 on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 48.333333% · Not used. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
Vending-Bench 2: 8163.61 · Contributes. Source · Evaluator-reported · GLM-5.3; Andon Labs Vending-Bench 2
Compare reasoning configurations using their mean over runs. Geometric mean, individual best runs, Fireworks Kimi endpoint, Vending Arena, and April DeepSeek Pro are not substituted.
CWE-bench: 31.1% · Contributes. Source · Evaluator-reported · GLM-5.3; high; opencode
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Mystery Game Puzzles v1.0.4 ; Epoch: 33% · Not used.
Not admitted after the September 6 model review: the collected 33 for GLM-5.3 on Mystery Game Puzzles v1.0.4 ; Epoch still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Tax Agent Bench v1 ; Vals: 73.09% · Contributes. Source · Evaluator-reported · zai_glm-5.3; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
RuneBench (30m, mean ln(1 + XP/min)): 5.001589 · Contributes. Source · Archived source review · glm53
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 1786 · Contributes. Source · Archived source review · GLM-5.3 (Max)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1609 · Contributes. Source · Archived source review · glm-5.3-max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 44.777392 · Not used. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Restored on 2026-10-28: 31.653382 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1504.57 · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1503.99 recorded 2026-10-27.
AA-LCR v1.1: 79.666667% · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 62.202865% · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 11.2% · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1657.93 · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1648.98 recorded 2026-10-21.
HLE text-only ; AA harness: 42.261353% · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 41.919192% · Contributes. Source · Evaluator-reported · GLM-5.3 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

Gemini 3.7 Flash · 44 used / 70 collected

40 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.3 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1435.34 · Not used. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 85.76779% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 65.486726% · Contributes. Source · Evaluator-reported · gemini-3-7-flash; medium; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1490 · Not used.
Not admitted after the September 6 model review: the collected 1490 for Gemini 3.7 Flash on LMArena (Chatbot Arena) ; Text still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA Intelligence Index v4.1.1 (score): 56 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 94.545455% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 80% · Not used. Source · Evaluator-reported · Gemini 3.7 Flash (high)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 59.04% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 71.33% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 59.837963% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (medium); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 47.868397% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 34.62% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 8.75% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1116.43 · Not used. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 83.94% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 80.8% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 87.26% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 74.73% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vals Index: 59.31% · Not used. Source · Evaluator-reported · gemini-3.7-flash; high effort; Vals AI index methodology
Held-out composite, never a scoring input
MedCode ; Vals: 53.39% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AutomationBench v1.0.6: 30.44% · Not used.
Not admitted after the September 6 model review: the collected 30.44 for Gemini 3.7 Flash on AutomationBench v1.0.6 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Omniscience: 26.483333 · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 90.12% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 78.8% · Not used.
Not admitted after the September 6 model review: the collected 78.8 for Gemini 3.7 Flash on LiveBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MortgageTax ; Vals: 66.65% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SAGE ; Vals: 49.23% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Code Migration †: 34.8% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Agents' Last Exam: 26.3% · Not used.
Not admitted after the September 6 model review: the collected 26.3 for Gemini 3.7 Flash on Agents' Last Exam still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Terminal-Bench v3.0: 14.9% · Not used.
Not admitted after the September 6 model review: the collected 14.9 for Gemini 3.7 Flash on Terminal-Bench v3.0 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
LiveCodeBench: 88.65% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-Marathon v1.1: 22.5% · Not used.
Not admitted after the September 6 model review: the collected 22.5 for Gemini 3.7 Flash on SWE-Marathon v1.1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ProgramBench v1 (Raw Pass Rate) ; Vals: 68.663% · Contributes. Source · Archived source review · google/gemini-3.7-flash · reasoning_effort=high
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
FrontierMath v2 Tier 4: 36.59% · Not used.
Not admitted after the September 6 model review: the collected 36.59 for Gemini 3.7 Flash on FrontierMath v2 Tier 4 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CursorBench v3.2: 61.6% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash High; Cursor harness
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Pro: 60.4% · Not used.
Not admitted after the September 6 model review: the collected 60.4 for Gemini 3.7 Flash on SWE-bench Pro still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CharXiv (with CI / RQ): 88.7% · Not used.
Not admitted after the September 6 model review: the collected 88.7 for Gemini 3.7 Flash on CharXiv (with CI / RQ) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MMMU-Pro: 85.491329% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
ARC-AGI-2: 84.6% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash; High; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
LVBench (with Memory): 85.4% · Not used.
Not admitted after the September 6 model review: the collected 85.4 for Gemini 3.7 Flash on LVBench (with Memory) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Creative Writing v3: 1725.7 · Not used.
Not admitted after the September 6 model review: the collected 1725.7 for Gemini 3.7 Flash on Creative Writing v3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathArena ; ArXivMath Jun 2026: 63.27% · Not used.
Not admitted after the September 6 model review: the collected 63.27 for Gemini 3.7 Flash on MathArena ; ArXivMath Jun 2026 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathArena ; BrokenArXiv Jun 2026: 11.11% · Not used.
Not admitted after the September 6 model review: the collected 11.11 for Gemini 3.7 Flash on MathArena ; BrokenArXiv Jun 2026 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Main ; Anthropic H2H: 43.6% · Not used.
Not admitted after the September 6 model review: the collected 43.6 for Gemini 3.7 Flash on FrontierCode v1.1 Main ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 35.463918% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (medium); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 14.285714% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
Terminal-Bench 4.0: 11.2% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash; high; mini-SWE-agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
GDP.pdf: 23.8% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (High reasoning); Surge original evaluation
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 58% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 70.39% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MineBench: 1867 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
ARC-AGI-1: 95.5% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash; High; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
Harvey LAB-AA: 90.66% · Not used.
Not admitted after the September 6 model review: the collected 90.66 for Gemini 3.7 Flash on Harvey LAB-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EnterpriseOps-Gym-AA: 50.4% · Not used.
Not admitted after the September 6 model review: the collected 50.4 for Gemini 3.7 Flash on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 15% · Not used. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
AA-AnalystAgent: 60% · Not used. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-AnalystAgent, so every model is scored on its current set.
CWE-bench: 44% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash; high reasoning; Antigravity
Published high effort matches this model configuration. Other models pinned to max/xhigh are not approved from this high-only board.
Mystery Game Puzzles v1.0.4 ; Epoch: 37% · Not used.
Not admitted after the September 6 model review: the collected 37 for Gemini 3.7 Flash on Mystery Game Puzzles v1.0.4 ; Epoch still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Tax Agent Bench v1 ; Vals: 57.66% · Contributes. Source · Evaluator-reported · google_gemini-3.7-flash; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
RuneBench (30m, mean ln(1 + XP/min)): 5.963375 · Contributes. Source · Archived source review · gemini37flash
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 1854 · Contributes. Source · Archived source review · Gemini 3.7 Flash
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Roboflow Vision Evals (six-task mean, high tier): 86.216667% · Contributes. Source · Archived source review · Require a complete six-task high-tier panel, the highest published benchmark tier. Retain high-tier scores even when lower than low-tier. Do not substitute the best of three repetitions: use the published mean. Each panel enters the fit once; do not also fit its components.
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1587 · Contributes. Source · Archived source review · gemini-3.7-flash-high
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 39.059457 · Not used. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1113.09 · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1113.46 recorded 2026-10-22.
AA-LCR v1.1: 81.666667% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 62.027493% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 23.6% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1395.78 · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1391.9 recorded 2026-10-22.
HLE text-only ; AA harness: 47.868397% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 13.636364% · Contributes. Source · Evaluator-reported · Gemini 3.7 Flash (high); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

Claude Haiku 5.5 · 25 used / 26 collected

23 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±7.4 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1575.9 · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1575.37 recorded 2026-10-27.
GDPval-AA v2.1 (Elo): 1628.16 · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1617.13 recorded 2026-10-21.
AutomationBench-AA: 35.9538897734% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (xhigh); Artificial Analysis independent evaluation, 2026-10-07; fallback not reported
Exact model and effort in the evaluator chart payload.
Terminal-Bench 4.0 ; AA harness: 32.8282828283% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Artificial Analysis independent evaluation, 2026-10-07; fallback not reported
Exact model and effort in the evaluator chart payload.
SciCode: 54.9768518519% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Artificial Analysis independent evaluation, 2026-10-07; fallback not reported
Exact model and effort in the evaluator chart payload.
HLE text-only ; AA harness: 44.3929564411% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Artificial Analysis independent evaluation, 2026-10-07; fallback not reported
Exact model and effort in the evaluator chart payload.
GDP.pdf ; AA harness: 20.8% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Artificial Analysis independent evaluation, 2026-10-07; fallback not reported
Exact model and effort in the evaluator chart payload.
CritPt: 22.5714285714% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (xhigh); Artificial Analysis independent evaluation, 2026-10-07; fallback not reported
Exact model and effort in the evaluator chart payload.
AA-Omniscience: 10.6666666667 · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Artificial Analysis independent evaluation, 2026-10-07; fallback not reported
Restored on 2026-10-28: 12.3 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. Exact model and effort in the evaluator chart payload.
AA-LCR v1.1: 82.6666666667% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Artificial Analysis independent evaluation, 2026-10-07; fallback not reported
Exact model and effort in the evaluator chart payload.
AA Intelligence Index v4.3.2 (score): 43.3950199671 · Not used. Source · Evaluator-reported · Claude Haiku 5.5 (max); Artificial Analysis independent evaluation, 2026-10-07; fallback not reported
Restored on 2026-10-28: 36.234544 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Reference composite only, never scored alongside its component evaluations.
Humanity's Last Exam (no tools): 45.9% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (effort not stated); Anthropic Claude Haiku 5.5 launch post; see Haiku 5.5 system card for evaluation details; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (sonnet-5-5 56.9). Effort not stated in the table.
Humanity's Last Exam w/ tools: 57.4% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (effort not stated); Anthropic Claude Haiku 5.5 launch post; see Haiku 5.5 system card for evaluation details; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (sonnet-5-5 64.5). Effort not stated in the table.
Terminal-Bench 4.0: 39.2% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (effort not stated); Anthropic Claude Haiku 5.5 launch post; see Haiku 5.5 system card for evaluation details; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (sonnet-5-5 70.6). Effort not stated in the table.
BioMysteryBench v1 ; Vals: 70.37% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Vals AI BioMysteryBench harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 70.37 ± 1.33, rank 12/27. Vals notes some benchmarks may use a different provider or parameters.
Code Migration †: 44.57% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Vals AI Code Migration harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 44.57 ± 4.23, rank 21/77. Vals notes some benchmarks may use a different provider or parameters.
Excel Modeling Benchmark ; Vals: 59.02% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Vals AI EMB harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 59.02 ± 4.17, rank 36/74. Vals notes some benchmarks may use a different provider or parameters.
Finance Agent v2 ; Vals: 54.14% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Vals AI Finance Agent (v2) harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 54.14 ± 2.13, rank 26/78. Vals notes some benchmarks may use a different provider or parameters.
Harvey LAB (Vals): 1.25% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Vals AI Harvey's Legal Agent Benchmark harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 1.25 ± 0.72, rank 57/79. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 43.75% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Vals AI Legal Research Bench harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 43.75 ± 3.45, rank 24/86. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 51.54% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Vals AI MedCode harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 51.54 ± 2.17, rank 15/107. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 89.54% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Vals AI MedScribe harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 89.54 ± 1.92, rank 6/109. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 89% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Vals AI ProofBench v1.1 harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 89.0 ± 3.15, rank 10/50. Vals notes some benchmarks may use a different provider or parameters.
Public Benefits Bench ; Vals: 63.19% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Vals AI Public Benefits Bench v1.1 harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 63.19 ± 1.25, rank 25/50. Vals notes some benchmarks may use a different provider or parameters.
SAGE ; Vals: 45.78% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Vals AI SAGE harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 45.78 ± 3.42, rank 38/91. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 90.44% · Contributes. Source · Evaluator-reported · Claude Haiku 5.5 (max); Vals AI Vibe Code Bench v1.1 harness; provider Anthropic; fallback not reported
Vals AI model page, headline accuracy 90.44 ± 1.73, rank 3/112. Vals notes some benchmarks may use a different provider or parameters.

Qwen3.8-Max · 50 used / 86 collected

47 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.2 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1633.53 · Not used. Source · Evaluator-reported · Qwen3.8 Max; Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 81.273408% · Contributes. Source · Evaluator-reported · Qwen3.8 Max; Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 57.461024% · Contributes. Source · Evaluator-reported · qwen3-8-max; xhigh; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1479.556758 · Contributes. Source · Evaluator-reported · qwen3.8-max; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 58 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 92.727273% · Contributes. Source · Evaluator-reported · Qwen3.8 Max; Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 74.3% · Not used. Source · Evaluator-reported · Qwen3.8 Max
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 50.59% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 60.07% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 53.240741% · Contributes. Source · Evaluator-reported · Qwen3.8 Max; Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 43.04912% · Contributes. Source · Evaluator-reported · Qwen3.8 Max; Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 47.6% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 10.42% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1392.33 · Not used. Source · Evaluator-reported · Qwen3.8 Max; Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 84.95% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 85.6% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 83.61% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 75.55% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
KingBench 3: 81.25% · Not used.
Not admitted after the September 6 model review: the collected 81.25 for Qwen3.8 Max on KingBench 3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vals Index: 66.12% · Not used. Source · Evaluator-reported · qwen3.8-max; max effort; Vals AI index methodology
Held-out composite, never a scoring input
Public Benefits Bench ; Vals: 67.12% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 40.67% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AutomationBench v1.0.6: 39.8% · Not used.
Not admitted after the September 6 model review: the collected 39.8 for Qwen3.8 Max on AutomationBench v1.0.6 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Omniscience: 3.4 · Contributes. Source · Evaluator-reported · Qwen3.8 Max; Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 88.6% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 78.5% · Not used.
Not admitted after the September 6 model review: the collected 78.5 for Qwen3.8 Max on LiveBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CorpFin v2 ; Vals: 65.85% · Contributes. Source · Archived source review · qwen3.8-max; max effort; Vals AI CorpFin v2 evaluation with Sonnet 4.5 judge
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
MortgageTax ; Vals: 63.99% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SAGE ; Vals: 51.25% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Code Migration †: 23.96% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Toolathlon-Verified: 72.5% · Not used.
Not admitted after the September 6 model review: the collected 72.5 for Qwen3.8 Max on Toolathlon-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Agents' Last Exam: 27% · Contributes. Source · Evaluator-reported · Qwen3.8 Max; XHigh; Claude Code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveCodeBench: 87.85% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CyberGym: 78.5% · Contributes. Source · Evaluator-reported · Qwen3.8 Max; Z.ai GLM-5.3 model-card comparison table; GLM max, other comparator efforts not fully disclosed
CyberGym release-table percentage. Z.ai documents single-run pass@1 on 1,507 tasks, unlimited task timeout, Claude Code 2.1.207 and restricted network for GLM-5.3. Do not transfer those GLM-specific settings to every comparator. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
Humanity's Last Exam w/ tools: 56.2% · Contributes. Source · Evaluator-reported · Qwen3.8 Max; Z.ai GLM-5.3 model-card comparison table; GLM max, other comparator efforts not fully disclosed
HLE with tools release-table percentage. Z.ai documents temperature=1, top_p=.95, 163,840 output tokens, 300K context and GPT-5.6 Luna medium judge for its evaluation. Comparator-specific tools and judging settings are not fully disclosed. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
Design Arena (Elo): 1388 · Contributes. Source · Archived source review · qwen3.8-max; max effort; Design Arena 2026-08-11 rating snapshot
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Vals Multimodal Index: 65.39% · Not used. Source · Evaluator-reported · qwen3.8-max; max effort; Vals AI multimodal index methodology
Held-out composite, never a scoring input
ProgramBench v1 (Raw Pass Rate) ; Vals: 39.971% · Contributes. Source · Archived source review · alibaba/qwen3.8-max · Vals published row (effort not stated)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
SkillsBench †: 42.01% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierSWE: 73.5% · Not used.
Not admitted after the September 6 model review: the collected 73.5 for Qwen3.8 Max on FrontierSWE still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierMath v2 Tier 4: 46.3% · Not used.
Not admitted after the September 6 model review: the collected 46.3 for Qwen3.8 Max on FrontierMath v2 Tier 4 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
SWE-bench Pro: 67.7% · Not used.
Not admitted after the September 6 model review: the collected 67.7 for Qwen3.8 Max on SWE-bench Pro still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CharXiv (with CI / RQ): 93.5% · Not used.
Not admitted after the September 6 model review: the collected 93.5 for Qwen3.8 Max on CharXiv (with CI / RQ) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MMMU-Pro: 82.312139% · Contributes. Source · Evaluator-reported · Qwen3.8 Max; Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
OSWorld 2.0 ; Anthropic H2H: 46.7% · Contributes. Source · Archived source review · qwen3.8-max; max effort; Anthropic Opus 5 published head-to-head setup
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
ExploitBench: 28.8% · Not used.
Inherited comparator metric/scaffold equivalence to Astra Cap Percent is not established; the source flags possible contamination. Excluded pending protocol audit.
JobBench: 53.4% · Not used.
Not admitted after the September 6 model review: the collected 53.4 for Qwen3.8 Max on JobBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
BabyVision (with CI): 91.3% · Not used.
Not admitted after the September 6 model review: the collected 91.3 for Qwen3.8 Max on BabyVision (with CI) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
PerceptionBench: 63.5% · Not used.
Not admitted after the September 6 model review: the collected 63.5 for Qwen3.8 Max on PerceptionBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
LVBench (with Memory): 85.6% · Not used.
Not admitted after the September 6 model review: the collected 85.6 for Qwen3.8 Max on LVBench (with Memory) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
OSWorld-Verified: 86.1% · Not used.
Not admitted after the September 6 model review: the collected 86.1 for Qwen3.8 Max on OSWorld-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Creative Writing v3: 1842 · Contributes. Source · Evaluator-reported · *Qwen/Qwen3.8-2.4T-A95B; published evaluation, effort unspecified
Exact checkpoint on the evaluator-hosted board. A starred model is externally submitted; it is not claimed as independently rerun by the benchmark owner.
NL2Repo: 55.9% · Contributes. Source · Evaluator-reported · Qwen3.8 Max; Z.ai GLM-5.3 model-card comparison table; GLM max, other comparator efforts not fully disclosed
NL2Repo release-table percentage. Z.ai documents temperature=1, top_p=1, 64K output, 1M context and anti-cheating judges for its evaluation. Comparator-specific harness settings are not fully disclosed. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
CoWorkBench: 74.8% · Not used.
Not admitted after the September 6 model review: the collected 74.8 for Qwen3.8 Max on CoWorkBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
PaperBench (Replication Score): 93% · Not used.
Not admitted after the September 6 model review: the collected 93 for Qwen3.8 Max on PaperBench (Replication Score) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
QwenReactBench (Elo): 1724 · Not used.
Not admitted after the September 6 model review: the collected 1724 for Qwen3.8 Max on QwenReactBench (Elo) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ERQA: 77.8% · Not used.
Not admitted after the September 6 model review: the collected 77.8 for Qwen3.8 Max on ERQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vision2Web (Avg. Frontend/Webpage/etc.): 69% · Not used.
Not admitted after the September 6 model review: the collected 69 for Qwen3.8 Max on Vision2Web (Avg. Frontend/Webpage/etc.) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MobileWorld: 77.8% · Not used.
Not admitted after the September 6 model review: the collected 77.8 for Qwen3.8 Max on MobileWorld still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
WebArena-Verified: 66.8% · Not used.
Not admitted after the September 6 model review: the collected 66.8 for Qwen3.8 Max on WebArena-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 51.340206% · Contributes. Source · Evaluator-reported · Qwen3.8 Max; Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 20% · Contributes. Source · Evaluator-reported · Qwen3.8 Max; Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
GDP.pdf: 23.2% · Contributes. Source · Evaluator-reported · Qwen 3.8 Max (xHigh reasoning); Surge original evaluation
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 58% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 64.7% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-max; Vals sole published default; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MineBench: 1333 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
MLCR-AA (Medical Long Context Reasoning): 19.444444% · Not used. Source · Evaluator-reported · Qwen3.8 Max; Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
IFBench: 82.8% · Not used.
Not admitted after the September 6 model review: the collected 82.8 for Qwen3.8 Max on IFBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AndroidWorld: 85.3% · Not used.
Not admitted after the September 6 model review: the collected 85.3 for Qwen3.8 Max on AndroidWorld still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CWE-bench: 37.5% · Contributes. Source · Evaluator-reported · Qwen3.8-Max; high; opencode
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ZeroBench: 24% · Not used.
Not admitted after the September 6 model review: the collected 24 for Qwen3.8 Max on ZeroBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
SimpleVQA: 75% · Not used.
Not admitted after the September 6 model review: the collected 75 for Qwen3.8 Max on SimpleVQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathVision (with CI): 97.7% · Not used.
Not admitted after the September 6 model review: the collected 97.7 for Qwen3.8 Max on MathVision (with CI) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
OmniDocBench 1.5: 92.1% · Not used.
Not admitted after the September 6 model review: the collected 92.1 for Qwen3.8 Max on OmniDocBench 1.5 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
RuneBench (30m, mean ln(1 + XP/min)): 5.347003 · Contributes. Source · Archived source review · qwen38max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 1847 · Contributes. Source · Archived source review · Qwen3.8-Max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Roboflow Vision Evals (six-task mean, high tier): 85% · Contributes. Source · Archived source review · Require a complete six-task high-tier panel, the highest published benchmark tier. Retain high-tier scores even when lower than low-tier. Do not substitute the best of three repetitions: use the published mean. Each panel enters the fit once; do not also fit its components.
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1670 · Contributes. Source · Archived source review · qwen3.8-max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 45.415208 · Not used. Source · Evaluator-reported · Qwen3.8 Max (none); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Restored on 2026-10-28: 33.554563 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1607.7 · Contributes. Source · Evaluator-reported · Qwen3.8 Max (none); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1612.62 recorded 2026-10-21.
AA-LCR v1.1: 80.333333% · Contributes. Source · Evaluator-reported · Qwen3.8 Max (none); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 56.190538% · Contributes. Source · Evaluator-reported · Qwen3.8 Max (none); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 22.8% · Contributes. Source · Evaluator-reported · Qwen3.8 Max (none); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1673.46 · Contributes. Source · Evaluator-reported · Qwen3.8 Max (none); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1664.63 recorded 2026-10-21.
HLE text-only ; AA harness: 43.095459% · Contributes. Source · Evaluator-reported · Qwen3.8 Max (none); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 38.888889% · Contributes. Source · Evaluator-reported · Qwen3.8 Max (none); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

GPT-5.6 Terra · 57 used / 88 collected

51 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.1 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1482.49 · Not used. Source · Evaluator-reported · GPT-5.6 Terra (xhigh); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 88.014981% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 69.62306% · Contributes. Source · Evaluator-reported · gpt-5-6-terra; max; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1466.420286 · Contributes. Source · Evaluator-reported · gpt-5.6-terra-xhigh; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 57 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 92.525253% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 79.7% · Not used. Source · Evaluator-reported · GPT-5.6 Terra (max)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 54.44% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 66.2% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 54.976852% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis published evaluation harness
Best published result among 3 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 42.910102% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 41.35% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 0.83% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1341.29 · Not used. Source · Evaluator-reported · GPT-5.6 Terra (xhigh); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 82.87% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 95.4% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 85.11% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 76.17% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
KingBench 3: 62.9% · Not used.
Not admitted after the September 6 model review: the collected 62.9 for GPT-5.6 Terra on KingBench 3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vals Index: 56.53% · Not used.
Held-out composite, never a scoring input
Public Benefits Bench ; Vals: 62.38% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 43.41% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AutomationBench v1.0.6: 37.17% · Not used.
Not admitted after the September 6 model review: the collected 37.17 for GPT-5.6 Terra on AutomationBench v1.0.6 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Omniscience: 0.05 · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 86.66% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 77.93625% · Contributes. Source · Evaluator-reported · gpt-5.6-terra-max; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
CorpFin v2 ; Vals: 65.31% · Not used.
Not admitted after the September 6 model review: the collected 65.31 for GPT-5.6 Terra on CorpFin v2 ; Vals still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MortgageTax ; Vals: 67.33% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SAGE ; Vals: 47% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Code Migration †: 47.8% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Toolathlon-Verified: 74.9% · Not used.
Not admitted after the September 6 model review: the collected 74.9 for GPT-5.6 Terra on Toolathlon-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Agents' Last Exam: 28% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra; Max; Codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Terminal-Bench v3.0: 20.8% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveCodeBench: 85.93% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CyberGym: 81.8% · Not used.
Not admitted after the September 6 model review: the collected 81.8 for GPT-5.6 Terra on CyberGym still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Design Arena (Elo): 1280 · Not used.
Not admitted after the September 6 model review: the collected 1280 for GPT-5.6 Terra on Design Arena (Elo) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vals Multimodal Index: 65.07% · Not used.
Held-out composite, never a scoring input
SWE-Marathon v1.1: 32.5% · Not used.
Not admitted after the September 6 model review: the collected 32.5 for GPT-5.6 Terra on SWE-Marathon v1.1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ProgramBench v1 (Raw Pass Rate) ; Vals: 72.345% · Contributes. Source · Archived source review · openai/gpt-5.6-terra · reasoning_effort=max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
SkillsBench †: 58.9% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierMath v2 Tier 4: 70.731707% · Contributes. Source · Evaluator-reported · gpt-5.6-terra_max; task version 2.0.0
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
CursorBench v3.2: 64.9% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra Max; Cursor harness
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Pro: 63.4% · Not used.
Not admitted after the September 6 model review: the collected 63.4 for GPT-5.6 Terra on SWE-bench Pro still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CharXiv (with CI / RQ): 88% · Not used.
Not admitted after the September 6 model review: the collected 88 for GPT-5.6 Terra on CharXiv (with CI / RQ) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
BrowseComp: 87.5% · Not used.
Not admitted after the September 6 model review: the collected 87.5 for GPT-5.6 Terra on BrowseComp still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MMMU-Pro: 80.693642% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
ARC-AGI-2: 83.9% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
OSWorld 2.0 ; Anthropic H2H: 50.2% · Not used.
Not admitted after the September 6 model review: the collected 50.2 for GPT-5.6 Terra on OSWorld 2.0 ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ExploitBench: 52.9% · Not used.
Inherited comparator metric/scaffold equivalence to Astra Cap Percent is not established; the source flags possible contamination. Excluded pending protocol audit.
SimpleQA: 43.1% · Not used.
Not admitted after the September 6 model review: the collected 43.1 for GPT-5.6 Terra on SimpleQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Extended: 55.84% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra; max; codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Creative Writing v3: 1850 · Contributes. Source · Evaluator-reported · gpt-5.6-terra; published evaluation, effort unspecified
Exact checkpoint on the evaluator-hosted board. A starred model is externally submitted; it is not claimed as independently rerun by the benchmark owner.
AIIQ Composite IQ: 132 · Not used.
Held-out composite, never a scoring input
EQ-Bench 4: 1234 · Contributes. Source · Evaluator-reported · openai/gpt-5.6-terra; evaluator published configuration, effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
τ²-Bench Telecom: 86.3% · Not used.
Not admitted after the September 6 model review: the collected 86.3 for GPT-5.6 Terra on τ²-Bench Telecom still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Main ; Anthropic H2H: 41.3% · Not used.
Not admitted after the September 6 model review: the collected 41.3 for GPT-5.6 Terra on FrontierCode v1.1 Main ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 40.206186% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 30% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
Terminal-Bench 4.0: 21.5% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra; max; Codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
GDP.pdf: 24.7% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (Medium reasoning); Surge original evaluation
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 74% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 74.59% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ARC-AGI-1: 96.5% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
ARC-AGI-3 (standard harness): 0.8% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
HealthBench: 58.7% · Not used. Source · Developer-reported · GPT-5.6 Terra; OpenAI published HealthBench research evaluation; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
HealthBench Hard: 34.3% · Not used. Source · Developer-reported · GPT-5.6 Terra; OpenAI published HealthBench research evaluation; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
HealthBench Professional (length-adjusted): 57.7% · Not used.
Source selects the maximum score across efforts; exact highest-effort result is not established.
Harvey LAB-AA: 85.18% · Not used.
Not admitted after the September 6 model review: the collected 85.18 for GPT-5.6 Terra on Harvey LAB-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EnterpriseOps-Gym-AA: 38.5% · Not used.
Not admitted after the September 6 model review: the collected 38.5 for GPT-5.6 Terra on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 31.666667% · Not used. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
ITBench-AA: 51.035782% · Not used. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes ITBench-AA, so every model is scored on its current set.
Vending-Bench 2: 7343.206 · Contributes. Source · Evaluator-reported · GPT-5.6 Terra; Andon Labs Vending-Bench 2
Compare reasoning configurations using their mean over runs. Geometric mean, individual best runs, Fireworks Kimi endpoint, Vending Arena, and April DeepSeek Pro are not substituted.
IFBench: 71.2% · Not used.
Not admitted after the September 6 model review: the collected 71.2 for GPT-5.6 Terra on IFBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ZeroBench: 19% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); official main questions
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Mystery Game Puzzles v1.0.4 ; Epoch: 35% · Contributes. Source · Evaluator-reported · gpt-5.6-terra_max; task version 1.0.4
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
Tax Agent Bench v1 ; Vals: 65.2% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-terra; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
RuneBench (30m, mean ln(1 + XP/min)): 5.881388 · Contributes. Source · Archived source review · gpt56terra-xhigh
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 1963 · Contributes. Source · Archived source review · GPT-5.6-Terra
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Roboflow Vision Evals (six-task mean, high tier): 74.25% · Contributes. Source · Archived source review · Require a complete six-task high-tier panel, the highest published benchmark tier. Retain high-tier scores even when lower than low-tier. Do not substitute the best of three repetitions: use the published mean. Each panel enters the fit once; do not also fit its components.
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1520 · Contributes. Source · Archived source review · gpt-5.6-terra-xhigh (codex-harness)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 42.082925 · Not used. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1332.55 · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1332.53 recorded 2026-10-22.
AA-LCR v1.1: 83% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 59.649958% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 24% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1461.97 · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1450.5 recorded 2026-10-22.
HLE text-only ; AA harness: 42.910102% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 35.353535% · Contributes. Source · Evaluator-reported · GPT-5.6 Terra (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

GLM 5.3 Flash · 22 used / 23 collected

22 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±8.4 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1451.01 · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1450.28 recorded 2026-10-27.
GDPval-AA v2.1 (Elo): 1650.37 · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1642.18 recorded 2026-10-17.
AutomationBench-AA: 60.3686278217% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
Terminal-Bench 4.0 ; AA harness: 32.8282828283% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
SciCode: 51.6203703704% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
HLE text-only ; AA harness: 39.8517145505% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
GDP.pdf ; AA harness: 15.4% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
CritPt: 15.4285714286% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
AA-Omniscience: 7.4666666667 · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
AA-LCR v1.1: 80% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Exact model and effort in the evaluator chart payload.
AA Intelligence Index v4.3.2 (score): 41.8074661135 · Not used. Source · Evaluator-reported · GLM 5.3 Flash (max); Artificial Analysis independent evaluation, 2026-09-30; fallback not reported
Reference composite only, never scored alongside its component evaluations.
Toolathlon-Verified: 78.4% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (effort not stated); Z.ai GLM-5.3-Flash launch post benchmark table; per-benchmark settings in its footnotes; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (gpt-5-6-terra 74.9). Official evaluation service, pass@1 averaged over 3 runs (footnote). Effort not stated.
Agents' Last Exam: 26.3% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Z.ai GLM-5.3-Flash launch post benchmark table; per-benchmark settings in its footnotes; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (gpt-5-6-terra 28.0). Official ALE protocol, Claude Code harness, reasoning effort max, 1M context (footnote).
CharXiv (with CI / RQ): 89.4% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (effort not stated); Z.ai GLM-5.3-Flash launch post benchmark table; per-benchmark settings in its footnotes; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (gpt-5-6-terra 88.0, gemini-3-7-flash 88.7). Effort not stated.
OSWorld 2.0 ; Anthropic H2H: 59.1% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (effort not stated); Z.ai GLM-5.3-Flash launch post benchmark table; per-benchmark settings in its footnotes; fallback not reported
Developer-reported aggregate. Same protocol as the catalog row: the lab table reproduces existing cells (gpt-5-6-terra 50.2). Effort not stated.
LegalBench ; Vals: 83.93% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Vals AI LegalBench harness; provider Fireworks AI; fallback not reported
Vals AI model page, headline accuracy 83.93 ± 0.43, rank 45/151. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 45.19% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Vals AI Legal Research Bench harness; provider Fireworks AI; fallback not reported
Vals AI model page, headline accuracy 45.19 ± 3.46, rank 18/82. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 88.94% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Vals AI MedScribe harness; provider Fireworks AI; fallback not reported
Vals AI model page, headline accuracy 88.94 ± 1.91, rank 9/109. Vals notes some benchmarks may use a different provider or parameters.
MMLU Pro ; Vals: 86.06% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Vals AI MMLU Pro harness; provider Fireworks AI; fallback not reported
Vals AI model page, headline accuracy 86.06 ± 0.34, rank 49/138. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 21% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Vals AI ProofBench v1.1 harness; provider Fireworks AI; fallback not reported
Vals AI model page, headline accuracy 21.0 ± 4.09, rank 40/50. Vals notes some benchmarks may use a different provider or parameters.
SWE-bench Verified ; Vals: 92% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Vals AI SWE-bench harness; provider Fireworks AI; fallback not reported
Vals AI model page, headline accuracy 92.0 ± 1.21, rank 10/88. Vals notes some benchmarks may use a different provider or parameters.
TaxEval v2 ; Vals: 75.59% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Vals AI TaxEval v2 harness; provider Fireworks AI; fallback not reported
Vals AI model page, headline accuracy 75.59 ± 0.84, rank 16/145. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 30.76% · Contributes. Source · Evaluator-reported · GLM 5.3 Flash (max); Vals AI Vibe Code Bench v1.1 harness; provider Fireworks AI; fallback not reported
Vals AI model page, headline accuracy 30.76 ± 5.28, rank 75/116. Vals notes some benchmarks may use a different provider or parameters.

GPT-6 Luna · 22 used / 35 collected

22 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±8.4 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1336.99 · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1336.19 recorded 2026-10-27.
GDPval-AA v2.1 (Elo): 1444.72 · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1431.02 recorded 2026-10-21.
AutomationBench-AA: 53.1681403448% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-luna / automationbench-aa. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
Terminal-Bench 4.0 ; AA harness: 12.6262626263% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-luna / terminalbench-4-0. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
SciCode: 54.6296296296% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-luna / scicode. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
HLE text-only ; AA harness: 38.5078776645% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-luna / humanitys-last-exam. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
GDP.pdf ; AA harness: 22.8% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Daily evaluator refresh; replaces 20.4 recorded 2026-09-22.
CritPt: 19.4285714286% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-luna / critpt. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
AA-Omniscience: 0.65 · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Artificial Analysis independent evaluation; fallback not reported
Restored on 2026-10-28: -8.266667 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. AA payload gpt-6-luna / omniscience. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
AA-LCR v1.1: 83.3333333333% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Artificial Analysis independent evaluation; fallback not reported
AA payload gpt-6-luna / artificial-analysis-long-context-reasoning. HLE is text-only; Omniscience is the native index, not accuracy; AutomationBench-AA is guardrail-adjusted objective credit, not workflow pass rate.
MMMU-Pro: 79.710983% · Contributes. Source · Evaluator-reported · GPT-6 Luna (xhigh); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Daily evaluator refresh; replaces 75.2601156069 recorded 2026-09-22.
AA Intelligence Index v4.3.2 (score): 38.124519 · Not used. Source · Evaluator-reported · GPT-6 Luna (max); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Restored on 2026-10-28: 27.7552 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Daily evaluator refresh; replaces 37.255968687 recorded 2026-09-22.
FrontierCode v1.1 Main ; Anthropic H2H: 42.42% · Not used. Source · Evaluator-reported · GPT-6 Luna (max); Cognition FrontierCode 1.1; Codex harness; fallback not reported
Cognition v1_1 data, main. new_score is mergeability; correct is a different metric. Unfair internet use is zeroed. Existing catalog name Main ; Anthropic H2H tracks this same public Cognition Main protocol.
FrontierCode v1.1 Extended: 56.1% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Cognition FrontierCode 1.1; Codex harness; fallback not reported
Cognition v1_1 data, extended. new_score is mergeability; correct is a different metric. Unfair internet use is zeroed. Existing catalog name Main ; Anthropic H2H tracks this same public Cognition Main protocol.
DeepSWE v1.1: 66.6% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); OpenAI launch research/API evaluation; fallback not reported
Explicit OpenAI launch paragraph. The archived operator leaderboard predates these additions; not inferred from GPT-5.6 results.
HealthBench Professional ; Sep 22 length-adjusted: 60.8% · Not used. Source · Evaluator-reported · GPT-6 Luna (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
Table 29 length-adjusted column, not parenthesized raw score. Revised comparison baselines differ from September 3; kept separate and uncalibrated.
HealthBench Overall ; Sep 22 length-adjusted: 54.5% · Not used. Source · Evaluator-reported · GPT-6 Luna (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
Table 29 length-adjusted column, not parenthesized raw score. Revised comparison baselines differ from September 3; kept separate and uncalibrated.
HealthBench Hard ; Sep 22 length-adjusted: 31.4% · Not used. Source · Evaluator-reported · GPT-6 Luna (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
Table 29 length-adjusted column, not parenthesized raw score. Revised comparison baselines differ from September 3; kept separate and uncalibrated.
HealthBench Consensus ; Sep 22 length-adjusted: 95.9% · Not used. Source · Evaluator-reported · GPT-6 Luna (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
Table 29 length-adjusted column, not parenthesized raw score. Revised comparison baselines differ from September 3; kept separate and uncalibrated.
ExploitBench: 43.4% · Not used. Source · Evaluator-reported · GPT-6 Luna (max); OpenAI September 22 system-card appendix; fallback not reported
§11.8.1.2.1. Publisher warns of possible historical vulnerability contamination; not the recent internal port.
SEC-Bench Pro: 34.2% · Not used. Source · Evaluator-reported · GPT-6 Luna (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
§11.8.1.2, same OpenAI V8/SpiderMonkey vulnerability-discovery evaluation; named effort not specified in prose.
ExploitBench Internal Port ; Sep 22 release: 0% · Not used. Source · Evaluator-reported · GPT-6 Luna (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
§11.8.1.2.2 June–August vulnerabilities. Astra comparison is 31.5 versus 39 in earlier release; no substitution into the old calibrated row.
ExploitGym ; Sep 22 release: 11.6% · Not used. Source · Evaluator-reported · GPT-6 Luna (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
Working-exploit success, §11.8.1.2. Distinct from honeypot compliance and safety behavior.
Sandbox Bench ; 22 targets: 4.5454545455% · Not used. Source · Evaluator-reported · GPT-6 Luna (undisclosed); OpenAI September 22 system-card appendix; fallback not reported
§11.8.1.2.3 one of 22 targets; converted from count to percent. Not a best-trial or per-category observation.
Internal Research Debugging ; Sep 22: 46.62% · Not used. Source · Evaluator-reported · GPT-6 Luna (undisclosed); OpenAI system-card figure 84; fallback not reported
Figure 84: exact printed mean rubric reward (%), not task success.
KernelGen 1P ; Sep 22: 21.83% · Not used. Source · Evaluator-reported · GPT-6 Luna (undisclosed); OpenAI system-card figure 85; fallback not reported
Figure 85: exact printed rubric reward (%), not speedup ratio.
AutomationBench ; Anthropic H2H: 20.7% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Zapier operator leaderboard; native full-workflow success; fallback not reported
Zapier native operator leaderboard, September 22. Same held-out evaluation as H2H row (Opus5 max 26.94), not public 600-task v1.0.6 or AA partial credit.
BioMysteryBench v1 ; Vals: 61.48% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Vals AI BioMysteryBench harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 61.48 ± 2.59, rank 24/27. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 30.29% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Vals AI Legal Research Bench harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 30.29 ± 3.19, rank 49/82. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 44.69% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Vals AI MedCode harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 44.69 ± 2.3, rank 41/107. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 83.71% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Vals AI MedScribe harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 83.71 ± 1.95, rank 43/109. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 64% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Vals AI ProofBench v1.1 harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 64.0 ± 4.82, rank 19/50. Vals notes some benchmarks may use a different provider or parameters.
Public Benefits Bench ; Vals: 57.65% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Vals AI Public Benefits Bench v1.1 harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 57.65 ± 1.28, rank 38/50. Vals notes some benchmarks may use a different provider or parameters.
SAGE ; Vals: 48.09% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Vals AI SAGE harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 48.09 ± 3.38, rank 28/91. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 81.65% · Contributes. Source · Evaluator-reported · GPT-6 Luna (max); Vals AI Vibe Code Bench v1.1 harness; provider OpenAI; fallback not reported
Vals AI model page, headline accuracy 81.65 ± 3.38, rank 23/116. Vals notes some benchmarks may use a different provider or parameters.

Muse Spark 1.2 · 40 used / 60 collected

38 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.3 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1527.08 · Not used. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 80.149813% · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 54.867257% · Contributes. Source · Evaluator-reported · muse-spark-1-2; xhigh; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1499 · Not used.
Not admitted after the September 6 model review: the collected 1499 for Muse Spark 1.2 on LMArena (Chatbot Arena) ; Text still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA Intelligence Index v4.1.1 (score): 57 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 90.40404% · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 83.3% · Not used. Source · Evaluator-reported · Muse Spark 1.2 (xhigh)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 60.6% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 56.98% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 57.407407% · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 45.458758% · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 43.75% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 25.42% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1346.23 · Not used. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 90.06% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 86.6% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 85.26% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 80.38% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
KingBench 3: 76.25% · Not used.
Not admitted after the September 6 model review: the collected 76.25 for Muse Spark 1.2 on KingBench 3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vals Index: 71.88% · Not used. Source · Evaluator-reported · muse-spark-1.2; xhigh effort; Vals AI index methodology
Held-out composite, never a scoring input
Public Benefits Bench ; Vals: 68.47% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 49.35% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AutomationBench v1.0.6: 38.2% · Not used.
Not admitted after the September 6 model review: the collected 38.2 for Muse Spark 1.2 on AutomationBench v1.0.6 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Omniscience: 27.2 · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 88.28% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 78% · Not used.
Not admitted after the September 6 model review: the collected 78 for Muse Spark 1.2 on LiveBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CorpFin v2 ; Vals: 70.94% · Contributes. Source · Archived source review · muse-spark-1.2; xhigh effort; Vals AI CorpFin v2 evaluation with Sonnet 4.5 judge
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
MortgageTax ; Vals: 65.42% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SAGE ; Vals: 47.66% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Code Migration †: 29.95% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Toolathlon-Verified: 75.9% · Contributes. Source · Evaluator-reported · Meta Muse Spark 1.2 (xhigh) ✓; Default agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Design Arena (Elo): 1373 · Contributes. Source · Archived source review · muse-spark-1.2; xhigh effort; Design Arena 2026-08-11 rating snapshot
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Vals Multimodal Index: 69.8% · Not used. Source · Evaluator-reported · muse-spark-1.2; xhigh effort; Vals AI multimodal index methodology
Held-out composite, never a scoring input
SkillsBench †: 53.04% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
OSWorld 2.0 ; Anthropic H2H: 47.6% · Not used.
Not admitted after the September 6 model review: the collected 47.6 for Muse Spark 1.2 on OSWorld 2.0 ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
JobBench: 61.6% · Not used.
Not admitted after the September 6 model review: the collected 61.6 for Muse Spark 1.2 on JobBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Creative Writing v3: 1835.4 · Not used.
Not admitted after the September 6 model review: the collected 1835.4 for Muse Spark 1.2 on Creative Writing v3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MCP Atlas †: 90.3% · Not used.
Not admitted after the September 6 model review: the collected 90.3 for Muse Spark 1.2 on MCP Atlas † still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 34.845361% · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 17.714286% · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
GDP.pdf: 16% · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xHigh reasoning); Surge original evaluation
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 43% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 79.1% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MineBench: 1653 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
OSWorld 2.0: 17.9% · Not used.
Not admitted after the September 6 model review: the collected 17.9 for Muse Spark 1.2 on OSWorld 2.0 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EnterpriseOps-Gym-AA: 47.27% · Not used.
Not admitted after the September 6 model review: the collected 47.27 for Muse Spark 1.2 on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 31.111111% · Not used. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
CWE-bench: 30.3% · Not used.
Not admitted after the September 6 model review: the collected 30.3 for Muse Spark 1.2 on CWE-bench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
BioMysteryBench v1 ; Vals: 64.81% · Contributes. Source · Evaluator-reported · meta_muse_spark_1_2; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
RuneBench (30m, mean ln(1 + XP/min)): 5.238233 · Contributes. Source · Archived source review · muse12
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Roboflow Vision Evals (six-task mean, high tier): 80.016667% · Contributes. Source · Archived source review · Require a complete six-task high-tier panel, the highest published benchmark tier. Retain high-tier scores even when lower than low-tier. Do not substitute the best of three repetitions: use the published mean. Each panel enters the fit once; do not also fit its components.
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1534 · Contributes. Source · Archived source review · muse-spark-1.2 (xHigh)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 39.575861 · Not used. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1326.93 · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1327.05 recorded 2026-10-22.
AA-LCR v1.1: 79% · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 40.613378% · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 17.4% · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1474.64 · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1472.52 recorded 2026-10-22.
HLE text-only ; AA harness: 45.458758% · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 7.070707% · Contributes. Source · Evaluator-reported · Muse Spark 1.2 (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

Muse Spark Open · 19 used / 20 collected

19 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±11.7 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1279.13 · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1278.38 recorded 2026-10-27.
GDPval-AA v2.1 (Elo): 1401.12 · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
AutomationBench-AA: 49.7081182731% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
Terminal-Bench 4.0 ; AA harness: 8.0808080808% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
SciCode: 58.4490740741% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
HLE text-only ; AA harness: 43.8832252085% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
GDP.pdf ; AA harness: 18.6% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
CritPt: 14.8571428571% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
AA-Omniscience: 24.65 · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Restored on 2026-10-28: 25.8 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. Exact model and effort in the evaluator chart payload.
AA-LCR v1.1: 78.3333333333% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Exact model and effort in the evaluator chart payload.
MMMU-Pro: 80.9248554913% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Independent MMMU-Pro aggregate for this effort.
AA Intelligence Index v4.3.2 (score): 38.8666662998 · Not used. Source · Evaluator-reported · Muse Spark Open (xhigh); Artificial Analysis independent evaluation, 2026-10-15; fallback not reported
Restored on 2026-10-28: 24.681587 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Reference composite only, never scored alongside its component evaluations.
JobBench: 61.8% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Muse Spark Open developer-reported result (meta); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (kimi-k3 54.3). Value read from an image of the lab table.
BabyVision (with CI): 80.2% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Muse Spark Open developer-reported result (meta); fallback not reported
Developer-reported result entered in the newsroom. Same protocol as the catalogue row: the lab table reproduces cells we hold (kimi-k3 85.7). Value read from an image of the lab table.
Code Migration †: 21.17% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Vals AI Code Migration harness; provider Together AI; fallback not reported
Vals AI model page, headline accuracy 21.17 ± 3.47, rank 51/78. Vals notes some benchmarks may use a different provider or parameters.
Excel Modeling Benchmark ; Vals: 53.67% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Vals AI EMB harness; provider Together AI; fallback not reported
Vals AI model page, headline accuracy 53.67 ± 3.2, rank 50/75. Vals notes some benchmarks may use a different provider or parameters.
Finance Agent v2 ; Vals: 61.34% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Vals AI Finance Agent (v2) harness; provider Together AI; fallback not reported
Vals AI model page, headline accuracy 61.34 ± 2.09, rank 3/79. Vals notes some benchmarks may use a different provider or parameters.
Harvey LAB (Vals): 16.67% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Vals AI Harvey's Legal Agent Benchmark harness; provider Together AI; fallback not reported
Vals AI model page, headline accuracy 16.67 ± 3.09, rank 9/80. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 38.46% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Vals AI Legal Research Bench harness; provider Together AI; fallback not reported
Vals AI model page, headline accuracy 38.46 ± 3.38, rank 35/79. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 76.19% · Contributes. Source · Evaluator-reported · Muse Spark Open (xhigh); Vals AI Vibe Code Bench v1.1 harness; provider Together AI; fallback not reported
Vals AI model page, headline accuracy 76.19 ± 3.32, rank 34/113. Vals notes some benchmarks may use a different provider or parameters.

DeepSeek V4 Pro · 42 used / 64 collected

39 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.3 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1496.68 · Not used. Source · Evaluator-reported · DeepSeek V4 Pro 0813 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 78.651685% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro 0813 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 62.831858% · Contributes. Source · Evaluator-reported · deepseek-v4-pro; max; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1459.610748 · Contributes. Source · Evaluator-reported · deepseek-v4-pro-high-20260813; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 53 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 92.828283% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro 0813 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 75.3% · Not used. Source · Evaluator-reported · DeepSeek V4 Pro 0813 (Reasoning, Max Effort)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 50.39% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 52.8% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 51.041667% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro 0813 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 41.010195% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro 0813 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 40.87% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 7.5% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1269.34 · Not used. Source · Evaluator-reported · DeepSeek V4 Pro 0813 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 80.17% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 96.4% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 82.36% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 73.06% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
KingBench 3: 76.25% · Not used.
Not admitted after the September 6 model review: the collected 76.25 for DeepSeek V4 Pro Max on KingBench 3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vals Index: 66.25% · Not used. Source · Evaluator-reported · deepseek-v4-pro; max effort; Vals AI index methodology
Held-out composite, never a scoring input
Public Benefits Bench ; Vals: 62.92% · Contributes. Source · Archived source review · deepseek-v4-pro; max effort; Vals AI public-benefits evaluation
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
MedCode ; Vals: 42.47% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AutomationBench v1.0.6: 43.2% · Not used.
Not admitted after the September 6 model review: the collected 43.2 for DeepSeek V4 Pro Max on AutomationBench v1.0.6 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Omniscience: 0.833333 · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro 0813 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 86.97% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 77.436143% · Contributes. Source · Evaluator-reported · deepseek-v4-pro-0813; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
CorpFin v2 ; Vals: 65.42% · Contributes. Source · Archived source review · deepseek-v4-pro; max effort; Vals AI CorpFin v2 evaluation with Sonnet 4.5 judge
Exact-value match to an existing verified, comparable, pinned-configuration observation. Archived review retained; not asserted newly independently replicated.
Code Migration †: 41.54% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Toolathlon-Verified: 74.4% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro 0813 (max) ✓; Default agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Agents' Last Exam: 12.4% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max; High; OpenClaw
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveCodeBench: 87.53% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CyberGym: 83.3% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max 0813; official dated DeepSeek release aggregate
DeepSeek August 13 (Pro 0813) / July 31 (Flash 0731) official release. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
Humanity's Last Exam w/ tools: 60% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max 0813; official dated DeepSeek release aggregate
DeepSeek August 13 (Pro 0813) / July 31 (Flash 0731) official release. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
SWE-Marathon v1.1: 10.6% · Not used.
Not admitted after the September 6 model review: the collected 10.6 for DeepSeek V4 Pro Max on SWE-Marathon v1.1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ProgramBench v1 (Raw Pass Rate) ; Vals: 70.103% · Contributes. Source · Archived source review · deepseek/deepseek-v4-pro-0813 · reasoning_effort=max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
FrontierMath v2 Tier 4: 26.83% · Not used.
Not admitted after the September 6 model review: the collected 26.83 for DeepSeek V4 Pro Max on FrontierMath v2 Tier 4 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
SWE-bench Pro: 55.4% · Not used.
Not admitted after the September 6 model review: the collected 55.4 for DeepSeek V4 Pro Max on SWE-bench Pro still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
BrowseComp: 83.4% · Not used.
Not admitted after the September 6 model review: the collected 83.4 for DeepSeek V4 Pro Max on BrowseComp still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ARC-AGI-2: 61.3% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
SimpleQA: 57.9% · Not used.
Not admitted after the September 6 model review: the collected 57.9 for DeepSeek V4 Pro Max on SimpleQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MCP Atlas †: 73.6% · Not used.
Not admitted after the September 6 model review: the collected 73.6 for DeepSeek V4 Pro Max on MCP Atlas † still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
NL2Repo: 61.5% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max 0813; official dated DeepSeek release aggregate
DeepSeek August 13 (Pro 0813) / July 31 (Flash 0731) official release. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial. Corrects the formerly unresolved 61.1 to the model owner’s explicit 0813 result, 61.5; the old observation remains archived.
CoWorkBench: 66.3% · Not used.
Not admitted after the September 6 model review: the collected 66.3 for DeepSeek V4 Pro Max on CoWorkBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ²-Bench Telecom: 96.2% · Not used.
Not admitted after the September 6 model review: the collected 96.2 for DeepSeek V4 Pro Max on τ²-Bench Telecom still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Main ; Anthropic H2H: 17.6% · Not used.
Not admitted after the September 6 model review: the collected 17.6 for DeepSeek V4 Pro Max on FrontierCode v1.1 Main ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EQ-Bench 3: 1570.2 · Not used.
Not admitted after the September 6 model review: the collected 1570.2 for DeepSeek V4 Pro Max on EQ-Bench 3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 39.587629% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro 0813 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 18% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro 0813 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
ProofBench v1.1 ; Vals: 50% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 82.3% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ARC-AGI-1: 90.5% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max; Low; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
EnterpriseOps-Gym-AA: 49.6% · Not used.
Not admitted after the September 6 model review: the collected 49.6 for DeepSeek V4 Pro Max on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 17.777778% · Not used. Source · Evaluator-reported · DeepSeek V4 Pro 0813 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
IFBench: 76.5% · Not used.
Not admitted after the September 6 model review: the collected 76.5 for DeepSeek V4 Pro Max on IFBench still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Mystery Game Puzzles v1.0.4 ; Epoch: 43% · Not used.
Not admitted after the September 6 model review: the collected 43 for DeepSeek V4 Pro Max on Mystery Game Puzzles v1.0.4 ; Epoch still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Tax Agent Bench v1 ; Vals: 58.66% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-pro-0813; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA Intelligence Index v4.3.2 (score): 35.996779 · Not used. Source · Evaluator-reported · DeepSeek V4 Pro Max (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Restored on 2026-10-28: 23.063298 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1254.44 · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max (max); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1254.52 recorded 2026-10-27.
AA-LCR v1.1: 80.333333% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 56.711655% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 11.4% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1458.08 · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1448.52 recorded 2026-10-21.
HLE text-only ; AA harness: 41.010195% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 14.141414% · Contributes. Source · Evaluator-reported · DeepSeek V4 Pro Max (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

Mistral Large 4 · 21 used / 22 collected

21 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±9.4 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

AA-Briefcase v1.1 (Elo): 1390.8 · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (none); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1391.14 recorded 2026-10-21.
GDPval-AA v2.1 (Elo): 1433.85 · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (none); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1420.33 recorded 2026-10-21.
AutomationBench-AA: 59.9013624681% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (none); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Exact model and effort in the evaluator chart payload. Preview release; scores may change at general availability.
Terminal-Bench 4.0 ; AA harness: 26.7676767677% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (none); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Exact model and effort in the evaluator chart payload. Preview release; scores may change at general availability.
SciCode: 54.1666666667% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (none); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Exact model and effort in the evaluator chart payload. Preview release; scores may change at general availability.
HLE text-only ; AA harness: 35.0324374421% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (none); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Exact model and effort in the evaluator chart payload. Preview release; scores may change at general availability.
GDP.pdf ; AA harness: 18.6% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (none); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Exact model and effort in the evaluator chart payload. Preview release; scores may change at general availability.
CritPt: 10.5714285714% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (none); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Exact model and effort in the evaluator chart payload. Preview release; scores may change at general availability.
AA-Omniscience: -5.3 · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (none); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Restored on 2026-10-28: -7.5 recorded 2026-10-28 came from the evaluator's renamed AA-Omniscience (AA-Omniscience v1.1: Knowledge and Hallucination Benchmark), a different version, so this row keeps its earlier value. Exact model and effort in the evaluator chart payload. Preview release; scores may change at general availability.
AA-LCR v1.1: 81.3333333333% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (none); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Exact model and effort in the evaluator chart payload. Preview release; scores may change at general availability.
MMMU-Pro: 76.4161849711% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (none); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Independent MMMU-Pro aggregate for this effort. Preview release; scores may change at general availability.
AA Intelligence Index v4.3.2 (score): 38.3770408917 · Not used. Source · Evaluator-reported · Mistral Large 4 Preview (none); Artificial Analysis independent evaluation, 2026-10-06; fallback not reported
Restored on 2026-10-28: 25.286451 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Reference composite only, never scored alongside its component evaluations. Preview release; scores may change at general availability.
BioMysteryBench v1 ; Vals: 67.04% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (high); Vals AI BioMysteryBench harness; provider Mistral AI; fallback not reported
Vals AI model page, headline accuracy 67.04 ± 2.67, rank 20/27. Vals notes some benchmarks may use a different provider or parameters.
Code Migration †: 30.56% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (high); Vals AI Code Migration harness; provider Mistral AI; fallback not reported
Vals AI model page, headline accuracy 30.56 ± 4.22, rank 41/77. Vals notes some benchmarks may use a different provider or parameters.
Excel Modeling Benchmark ; Vals: 56.21% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (high); Vals AI EMB harness; provider Mistral AI; fallback not reported
Vals AI model page, headline accuracy 56.21 ± 3.22, rank 46/74. Vals notes some benchmarks may use a different provider or parameters.
Finance Agent v2 ; Vals: 54.68% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (high); Vals AI Finance Agent (v2) harness; provider Mistral AI; fallback not reported
Vals AI model page, headline accuracy 54.68 ± 0.58, rank 24/78. Vals notes some benchmarks may use a different provider or parameters.
Harvey LAB (Vals): 15.83% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (high); Vals AI Harvey's Legal Agent Benchmark harness; provider Mistral AI; fallback not reported
Vals AI model page, headline accuracy 15.83 ± 2.96, rank 9/79. Vals notes some benchmarks may use a different provider or parameters.
Legal Research Bench ; Vals: 31.73% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (high); Vals AI Legal Research Bench harness; provider Mistral AI; fallback not reported
Vals AI model page, headline accuracy 31.73 ± 3.23, rank 42/78. Vals notes some benchmarks may use a different provider or parameters.
MedCode ; Vals: 40.69% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (high); Vals AI MedCode harness; provider Mistral AI; fallback not reported
Vals AI model page, headline accuracy 40.69 ± 2.17, rank 61/107. Vals notes some benchmarks may use a different provider or parameters.
MedScribe ; Vals: 80.43% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (high); Vals AI MedScribe harness; provider Mistral AI; fallback not reported
Vals AI model page, headline accuracy 80.43 ± 2.0, rank 57/109. Vals notes some benchmarks may use a different provider or parameters.
ProofBench v1.1 ; Vals: 10% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (high); Vals AI ProofBench v1.1 harness; provider Mistral AI; fallback not reported
Vals AI model page, headline accuracy 10.0 ± 3.02, rank 45/50. Vals notes some benchmarks may use a different provider or parameters.
Vibe Code Bench v1.1 ; Vals: 78.4% · Contributes. Source · Evaluator-reported · Mistral Large 4 Preview (high); Vals AI Vibe Code Bench v1.1 harness; provider Mistral AI; fallback not reported
Vals AI model page, headline accuracy 78.4 ± 3.55, rank 28/112. Vals notes some benchmarks may use a different provider or parameters.

GPT-5.6 Luna · 57 used / 85 collected

51 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.1 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1492.73 · Not used. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 80.898876% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 67.1875% · Contributes. Source · Evaluator-reported · gpt-5-6-luna; max; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1452.599383 · Contributes. Source · Evaluator-reported · gpt-5.6-luna-xhigh; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 52 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 91.111111% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 78.3% · Not used. Source · Evaluator-reported · GPT-5.6 Luna (max)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 55.04% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 67.12% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 53.587963% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 39.481001% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 36.54% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 1.25% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1342.42 · Not used. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 84.39% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 93% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 84.03% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 76.17% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vals Index: 59.88% · Not used.
Held-out composite, never a scoring input
Public Benefits Bench ; Vals: 61.16% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 42.39% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Omniscience: -10.283333 · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 86.04% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 73.559381% · Contributes. Source · Evaluator-reported · gpt-5.6-luna-max; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
CorpFin v2 ; Vals: 64.22% · Not used.
Not admitted after the September 6 model review: the collected 64.22 for GPT-5.6 Luna on CorpFin v2 ; Vals still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MortgageTax ; Vals: 67.29% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SAGE ; Vals: 44.22% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Code Migration †: 44.55% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Agents' Last Exam: 30.3% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna; XHigh; Codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Terminal-Bench v3.0: 14.3% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CyberGym: 77.9% · Not used.
Not admitted after the September 6 model review: the collected 77.9 for GPT-5.6 Luna on CyberGym still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Humanity's Last Exam w/ tools: 48.9% · Not used.
Not admitted after the September 6 model review: the collected 48.9 for GPT-5.6 Luna on Humanity's Last Exam w/ tools still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Design Arena (Elo): 1283 · Not used.
Not admitted after the September 6 model review: the collected 1283 for GPT-5.6 Luna on Design Arena (Elo) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vals Multimodal Index: 69.06% · Not used.
Held-out composite, never a scoring input
SWE-Marathon v1.1: 24.4% · Not used.
Not admitted after the September 6 model review: the collected 24.4 for GPT-5.6 Luna on SWE-Marathon v1.1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ProgramBench v1 (Raw Pass Rate) ; Vals: 68.291% · Contributes. Source · Archived source review · openai/gpt-5.6-luna · reasoning_effort=max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
SkillsBench †: 60.45% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierMath v2 Tier 4: 60.97561% · Contributes. Source · Evaluator-reported · gpt-5.6-luna_max; task version 2.0.0
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
CursorBench v3.2: 61.1% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna Max; Cursor harness
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Pro: 62.7% · Not used.
Not admitted after the September 6 model review: the collected 62.7 for GPT-5.6 Luna on SWE-bench Pro still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
BrowseComp: 83.3% · Not used.
Not admitted after the September 6 model review: the collected 83.3 for GPT-5.6 Luna on BrowseComp still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MMMU-Pro: 78.554913% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (xhigh); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
ARC-AGI-2: 59.5% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
OSWorld 2.0 ; Anthropic H2H: 45.6% · Not used.
Not admitted after the September 6 model review: the collected 45.6 for GPT-5.6 Luna on OSWorld 2.0 ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ExploitBench: 33.2% · Not used.
Inherited comparator metric/scaffold equivalence to Astra Cap Percent is not established; the source flags possible contamination. Excluded pending protocol audit.
SimpleQA: 41.7% · Not used.
Not admitted after the September 6 model review: the collected 41.7 for GPT-5.6 Luna on SimpleQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
FrontierCode v1.1 Extended: 55.06% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna; max; codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Creative Writing v3: 1826.6 · Contributes. Source · Evaluator-reported · gpt-5.6-luna; published evaluation, effort unspecified
Exact checkpoint on the evaluator-hosted board. A starred model is externally submitted; it is not claimed as independently rerun by the benchmark owner.
AIIQ Composite IQ: 129 · Not used.
Held-out composite, never a scoring input
EQ-Bench 4: 1156.3 · Contributes. Source · Evaluator-reported · openai/gpt-5.6-luna; evaluator published configuration, effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ERQA: 62.42% · Not used.
Not admitted after the September 6 model review: the collected 62.42 for GPT-5.6 Luna on ERQA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AutomationBench ; Anthropic H2H: 14.9% · Not used.
Not admitted after the September 6 model review: the collected 14.9 for GPT-5.6 Luna on AutomationBench ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 31.134021% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 20.571429% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (xhigh); Artificial Analysis published evaluation harness
Best published result among 6 reasoning configurations of the exact AA model release; fallback configurations excluded.
Terminal-Bench 4.0: 17.3% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna; max; Codex
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
GDP.pdf: 22.7% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (Medium reasoning); Surge original evaluation
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 60% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 77.06% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MineBench: 1784 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
ARC-AGI-1: 88% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
ARC-AGI-3 (standard harness): 0.18% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
HealthBench: 55.4% · Not used. Source · Developer-reported · GPT-5.6 Luna; OpenAI published HealthBench research evaluation; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
HealthBench Hard: 31.4% · Not used. Source · Developer-reported · GPT-5.6 Luna; OpenAI published HealthBench research evaluation; effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
HealthBench Professional (length-adjusted): 55.7% · Not used.
Source selects the maximum score across efforts; exact highest-effort result is not established.
Harvey LAB-AA: 87.9% · Not used.
Not admitted after the September 6 model review: the collected 87.9 for GPT-5.6 Luna on Harvey LAB-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EnterpriseOps-Gym-AA: 40.82% · Not used.
Not admitted after the September 6 model review: the collected 40.82 for GPT-5.6 Luna on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 19.444444% · Not used. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
ITBench-AA: 40.320151% · Not used. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes ITBench-AA, so every model is scored on its current set.
Vending-Bench 2: 4094.712 · Contributes. Source · Evaluator-reported · GPT-5.6 Luna; Andon Labs Vending-Bench 2
Compare reasoning configurations using their mean over runs. Geometric mean, individual best runs, Fireworks Kimi endpoint, Vending Arena, and April DeepSeek Pro are not substituted.
ZeroBench: 21% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); official main questions
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Mystery Game Puzzles v1.0.4 ; Epoch: 21% · Contributes. Source · Evaluator-reported · gpt-5.6-luna_max; task version 1.0.4
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
BioMysteryBench v1 ; Vals: 61.48% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Tax Agent Bench v1 ; Vals: 60.81% · Contributes. Source · Evaluator-reported · openai_gpt-5.6-luna; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
RuneBench (30m, mean ln(1 + XP/min)): 5.285493 · Contributes. Source · Archived source review · gpt56luna-xhigh
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 1846 · Contributes. Source · Archived source review · GPT-5.6-Luna
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Roboflow Vision Evals (six-task mean, high tier): 76.05% · Contributes. Source · Archived source review · Require a complete six-task high-tier panel, the highest published benchmark tier. Retain high-tier scores even when lower than low-tier. Do not substitute the best of three repetitions: use the published mean. Each panel enters the fit once; do not also fit its components.
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1519 · Contributes. Source · Archived source review · gpt-5.6-luna-xhigh (codex-harness)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 37.324424 · Not used. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1339.22 · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1339.71 recorded 2026-10-22.
AA-LCR v1.1: 83.666667% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 50.208618% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 24% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1460.3 · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1458.32 recorded 2026-10-22.
HLE text-only ; AA harness: 39.481001% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 11.616162% · Contributes. Source · Evaluator-reported · GPT-5.6 Luna (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

Sonnet 5 · 54 used / 78 collected

51 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.1 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1504.87 · Not used. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 80.524345% · Contributes. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 53.846154% · Contributes. Source · Evaluator-reported · claude-sonnet-5; max; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1462.365844 · Contributes. Source · Evaluator-reported · claude-sonnet-5-high; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 55 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 91.111111% · Contributes. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 77% · Not used. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 53.91% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 66.32% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 54.282407% · Contributes. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 1 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 41.28823% · Contributes. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 41.83% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 5% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1358.11 · Not used. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 76.05% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 79.6% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 83.92% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 75.63% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vals Index: 59.61% · Not used.
Held-out composite, never a scoring input
Public Benefits Bench ; Vals: 66.03% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 47.54% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Omniscience: 16.45 · Contributes. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 87.55% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 76.039714% · Contributes. Source · Evaluator-reported · claude-sonnet-5-xhigh-effort; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
CorpFin v2 ; Vals: 67.95% · Not used.
Not admitted after the September 6 model review: the collected 67.95 for Sonnet 5 on CorpFin v2 ; Vals still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MortgageTax ; Vals: 70.03% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SAGE ; Vals: 48.92% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Code Migration †: 44.39% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Toolathlon-Verified: 71.6% · Contributes. Source · Evaluator-reported · Claude Claude Sonnet 5 (max) ✓; Default agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Terminal-Bench v3.0: 14.6% · Contributes. Source · Evaluator-reported · Claude Sonnet 5 (max); Claude Code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveCodeBench: 82.43% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CyberGym: 52.7% · Not used.
Not admitted after the September 6 model review: the collected 52.7 for Sonnet 5 on CyberGym still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Humanity's Last Exam w/ tools: 57.4% · Not used.
Not admitted after the September 6 model review: the collected 57.4 for Sonnet 5 on Humanity's Last Exam w/ tools still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Design Arena (Elo): 1297 · Not used.
Not admitted after the September 6 model review: the collected 1297 for Sonnet 5 on Design Arena (Elo) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vals Multimodal Index: 68.83% · Not used.
Held-out composite, never a scoring input
ProgramBench v1 (Raw Pass Rate) ; Vals: 72.067% · Contributes. Source · Archived source review · anthropic/claude-sonnet-5 · compute=max
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
SkillsBench †: 46.48% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierMath v2 Tier 4: 29.268293% · Contributes. Source · Evaluator-reported · claude-sonnet-5_max; task version 2.0.0
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
CursorBench v3.2: 61.5% · Contributes. Source · Evaluator-reported · Sonnet 5 Max; Cursor harness
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
APEX-Agents Mean Criteria Passed: 48.5% · Contributes. Source · Evaluator-reported · claude-sonnet-5; high; loop_truncated_tools_agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Pro: 63.2% · Not used.
Not admitted after the September 6 model review: the collected 63.2 for Sonnet 5 on SWE-bench Pro still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
CharXiv (with CI / RQ): 88.3% · Not used.
Not admitted after the September 6 model review: the collected 88.3 for Sonnet 5 on CharXiv (with CI / RQ) still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
APEX-SWE (Pass@1, Terminus-2): 46.4% · Contributes. Source · Evaluator-reported · claude-sonnet-5-max; max; terminus-2
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
BrowseComp: 84.7% · Not used.
Not admitted after the September 6 model review: the collected 84.7 for Sonnet 5 on BrowseComp still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MMMU-Pro: 77.283237% · Contributes. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
FrontierCode v1.1 Extended: 56.18% · Contributes. Source · Evaluator-reported · Claude Sonnet 5; xhigh; claude-code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
OSWorld-Verified: 81.2% · Not used.
Not admitted after the September 6 model review: the collected 81.2 for Sonnet 5 on OSWorld-Verified still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Creative Writing v3: 1787.6 · Contributes. Source · Evaluator-reported · claude-sonnet-5; published evaluation, effort unspecified
Exact checkpoint on the evaluator-hosted board. A starred model is externally submitted; it is not claimed as independently rerun by the benchmark owner.
OfficeQA Pro †: 59.4% · Not used.
Not admitted after the September 6 model review: the collected 59.4 for Sonnet 5 on OfficeQA Pro † still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EQ-Bench 4: 1236 · Contributes. Source · Evaluator-reported · claude-sonnet-5; evaluator published configuration, effort unspecified
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierCode v1.1 Main ; Anthropic H2H: 42.7% · Not used.
Not admitted after the September 6 model review: the collected 42.7 for Sonnet 5 on FrontierCode v1.1 Main ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AutomationBench ; Anthropic H2H: 13.5% · Not used.
Not admitted after the September 6 model review: the collected 13.5 for Sonnet 5 on AutomationBench ; Anthropic H2H still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Harvey Legal Agent Benchmark ; held-out: 5.8% · Not used.
Not admitted after the September 6 model review: the collected 5.8 for Sonnet 5 on Harvey Legal Agent Benchmark ; held-out still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 37.319588% · Contributes. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 16.857143% · Contributes. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
Terminal-Bench 4.0: 12.4% · Contributes. Source · Evaluator-reported · Sonnet 5; max; Claude Code
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProofBench v1.1 ; Vals: 77% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 81.33% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MineBench: 1720 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
Harvey LAB-AA: 90.07% · Not used.
Not admitted after the September 6 model review: the collected 90.07 for Sonnet 5 on Harvey LAB-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EnterpriseOps-Gym-AA: 44.67% · Not used.
Not admitted after the September 6 model review: the collected 44.67 for Sonnet 5 on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 55% · Not used. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
AA-AnalystAgent: 46.25% · Not used. Source · Evaluator-reported · Claude Sonnet 5 (Adaptive Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-AnalystAgent, so every model is scored on its current set.
Vending-Bench 2: 6377.701667 · Contributes. Source · Evaluator-reported · Claude Sonnet 5; Andon Labs Vending-Bench 2
Compare reasoning configurations using their mean over runs. Geometric mean, individual best runs, Fireworks Kimi endpoint, Vending Arena, and April DeepSeek Pro are not substituted.
ZeroBench: 13% · Contributes. Source · Evaluator-reported · Claude Sonnet 5 (max); official main questions
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Mystery Game Puzzles v1.0.4 ; Epoch: 35% · Contributes. Source · Evaluator-reported · claude-sonnet-5_max; task version 1.0.4
Exact Epoch task version; selected best mean score across reported reasoning configurations. No pooling of task versions or substitution of best scorer output.
Tax Agent Bench v1 ; Vals: 62.27% · Contributes. Source · Evaluator-reported · anthropic_claude-sonnet-5; Vals max effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
RuneBench (30m, mean ln(1 + XP/min)): 4.805888 · Contributes. Source · Archived source review · sonnet5-xhigh
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 1570 · Contributes. Source · Archived source review · Claude Sonnet 5 (xhigh)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1537 · Contributes. Source · Archived source review · claude-sonnet-5-high
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 38.163871 · Not used. Source · Evaluator-reported · Sonnet 5 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1356.6 · Contributes. Source · Evaluator-reported · Sonnet 5 (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1357.21 recorded 2026-10-22.
AA-LCR v1.1: 82% · Contributes. Source · Evaluator-reported · Sonnet 5 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 36.512676% · Contributes. Source · Evaluator-reported · Sonnet 5 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 13.2% · Contributes. Source · Evaluator-reported · Sonnet 5 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1463.86 · Contributes. Source · Evaluator-reported · Sonnet 5 (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1462.47 recorded 2026-10-22.
HLE text-only ; AA harness: 41.28823% · Contributes. Source · Evaluator-reported · Sonnet 5 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 14.141414% · Contributes. Source · Evaluator-reported · Sonnet 5 (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

Qwen3.8-27B · 44 used / 60 collected

43 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.3 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1466.21 · Not used. Source · Evaluator-reported · Qwen3.8 27B (xhigh); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 79.775281% · Contributes. Source · Evaluator-reported · Qwen3.8 27B (xhigh); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 42.2% · Not used.
Not admitted after the September 6 model review: the collected 42.2 for Qwen3.8-27B on DeepSWE v1.1 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
LMArena (Chatbot Arena) ; Text: 1435.892663 · Contributes. Source · Evaluator-reported · qwen3.8-27b; Arena text overall; style control enabled
Best published reasoning-mode rating within the same text overall board. Different leaderboard slices and undated earlier DeepSeek checkpoints are not pooled.
AA Intelligence Index v4.1.1 (score): 52 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 90.505051% · Contributes. Source · Evaluator-reported · Qwen3.8 27B (xhigh); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 77.3% · Not used. Source · Evaluator-reported · Qwen3.8 27B (xhigh)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 48.55% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 59.66% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 46.643519% · Contributes. Source · Evaluator-reported · Qwen3.8 27B (xhigh); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 33.920297% · Contributes. Source · Evaluator-reported · Qwen3.8 27B (xhigh); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 36.06% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 11.25% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1401.81 · Not used. Source · Evaluator-reported · Qwen3.8 27B (xhigh); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 83.85% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 86% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 82.43% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 70.85% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MedCode ; Vals: 28.7% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Omniscience: -7.95 · Contributes. Source · Evaluator-reported · Qwen3.8 27B (Non-reasoning); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 84.34% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 75.269107% · Contributes. Source · Evaluator-reported · qwen3.8-27b; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
MortgageTax ; Vals: 64.94% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SAGE ; Vals: 52.4% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Code Migration †: 14.16% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Agents' Last Exam: 20.4% · Not used.
Not admitted after the September 6 model review: the collected 20.4 for Qwen3.8-27B on Agents' Last Exam still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
LiveCodeBench: 84% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ProgramBench v1 (Raw Pass Rate) ; Vals: 11.168% · Contributes. Source · Archived source review · alibaba/qwen3.8-27b · reasoning_effort=xhigh
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
SWE-bench Pro: 61.7% · Not used. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CharXiv (with CI / RQ): 90.2% · Contributes. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; With CI
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MMMU-Pro: 76.300578% · Contributes. Source · Evaluator-reported · Qwen3.8 27B (xhigh); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
JobBench: 33.4% · Contributes. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
BabyVision (with CI): 85.6% · Contributes. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; With CI
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
OSWorld-Verified: 84.3% · Contributes. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Creative Writing v3: 1669.1 · Contributes. Source · Evaluator-reported · *Qwen/Qwen3.8-27B; published evaluation, effort unspecified
Exact checkpoint on the evaluator-hosted board. A starred model is externally submitted; it is not claimed as independently rerun by the benchmark owner.
NL2Repo: 42.3% · Contributes. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CoWorkBench: 70.7% · Not used. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
ERQA: 65.5% · Not used. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vision2Web (Avg. Frontend/Webpage/etc.): 62.9% · Not used. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
WebArena-Verified: 64.8% · Not used. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
τ³-Banking: 48.041237% · Contributes. Source · Evaluator-reported · Qwen3.8 27B (xhigh); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 5.428571% · Contributes. Source · Evaluator-reported · Qwen3.8 27B (xhigh); Artificial Analysis published evaluation harness
Best published result among 4 reasoning configurations of the exact AA model release; fallback configurations excluded.
ProofBench v1.1 ; Vals: 16% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 64.85% · Contributes. Source · Evaluator-reported · alibaba_qwen3.8-27b; Vals xhigh effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
EnterpriseOps-Gym-AA: 44.23% · Not used.
Not admitted after the September 6 model review: the collected 44.23 for Qwen3.8-27B on EnterpriseOps-Gym-AA still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MLCR-AA (Medical Long Context Reasoning): 21.666667% · Not used. Source · Evaluator-reported · Qwen3.8 27B (xhigh); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
IFBench: 79.5% · Not used. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AndroidWorld: 81.9% · Not used. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MathVision (with CI): 94.6% · Contributes. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; With CI
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
OmniDocBench 1.5: 91.1% · Contributes. Source · Developer-reported · Qwen3.8-27B; published developer evaluation; benchmark-specific configuration
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
RuneBench (30m, mean ln(1 + XP/min)): 2.794282 · Contributes. Source · Archived source review · qwen38
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
Code Arena WebDev (overall): 1594 · Contributes. Source · Archived source review · qwen3.8-27b
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 33.676286 · Not used. Source · Evaluator-reported · Qwen3.8-27B (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Restored on 2026-10-28: 20.106876 recorded 2026-10-28 came from the evaluator's renamed AA Intelligence Index v4.3.2 (score) (Artificial Analysis Intelligence Index v5.0), a different version, so this row keeps its earlier value. Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1393.08 · Contributes. Source · Evaluator-reported · Qwen3.8-27B (xhigh); Artificial Analysis independent evaluation, 2026-10-30; fallback not reported
Daily evaluator refresh; replaces 1392.67 recorded 2026-10-27.
AA-LCR v1.1: 82% · Contributes. Source · Evaluator-reported · Qwen3.8-27B (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 48.242444% · Contributes. Source · Evaluator-reported · Qwen3.8-27B (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 16.4% · Contributes. Source · Evaluator-reported · Qwen3.8-27B (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1427.26 · Contributes. Source · Evaluator-reported · Qwen3.8-27B (xhigh); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1416.92 recorded 2026-10-21.
HLE text-only ; AA harness: 33.920297% · Contributes. Source · Evaluator-reported · Qwen3.8-27B (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 5.555556% · Contributes. Source · Evaluator-reported · Qwen3.8-27B (xhigh); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.

DeepSeek V4 Flash · 44 used / 66 collected

42 distinct benchmark families contribute. Domains without direct evidence are estimates.

Reporting sensitivity: ±1.3 points. Sensitivity to selectively published results. This stress range is not a confidence interval or a probability statement.

GDPval-AA v2 (Elo): 1580.66 · Not used. Source · Evaluator-reported · DeepSeek V4 Flash Vision (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes GDPval-AA v2 (Elo), so every model is scored on its current set.
Terminal-Bench 2.1: 78.651685% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash 0731 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
DeepSWE v1.1: 53.318584% · Contributes. Source · Evaluator-reported · deepseek-v4-flash; max; mini-swe-agent
Selected the best pass@1 across published reasoning efforts within mini-swe-agent. Pass@4 is not substituted.
LMArena (Chatbot Arena) ; Text: 1435 · Not used.
Not admitted after the September 6 model review: the collected 1435 for DeepSeek V4 Flash on LMArena (Chatbot Arena) ; Text still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA Intelligence Index v4.1.1 (score): 52 · Not used.
Held-out composite, never a scoring input
GPQA Diamond: 91.313131% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash Vision (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
AA-LCR: 66% · Not used. Source · Evaluator-reported · DeepSeek V4 Flash 0731 (Reasoning, Max Effort)
Current AA result is LCR v1.1. The inherited AA-LCR row has no reviewed version match; a new version cannot silently substantiate the older value.
Finance Agent v2 ; Vals: 49.52% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Excel Modeling Benchmark ; Vals: 56.98% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SciCode: 50.347222% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash 0731 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
Humanity's Last Exam (no tools): 38.554217% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash 0731 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
Legal Research Bench ; Vals: 30.29% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Harvey LAB (Vals): 8.33% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AA-Briefcase Overall Elo: 1262.58 · Not used. Source · Evaluator-reported · DeepSeek V4 Flash 0731 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes AA-Briefcase Overall Elo, so every model is scored on its current set.
MedScribe ; Vals: 80.36% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Verified ; Vals: 88.8% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LegalBench ; Vals: 77.71% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
TaxEval v2 ; Vals: 70.69% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
KingBench 3: 72.5% · Not used.
Not admitted after the September 6 model review: the collected 72.5 for DeepSeek V4 Flash on KingBench 3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Vals Index: 53.57% · Not used.
Held-out composite, never a scoring input
MedCode ; Vals: 41.41% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
AutomationBench v1.0.6: 25.1% · Not used.
Not admitted after the September 6 model review: the collected 25.1 for DeepSeek V4 Flash on AutomationBench v1.0.6 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
AA-Omniscience: -14.283333 · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash 0731 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
MMLU Pro ; Vals: 86.21% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
LiveBench: 74.170786% · Contributes. Source · Evaluator-reported · deepseek-v4-flash-0731; LiveBench 2026-06-25 release
Task mapping archived from categories_2026_06_25.json. Aggregate follows the official UI, not a flat mean of the 23 tasks. Undated old DeepSeek checkpoints excluded.
CorpFin v2 ; Vals: 61.85% · Not used.
Not admitted after the September 6 model review: the collected 61.85 for DeepSeek V4 Flash on CorpFin v2 ; Vals still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
Code Migration †: 38.63% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Toolathlon-Verified: 70.7% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash 0731 (max) ✓; Default agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Agents' Last Exam: 25.2% · Not used.
Not admitted after the September 6 model review: the collected 25.2 for DeepSeek V4 Flash on Agents' Last Exam still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
LiveCodeBench: 87.26% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
CyberGym: 76.7% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash 0731; official dated DeepSeek release aggregate
DeepSeek August 13 (Pro 0813) / July 31 (Flash 0731) official release. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
Humanity's Last Exam w/ tools: 45.1% · Not used.
Not admitted after the September 6 model review: the collected 45.1 for DeepSeek V4 Flash on Humanity's Last Exam w/ tools still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
SkillsBench †: 50.67% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
FrontierMath v2 Tier 4: 24.4% · Not used.
Not admitted after the September 6 model review: the collected 24.4 for DeepSeek V4 Flash on FrontierMath v2 Tier 4 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
APEX-Agents Mean Criteria Passed: 51.6% · Contributes. Source · Evaluator-reported · deepseek-v4-flash; max; loop_truncated_tools_agent
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
SWE-bench Pro: 52.6% · Not used.
Not admitted after the September 6 model review: the collected 52.6 for DeepSeek V4 Flash on SWE-bench Pro still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
APEX-SWE (Pass@1, Terminus-2): 46.9% · Contributes. Source · Evaluator-reported · deepseek-v4-flash; max; terminus-2
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
BrowseComp: 73.2% · Not used.
Not admitted after the September 6 model review: the collected 73.2 for DeepSeek V4 Flash on BrowseComp still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
ARC-AGI-2: 61.4% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
FrontierCode v1.1 Extended: 31.7% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash 0731; high; chisel
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Creative Writing v3: 1438.3 · Contributes. Source · Evaluator-reported · *deepseek-ai/DeepSeek-V4-Flash-0731; published evaluation, effort unspecified
Exact checkpoint on the evaluator-hosted board. A starred model is externally submitted; it is not claimed as independently rerun by the benchmark owner.
MathArena ; ArXivMath Jun 2026: 42.86% · Not used.
Not admitted after the September 6 model review: the collected 42.86 for DeepSeek V4 Flash on MathArena ; ArXivMath Jun 2026 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
MathArena ; BrokenArXiv Jun 2026: 16.67% · Not used.
Not admitted after the September 6 model review: the collected 16.67 for DeepSeek V4 Flash on MathArena ; BrokenArXiv Jun 2026 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
NL2Repo: 54.2% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash 0731; official dated DeepSeek release aggregate
DeepSeek August 13 (Pro 0813) / July 31 (Flash 0731) official release. Reviewed developer-reported result for the exact named reference model. Cross-evaluator harness differences are retained as a limitation; no independent evaluation or matched-harness claim. Sole reported aggregate, not a selected individual trial.
MathArena ; ArXivMath May 2026: 55.83% · Not used.
Not admitted after the September 6 model review: the collected 55.83 for DeepSeek V4 Flash on MathArena ; ArXivMath May 2026 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
EQ-Bench 3: 1491.3 · Not used.
Not admitted after the September 6 model review: the collected 1491.3 for DeepSeek V4 Flash on EQ-Bench 3 still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
τ³-Banking: 41.030928% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash Vision (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
CritPt: 16.571429% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash 0731 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Best published result among 2 reasoning configurations of the exact AA model release; fallback configurations excluded.
ProofBench v1.1 ; Vals: 56% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Vibe Code Bench v1.1 ; Vals: 74.74% · Contributes. Source · Evaluator-reported · deepseek_deepseek-v4-flash-0731; Vals high effort
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
MineBench: 1601 · Not used.
Unresolved Pro/base model identities in source catalog; exclude until matched.
ARC-AGI-1: 89% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash; Max; ARC Prize standard harness
Best reasoning variant within the verified standard-harness summary. Provider Adapter, public tasks and community agent scores are not substituted.
MLCR-AA (Medical Long Context Reasoning): 13.333333% · Not used. Source · Evaluator-reported · DeepSeek V4 Flash 0731 (Reasoning, Max Effort); Artificial Analysis published evaluation harness
Retired from scoring on 2026-10-21: Artificial Analysis no longer publishes MLCR-AA (Medical Long Context Reasoning), so every model is scored on its current set.
CWE-bench: 30.4% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash; high; opencode
Reconciled against the exact model/configuration in the evaluator’s public source. Updated source value recorded explicitly; best published reasoning mode selected within the same benchmark protocol.
Mystery Game Puzzles v1.0.4 ; Epoch: 34% · Not used.
Not admitted after the September 6 model review: the collected 34 for DeepSeek V4 Flash on Mystery Game Puzzles v1.0.4 ; Epoch still lacks a matched source entry establishing its checkpoint and benchmark-specific metric, tools and retry setup. This is a citation/comparability gap, not a finding that the result is false or useless. More or maximum reasoning effort is not required.
RuneBench (30m, mean ln(1 + XP/min)): 4.195298 · Contributes. Source · Archived source review · deepseekflash0731
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
VoxelBench (text-to-voxel, Glicko-2): 1436 · Contributes. Source · Archived source review · DeepSeek V4 Flash (July 2026)
Matched exact metric/value and the archived benchmark-specific configuration review. Unknown native effort stays explicitly unknown.
AA Intelligence Index v4.3.2 (score): 34.330513 · Not used. Source · Evaluator-reported · DeepSeek V4 Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AA-Briefcase v1.1 (Elo): 1256.76 · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1257.69 recorded 2026-10-22.
AA-LCR v1.1: 79.666667% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
AutomationBench-AA: 53.967719% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDP.pdf ; AA harness: 11% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
GDPval-AA v2.1 (Elo): 1433.47 · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash (max); Artificial Analysis independent evaluation, 2026-10-27; fallback not reported
Daily evaluator refresh; replaces 1433.24 recorded 2026-10-22.
HLE text-only ; AA harness: 38.554217% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
Terminal-Bench 4.0 ; AA harness: 12.121212% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash (max); Artificial Analysis independent evaluation, 2026-10-21; fallback not reported
Daily evaluator refresh; the evaluator published this benchmark after admission.
BioMysteryBench v1 ; Vals: 64.44% · Contributes. Source · Evaluator-reported · DeepSeek V4 Flash (high); Vals AI BioMysteryBench harness; provider DeepSeek; fallback not reported
Vals AI model page, headline accuracy 64.44 ± 1.28, rank 22/27. Vals notes some benchmarks may use a different provider or parameters.