Since it launched in March, the ARC-AGI-3 benchmark has become the test AI labs point to when they want to argue their models can handle unfamiliar work. It drops a model into a small 2D video game and explains nothing: no rules, no controls, no stated goal. The model has to press things, watch what changes, and work out what the game wants. At launch, every frontier system scored under 1%.

Last Friday the leaderboard moved. Anthropic's Claude Opus 5 posted a verified 30.2% on ARC-AGI-3, roughly four times the previous verified high of 7.8%, which had been set by OpenAI's GPT-5.6 Sol. The ARC Prize Foundation, which builds and administers the benchmark, ran that test itself.

On Wednesday OpenAI published a reply of sorts. Its researchers argue that Sol's weak showing was mostly the fault of the software wrapped around the model rather than the model itself, and that with two settings changed, Sol scored 38.3% on the 25 public games.

The post turns on one sentence: "Benchmarks rarely measure AI models in isolation." The old assumption was that a score belongs to the model, so a low number means the model cannot do the task. OpenAI's claim is that a score belongs to the model plus every piece of code surrounding it, and that a low number can instead mean the surrounding code was discarding the model's work between moves. For anyone reading leaderboards as a proxy for capability, that changes what the numbers are evidence of.

What the test harness was doing

The wrapper has a name: the harness. It is the code that decides what the model sees each turn, what it remembers, and which tools it can reach. ARC's harness is deliberately plain, on the reasoning that a plain setup exposes model weaknesses and keeps comparisons between labs fair.

OpenAI found two things it disliked. Models like Sol generate private reasoning before each action, essentially working notes on why they are doing something. ARC's harness deleted those notes after every move, so the model could see that it had pressed a button but not what theory the press was testing. Second, once the running history passed 175,000 characters, the oldest entries were dropped, which could erase the early discovery that explained the whole game.

OpenAI rebuilt the harness to keep reasoning between moves and swapped deletion for a technique called compaction, which compresses old history into a condensed package the model carries into a fresh workspace instead of losing the beginning. Score went from 13.3% to 38.3%. Output tokens, the units of text the model generates and the thing customers pay for, fell roughly sixfold. No retraining was involved.

Why 38.3% is not a leaderboard record

That figure is OpenAI running its own configuration and reporting its own result. ARC's verified testing policy rules out client-side harnesses, hand-built tools and tailored model settings, and routes harness-driven results to a separate community leaderboard where scores are self-reported and ARC explicitly cautions against treating them as evidence of progress toward general intelligence.

The public games are also the easy half. On the held-out set that models cannot study in advance, Sol's verified score drops from 13.33% to 7.78%. Nobody knows what 38.3% becomes under the same haircut.

Nor is 38.3% the highest self-reported harness number on those games. Earlier this month a team at Impossible Research reported that its Schema harness reached about 99% on the public set using Claude Opus 4.8 and Fable 5, and 95.35% using Sol. ARC Prize president Greg Kamradt's public response was that without seeing the code, it is hard to judge how much human problem-solving was baked in.

One more number is worth holding onto. OpenAI estimates the average human tester scores 48% on these games. Its improved run still sits below that.

The post closes with a recommendation to use OpenAI's Responses API rather than its older interface, which is a sales pitch as much as a finding. The finding underneath survives it anyway: a benchmark score is a measurement of a whole system, and most published scores do not tell you how that system was assembled. The number to watch is not 38.3%. It is whatever the same setup scores on games it has never seen.