The ARC-AGI-3 framework, which tests AI models on their adaptability to unfamiliar environments and information, is now the go-to benchmark for evaluating an AI model’s contextual reasoning. It does this by having the model learn and complete a series of simple video games with zero prior knowledge. The models must test inputs, process positive and negative feedback, infer the game’s objectives and ultimately try to achieve them.
Although major strides have been made in other kinds of contextual reasoning, AI models have remained very poor at the type of task ARC-AGI-3 prescribes. When the benchmark launched in March, every single frontier system logged scores under 1%.
This has since crept up incrementally, with OpenAI's GPT-5.6 Sol managing a 7.8% score later in the year, a record smashed this week by rival firm Anthropic. Last Friday, Claude Opus 5 achieved a 30.2% score, placing it top of the leaderboard by a considerable margin. The test was administered by the ARC Prize Foundation, which created the benchmark.
OpenAI’s defensive response to the challenge
This Wednesday, researchers at OpenAI released a statement attempting to explain their model’s apparent weak showing. They asserted that the model itself was not to blame. Instead, it was the fault of the software that it was wrapped inside, which hamstrung its performance.
This software, called the harness, is code that manages the information available to the model on each turn and the toolset available to it. The ARC Prize Foundation prioritizes simplicity in its own harness, on the assumption that this lets the models speak for themselves and provides an even playing field.
However, OpenAI’s claim is that this stripped-down approach interfered with Sol’s methodology. The model generates private notes before each action it performs, allowing it to set each action within the context of a broader theory that it is testing. The ARC harness, OpenAI claims, wiped those notes after each move, resulting in an enforced amnesia and broken testing procedure.
Likewise, the game history available to the model had a maximum character count of 175,000. After this, the earliest logs were wiped to clear space, meaning that critical early discoveries could be wiped along with them. The result is that the model may have gotten quite adept at controlling the actions of a game, but forgotten what it was supposed to be doing in the first place.
The result, according to OpenAI, is that its model's chances were inadvertently scuppered before it had even provided a single input.
To prove the point, the researchers performed their own internal tests, which reportedly returned a 38.3% score across the 25 publicly available ARC-AGI-3 games (versus 13.3% using the original harness). They did this by editing the code of the harness to allow reasoning notes to persist between moves. They also implemented a technique called compaction, compressing game history rather than outright deleting it.
38.3% is not a self-reported ARC-AGI-3 record
It's worth noting that the publicly available version of ARC-AGI-3 is considerably stripped down, featuring just 25 of the 135 games in the official benchmark version administered by the ARC Prize Foundation.
As such, any results logged on this version of the test are not eligible for the foundation’s primary leaderboard. Custom harnesses, custom model settings and additional tools are also not permitted. The separate community leaderboard, where self-reported results are logged, has significantly lower standing. ARC itself warns that those results are not a reliable indicator of machine intelligence, as they do not verify the code and so human reasoning may play a part in results. Even with these caveats, OpenAI’s self-reported score is still perhaps less impressive than it may sound on paper.
Earlier this month, researchers at Impossible Research claimed to have achieved a score of 95.35% on the public game set using their custom harness with Sol. The same team logged a self-reported score of 99% with the same harness running Claude Opus 4.8 and Fable 5 in tandem.