A brand new, difficult AGI take a look at stumps most AI fashions | TechCrunch


The Arc Prize Basis, a nonprofit co-founded by distinguished AI researcher François Chollet, introduced in a blog post on Monday that it has created a brand new, difficult take a look at to measure the overall intelligence of main AI fashions.

To date, the brand new take a look at, referred to as ARC-AGI-2, has stumped most fashions.

“Reasoning” AI fashions like OpenAI’s o1-pro and DeepSeek’s R1 rating between 1% and 1.3% on ARC-AGI-2, based on the Arc Prize leaderboard. Highly effective non-reasoning fashions together with GPT-4.5, Claude 3.7 Sonnet, and Gemini 2.0 Flash rating round 1%.

The ARC-AGI exams encompass puzzle-like issues the place an AI has to establish visible patterns from a set of different-colored squares, and generate the right “reply” grid. The issues had been designed to pressure an AI to adapt to new issues it hasn’t seen earlier than.

The Arc Prize Basis had over 400 individuals take ARC-AGI-2 to determine a human baseline. On common, “panels” of those individuals obtained 60% of the take a look at’s questions proper — a lot better than any of the fashions’ scores.

a pattern query from Arc-AGI-2 (credit score: Arc Prize).

In a post on X, Chollet claimed ARC-AGI-2 is a greater measure of an AI mannequin’s precise intelligence than the primary iteration of the take a look at, ARC-AGI-1. The Arc Prize Basis’s exams are aimed toward evaluating whether or not an AI system can effectively purchase new expertise outdoors the information it was skilled on.

Chollet stated that in contrast to ARC-AGI-1, the brand new take a look at prevents AI fashions from counting on “brute pressure” — intensive computing energy — to seek out options. Chollet beforehand acknowledged this was a significant flaw of ARC-AGI-1.

To deal with the primary take a look at’s flaws, ARC-AGI-2 introduces a brand new metric: effectivity. It additionally requires fashions to interpret patterns on the fly as a substitute of counting on memorization.

“Intelligence isn’t solely outlined by the power to resolve issues or obtain excessive scores,” Arc Prize Basis co-founder Greg Kamradt wrote in a blog post. “The effectivity with which these capabilities are acquired and deployed is an important, defining part. The core query being requested is not only, ‘Can AI purchase [the] ability to resolve a activity?’ but in addition, ‘At what effectivity or price?’”

ARC-AGI-1 was unbeaten for roughly 5 years till December 2024, when OpenAI launched its superior reasoning mannequin, o3, which outperformed all different AI fashions and matched human efficiency on the analysis. Nonetheless, as we famous on the time, o3’s efficiency beneficial properties on ARC-AGI-1 got here with a hefty price ticket.

The model of OpenAI’s o3 mannequin — o3 (low) — that was first to achieve new heights on ARC-AGI-1, scoring 75.7% on the take a look at, obtained a measly 4% on ARC-AGI-2 utilizing $200 value of computing energy per activity.

Comparability of Frontier AI mannequin efficiency on ARC-AGI-1 and ARC-AGI-2 (credit score: Arc Prize).

The arrival of ARC-AGI-2 comes as many within the tech business are calling for brand spanking new, unsaturated benchmarks to measure AI progress. Hugging Face’s co-founder, Thomas Wolf, not too long ago instructed TechCrunch that the AI business lacks ample exams to measure the important thing traits of so-called synthetic basic intelligence, together with creativity.

Alongside the brand new benchmark, the Arc Prize Basis introduced a new Arc Prize 2025 contest, difficult builders to achieve 85% accuracy on the ARC-AGI-2 take a look at whereas solely spending $0.42 per activity.

Leave a Reply

Your email address will not be published. Required fields are marked *