← All stories

New METR evaluation finds AI models still lag on open-ended tasks

No. 25·179 words·1 min

The AI evaluation nonprofit METR has updated its time horizon plot, the graph that showed frontier models doubling the length of tasks they could complete every four months. The update adds tasks that take humans far longer, and tasks that are more open-ended.

On those, the models fall short. METR asked models to design new AI evaluations its own researchers could use; not one was good enough. Asked to score philosophical arguments, models again scored below human performance.

The newest models are inching up. Opus 5.5 now lands in what METR calls "borderline above the bar" on the evaluation-design task — not usable for the group's research, but a year ago every model scored "clearly below the bar."

The caveats are large. Completion times in the new report are mostly estimates, because paying people for tasks that take hundreds of hours is prohibitively expensive, and one person rated most of the philosophical arguments. Matt Putz, of METR's technical staff, said hard-to-verify tasks are "much harder to study, and especially to quantify," and that clean trends may not be possible.

Discuss on the board