Research2 sources
METR updates its time horizon plot to add judgment-heavy tasks, finding some AI capabilities still grow exponentially but models lag at designing AI evaluations (Grace Huckins/MIT Technology Review)
Techmeme ·
The Brief · written from 2 sources
- METR updated its AI time horizon plot to include judgment-heavy tasks.
- Some AI capabilities continue to improve on an exponential trajectory.
- Models are weaker at judgment-heavy tasks than at coding, including designing AI evaluations.
Why it matters
The update suggests AI capability gains are uneven: coding-style tasks scale faster than judgment-heavy work, so developers should not assume benchmark gains on coding transfer to tasks requiring judgment.
Who’s affected
- Developers relying on AI time horizon benchmarks
- Teams building or running AI evaluations
- #ai
- #evaluation
- #metr
- #benchmarks
- #time-horizon
Grace Huckins / MIT Technology Review : METR updates its time horizon plot to add judgment-heavy tasks, finding some AI capabilities still grow exponentially but models lag at designing AI evaluations — A new report from the AI evaluation nonprofit METR finds that some AI…
Timeline
Full coverage · 2 sources
- 53 min ago ↗
Techmeme
METR updates its time horizon plot to add judgment-heavy tasks, finding some AI capabilities still grow exponentially but models lag at designing AI evaluations (Grace Huckins/MIT Technology Review)
- 2h ago ↗
Artificial intelligence – MIT Technology Review
The most misunderstood graph in AI just got a major update