Research2 sources
The most misunderstood graph in AI just got a major update
MIT Technology Review · By Grace Huckins ·
The Brief · written from 2 sources
- METR updated its AI time horizon plot to include judgment-heavy tasks.
- Some AI capabilities continue to improve on an exponential trajectory.
- Models are weaker at judgment-heavy tasks than at coding, including designing AI evaluations.
Why it matters
The update suggests AI capability gains are uneven: coding-style tasks scale faster than judgment-heavy work, so developers should not assume benchmark gains on coding transfer to tasks requiring judgment.
Who’s affected
- Developers relying on AI time horizon benchmarks
- Teams building or running AI evaluations
- #ai
- #evaluation
- #metr
- #benchmarks
- #time-horizon
A new report from the AI evaluation nonprofit METR finds that some AI capabilities are continuing to improve along an exponential trajectory—but it also indicates that models aren’t quite as good at tasks that require judgment as they are at coding. The report is an update to the…
Timeline
Full coverage · 2 sources
- 45 min ago ↗
- 2h ago ↗
Artificial intelligence – MIT Technology Review
The most misunderstood graph in AI just got a major update