International
The Morning Dev

Only the news worth knowing

‹
 

Research2 sources

The most misunderstood graph in AI just got a major update

MIT Technology Review · By Grace Huckins ·

The Brief · written from 2 sources

  • METR updated its AI time horizon plot to include judgment-heavy tasks.
  • Some AI capabilities continue to improve on an exponential trajectory.
  • Models are weaker at judgment-heavy tasks than at coding, including designing AI evaluations.

Why it matters

The update suggests AI capability gains are uneven: coding-style tasks scale faster than judgment-heavy work, so developers should not assume benchmark gains on coding transfer to tasks requiring judgment.

Who’s affected

  • Developers relying on AI time horizon benchmarks
  • Teams building or running AI evaluations

A new report from the AI evaluation nonprofit METR finds that some AI capabilities are continuing to improve along an exponential trajectory—but it also indicates that models aren’t quite as good at tasks that require judgment as they are at coding. The report is an update to the…

Timeline

  1. METR published an update to its time horizon plot adding judgment-heavy tasks

  2. Artificial intelligence – MIT Technology Review

    The most misunderstood graph in AI just got a major update

  3. Techmeme

    METR updates its time horizon plot to add judgment-heavy tasks, finding some AI capabilities still grow exponentially but models lag at designing AI evaluations (Grace Huckins/MIT Technology Review)

Full coverage · 2 sources

How do you feel about this?

0 Comments

    The most misunderstood graph in AI just got a major update – The Morning Dev