International
The Morning Dev

Only the news worth knowing

‹
 

Research · High impact2 sources

Why Your Coding Agent Costs 5x More Than It Should (It Is Not the Model)

DEV · By Felipe Castillo ·

The Brief · written from 2 sources

  • Runta's FrontierHarness Eval v1.1 tested 22 coding agent harness configurations across 5 models, running 27,280 agent runs over 62 terminal tasks.
  • The benchmark totals 83 billion tokens and finds the same model can cost up to five times more depending on the harness it is paired with.
  • The benchmark was run by Runta and Baseten and covers open-weight models.

Why it matters

Choice of agent harness, not only the model, can drive running costs of coding agents substantially, so teams should measure cost per task when selecting tooling.

Who’s affected

  • Teams running coding agents on open-weight models
  • Developers choosing a coding agent harness

Runta published FrontierHarness Eval v1.1 today: 27,280 agent runs, 83 billion tokens, 22 harness configurations, 5 models, 62 terminal tasks. It is the largest public comparison of coding agents I have seen, and it is free to read. I read it expecting one answer and found…

Timeline

  1. Runta published FrontierHarness Eval v1.1 results

  2. The New Stack

    Your AI coding agent could cost 5x more depending on the harness, benchmark finds

  3. DEV Community: ai

    Why Your Coding Agent Costs 5x More Than It Should (It Is Not the Model)

Full coverage · 2 sources

How do you feel about this?

0 Comments