Research · High impact2 sources
Your AI coding agent could cost 5x more depending on the harness, benchmark finds
The New Stack · By Amanda Caswell ·
The Brief · written from 2 sources
- Runta's FrontierHarness Eval v1.1 tested 22 coding agent harness configurations across 5 models, running 27,280 agent runs over 62 terminal tasks.
- The benchmark totals 83 billion tokens and finds the same model can cost up to five times more depending on the harness it is paired with.
- The benchmark was run by Runta and Baseten and covers open-weight models.
Why it matters
Choice of agent harness, not only the model, can drive running costs of coding agents substantially, so teams should measure cost per task when selecting tooling.
Who’s affected
- Teams running coding agents on open-weight models
- Developers choosing a coding agent harness
- FrontierHarness Eval v1.1
- #coding-agents
- #benchmark
- #llm-costs
- #agent-harness
- #open-weight-models
The same AI model can cost up to five times more to run depending on the coding agent harness it’s paired with, according to a new benchmark that tested 22 configurations across five open-weight models. Runta and Baseten ran those combinations 27,280 times for FrontierHarness…