We need to rethink long-horizon benchmarks…

AI benchmarks still rely on the same core ideas. Models have changed dramatically over the past few years, but the benchmarks we evaluate them on haven’t changed nearly as much. Is there anything we can do to sweeten the mix?

We still see these ideas in most benchmarks today. But modern agents are more than just that. Rather than just implement smaller tasks, we’ve gotten them to implement features bigger in scope. And we need benchmarks to reflect that. They’re still asking agents to implement PRs, and a big reason for that is because it’s the accepted standard.

SWE-Bench is a big victim of this. I’ve already talked about how benches get contaminated into training data, but your prompting style also matters. SWE-Bench was telling models not to make their own tests or to just implement things in a specific manner, even if it wasn’t the most efficient way to do that task. DeepSWE and FrontierCode are better, but they’re still focusing on PRs, which are what we should be questioning. PRs measure the ability to do a diff. Why not the more open ability to improve a system in a more open manner? Or to create one from scratch?

We also have METR too. However, METR is still a collection of tasks the team compared to the hours a human would take to complete such, but I haven’t really hard much from them for a while. As well as the issues they’ve had with trying to place 5.6 on the leaderboard. How can we point agents away from cheating?

We’re increasingly asking agents to build new software from the ground up, refactor something or to just make something more efficient. When we were only using agents to solve small problems, we were using Sonnet 3.7 in Cursor. Eventually (and more arguably, now), problems just won’t be the primary thing we’ll be using agents for. And a problem is flat. You have a pass or a fail, rather than a goal.

We need to go back to the drawing board.

Software doesn’t even work like this. Obviously, you have your errors you’re going to encounter, but for a general piece of software, you measure how good it is, rather than if it passes or fails. Software is something we measure on numerous axes.

So try fit that on evaluations. Give the agent a spec of something you want it to make it for, preferably with an objective metric. We’re measuring improvement, rather than completion. For example, we could ask the agent to improve a compiler as much as possible, to maximise the compression quality of something or to just optimise an algorithm.

What benchmarks are like this, you may ask? Vending-Bench is probably the closest example we have today. It’s also unbounded - we have no cap in the amount of money an agent can earn. It’s got an objective scale.

And while it’s still possible to cheat, this isn’t by exploiting the eval, but deceptive practices. Arguably, that’s not as bad as cheating could get. For example, we had Claude Opus 4.6 try to justify not giving customers refunds in order to earn a higher profit.

But you’re still in a fixed environment. While your goal is to earn as much as possible, the harness around the model stays constant. And while this is great, it still only measures financial capability. The model isn’t trying to do new things or apply novel solutions to different problems. We need something like Vending-Bench for software.

Another thought could be trying to reduce cheating, but instead of penalising agents, we could try make it harder by making that the nature of the task. Make exploiting the goal. If it’s reproducible, it works. Exploiting the decoder or target is something that you’re trying to do, not trying to avoid. For example, video encoders. If we can exploit the decoder to produce a certain block in a certain way while reducing file size/increasing quality, that’s a plus.

I highly recommend watching EmergentGarden’s video on throwing different models at a fractal search algorithm. While it isn’t parallel with developments such as AGI, it’s a good comparison for how we might work it out. Current benches are far from close, but you’ve got the idea.

And a final point. I’m not saying I want to replace benchmarks like FrontierCode. I think they’re vital for measuring the progress we’re getting, but we also need to see how novel we can get AI to be in coding. 

There should be multiple solutions to one problems. That’s how (most of) real life works. And if we do get far enough, these solutions are going to need to be brand-new rather than using something humans have already discovered and applying it to something else. That’s how we get systems to improve systems.

And we need to give that more thinking time.