MaskNinja

AI benchmarks still rely on the same core ideas. Models have changed dramatically over the past few years, but the benchmarks we evaluate them on haven’t changed nearly as much. Is there anything we can do to sweeten the mix? We still see these ideas in most benchmarks today. But modern agents are more than just that. Rather than just implement smaller tasks, we’ve gotten them to implement features bigger in scope. And we need benchmarks to reflect that. They’re still asking agents to implement PRs, and a big reason for that is because it’s the accepted standard. SWE-Bench is a big victim of this. I’ve already talked about how benches get contaminated into training data, but your prompting style also matters. SWE-Bench was telling models  not  to make their own tests or to just implement things in a specific manner, even if it wasn’t the most efficient way to do that task. DeepSWE and FrontierCode are better, but they’re still focusing on PRs, which are what we should be quest...

Most benchmarks are either a pass or a fail on smaller tasks. What if one wasn't? Recently, I’ve been poking around at video codecs. Specifically, I’ve been trying to reverse-engineer RAD Game Tools’ Bink 2 using GPT-5.5, adding an IDA plugin to the mix. Sin ce Bink 2 has no public encoder and only a decoder, I was interested to see how far a SOTA model would get. The total amount spent on this was around $70,000. It seems like a lot, but, the model ended up being quite stubborn instead of implementing things which I should have gotten it to do long ago. Combined with the fact that I'm far too soft on them. This is roughly what got me thinking about benchmarks. There’s a clear difference between Bink 2 and tests such as SWE-Bench. Bink 2 is quite novel and in general, there is little on the public internet in terms of just technical specs, despite it being over a decade old now. In contrast, I would also consider it well-known that by now, SWE-Bench has ended up in training dat...