“BinkBench”: can we compare LLM performance by seeing how well it can do one task?

Most benchmarks are either a pass or a fail on smaller tasks. What if one wasn't?

Recently, I’ve been poking around at video codecs. Specifically, I’ve been trying to reverse-engineer RAD Game Tools’ Bink 2 using GPT-5.5, adding an IDA plugin to the mix. Since Bink 2 has no public encoder and only a decoder, I was interested to see how far a SOTA model would get. The total amount spent on this was around $70,000. It seems like a lot, but, the model ended up being quite stubborn instead of implementing things which I should have gotten it to do long ago. Combined with the fact that I'm far too soft on them.

This is roughly what got me thinking about benchmarks. There’s a clear difference between Bink 2 and tests such as SWE-Bench. Bink 2 is quite novel and in general, there is little on the public internet in terms of just technical specs, despite it being over a decade old now. In contrast, I would also consider it well-known that by now, SWE-Bench has ended up in training data.

Even with possible solutions to that such as DeepSWE, I think they’re only buying time rather than fixing the underlying problem. The tasks might be original, but the repo and any writeups about it are sitting on GitHub right now, in plaintext, which is exactly where the next round of pretraining data gets scraped from. FrontierCode is more nuanced in that regard, since its tasks are not public.

I've been at Bink for a while, specifically an older revision (as distinct from the later versions, which have been ripped from Unreal Engine). There's effectively nothing about the older encoder’s heuristics on the public internet - no reference implementation or leak, and only a few places talking about the codec. Whatever a model knows about it, it must derive from starting at a decoder. And even for the Unreal rip, that’s just a compiled binary - LLM training data wouldn’t store the massive amount of binaries that are on the web right now.

I then realised that this codec itself could become a benchmark. Tasks like this I would say are closer to Andon Lab’s Vending-Bench 2. The agent isn’t solving many tasks, but trying to do one task as well as it can. With Bink 2, there’s video quality and compression efficiency. Why not compare an agent against these too? It seems there aren’t much long-horizon tasks for software engineering, but in the future, we might have a diverse mix of both long-horizon or traditional benchmarks. The dynamic nature of these benchmarks makes them especially enticing.

Things such as reverse-engineering are also resistant to getting put in training data. Sure, we have decoders and videos to test the agents on. But we’re meant to give the agent the decoder so it has the schema. The agents can generate their own test frames from a script and use this to compare their own encoder. They’re smart enough for that. Videos are interchangeable. We can give them frames from any fast-paced video online. As long as we control the harness for comparing the agent’s results and frames, then realistically,  this isn’t ending up in their training data.

But there’s no reason to stop there. We have lots of closed-source codebases. Bink 2 is the one I picked out because we have two clear-cut variables we can compare from. We also have no encoder references, which is a pro.

Of course, this brings us to an issue for such a bench: copyright. Unlike open-source software, closed-source software has a lot of rules. While I’d love to do exactly what Malus does for bigger, closed-source software, that’s a) very hard and b) legally expensive, since you might end up fighting off a lot of lawsuits. AI labs also don’t take kindly to reverse-engineering, specifically to prevent their models from infringing on copyright - even if there isn’t really an issue on the other side. Sadly, we’ll probably never end up with re5 (or re6) due to these reasons (and Rockstar).

I do think that’s the bigger issue here - even if Epic would never sue us, models like Fable would be at a much higher risk of refusing such a task. Surprisingly however, I found GPT-5.5 made a fully working (albeit inefficient) encoder, despite its approach being a little long-winded. A little ironic, considering OpenAI models, at least in my experience, have higher guardrails.

But we need to remember that dynamic benches don’t have to be reverse-engineering, even if they might be tougher.

I’ve recently come across this video. We don’t have to use software at all. For example, EmergentGarden used Fractal Search, just to show how companies achieving RSI might go about it. The models on that task even have the same flaw mine had: they just didn’t put enough architectural improvements. After a while, it just stopped trying anything new. However, I feel Opus models would probably have gotten better on this task, since they’re quite creative.

The sky is the limit (with some limitations). And we still have to be aware that no matter the benchmark, there will always be a chance it gets into training data. I don’t think that can happen with my bench idea as-is right now, but there is nothing stopping someone from publishing a valid encoder online at some point. Eventually, we’ll have to change the tasks, regardless of whatever they are. But that doesn’t mean we just have to just sit there.

Whatever shape or form these benchmarks take, I think long-horizon, contamination-resistant tasks are something we’re missing. A bench on Bink is one idea… but I feel it probably won’t be the last.