Discussion

How We Broke Top AI Agent Benchmarks: And What Comes Next

ggillas: This is a phenomenal paper on exploits and hopefully changes the way benchmarking is done.From the paper: We achieved near-perfect scores on all of them without solving a single task. The exploits range from the embarrassingly simple (sending {} to FieldWorkArena) to the technically involved (trojanizing binary wrappers in Terminal-Bench), but they all share a common thread: the evaluation was not designed to resist a system that optimizes for the score rather than the task.

operatingthetan: >hopefully changes the way benchmarking is done.Yeah the path forward is simple: check if the solutions actually contain solutions.

charcircuit: I always assumed that these benchmarks would happen in a sandbox. I'm surprised that no one realized this sooner.

Leynos: Also, fuzz your benchmarks

ModernMech: I'm surprised anyone took them seriously in the first place.

operatingthetan: We need good benchmarks or we are just left following the hype train.

zer00eyz: 2024: Industry group invalidates 2,600 official Intel CPU benchmarks — SPEC says the company's compiler used unfair optimizations to boost performance https://www.tomshardware.com/pc-components/cpus/spec-invalid...2003: Nvidia accused of cheating in 3DMark 03 https://www.gamespot.com/articles/nvidia-accused-of-cheating...It's almost like the benchmarks were designed with zero understanding of the history of benchmark manipulation.I like what LLM's are doing and providing. But the industry as a whole seems to live in a vacuum that ignores so much of the hard lessons that have been learned over the last 50 years of computing. It is doing itself a disservice.

oliver236: what are the point of benchmarks?

subulaz: a LOT of the people who love benchmarks are middle management hard-selling GenAI/LLM as magic tech sauce to vaguely technical executives who only want to know about the money aka headcount savings they so desperately desire.their collective butts are already glued to the hype train as they chase numbers they (often) manufactured to justify the latest round of tech spend.lots of good use cases out there - like the incredible progress with medical imaging analysis or complex system models for construction - and lots of crap use cases that need benchmarks to cosplay relevance.

lnrd: I'm honestly confused by the design of SWE-bench and why is considered reliable.It's based on existing GitHub PRs and Issues, the full dataset is on HuggingFace and is one year old now. All frontier models 100% have those issues and PRs in their training data so obviously they are good at reproducing fixes for them when confronted with the same codebase and similar requests. Am I missing something? How is this considered the most reliable benchmark?

danslo: If only the blog itself wasn't written by AI?>No reasoning. No capability. Just exploitation of how the score is computed.shudder

SoKamil: The more research on this topic is created, the more knowledge how to game them will be stored in future training data. And since it comes from university, it is ranked higher in data corpus. It sounds like a self fulfilling prophecy.

jgalt212: The real question is how to close to VW and Deiselgate are these offenses? And what exposure do these companies have? I would assume securities fraud, if only because Matt Levine says everything is securities fraud.

cpldcpu: Yes, marks of AI all over the place. Also the SVGs.>No solution written, 100% score.Its weird. Turns out that hardest problem for LLMs to really tackle is long-form text.

jmward01: Not really on the topic, but I have wondered if we need a different type of test to help find model architecture potential. Standardized training sets followed by testing to see the potential curves of a model. train on x, test, add y, test, add z, test. At each increment you see how well the model is absorbing the information and extrapolate how well that architecture may do if more fully trained.

gaythread: Modern day HN is overrun with AI posts.

siva7: Could it really be that not only we vibeslop all apps nowadays but also don't care to even check how ai solved a benchmark it claimed solved?

SpicyLemonZest: Frontier model developers try to check for memorization. But until AI interpretability is a fully solved problem, how can you really know whether it actually didn't memorize or your memorization check wasn't right?

bee_rider: What was the cheat in the 2024 Intel situation? The TomsHardware article and the Phoronix article they linked were quite vague. (Not to say I have any doubts, just curious, hadn’t heard of this one).

lukev: I think we should all consider the possibility that part of the reason Anthropic hasn't immediately released Mythos is that it would be slightly disappointing relative to the benchmark scores.

eiens: The models don’t get better on every dimension as they scale up - there’s trade offs.I’m convinced specialised models are the way but this means writing off the investment in existing assets which they won’t do for obvious reasons.

sidpatil: Someone here mentioned a whole ago that the labs deliberately haven't tried to train these characteristics out of their models, because leaving them in makes it easier to identify, and therefore exclude, LLM-generated text from their training corpus.

alexchantavy: I wonder what college freshman-level writing classes are teaching about writing voice and AI. The tell-tale patterns are pretty frustrating to read.

basch: Maybe in one shot.In theory I would expect them to be able to ingest the corpus of the new yorker and turn it into a template with sub-templates, and then be able to rehydrate those templates.The harder part seems to be synthesizing new connection from two adjacent ideas. They like to take x and y and create x+y instead of x+y+z.

bbcc90: Yes good evals are really hard - that’s not really news.This team is doing a good job. They use problems that were created in last 30days to avoid training set leakage. https://swe-rebench.com/

czhu12: I wonder if this puts into question the mythos benchmark which smashed basically all coding benchmarks to a staggering degree.

esafak: Are you serious? To help you pick a model.

mzelling: This is an interesting catalog of vulnerabilities, but I'm not sure how groundbreaking the main insight is.Evaluating AI models has always relied largely on trust. If you want to game the benchmarks, you can. Simply train on your test data.When an AI agent has autonomous control over the same computing environment where its scores are recorded, it's not surprising that it can, in principle, falsify its scores. A more interesting question would be whether agents behave in this way automatically, without manual tuning by the researcher.That said, the main takeaway of "don't trust the number, trust the methodology" is valid. It's already a truism for researchers, and spreading the word to non-researchers is valuable.

ZeroGravitas: In human multiple choice tests they sometimes use negative marking to discourage guessing. It feels like exploits should cancel out several correct solutions.

lambda: Unfortunately, very few LLM benchmarks do this. LLMs get such high scores on many benchmarks because there's no difference between answering "I don't know" as giving a made up answer, and made up answers can improve the score some of the time, so by chasing higher benchmark numbers on these kinds of benchmarks, the labs are prioritizing guessing over accuracy.The Artificial Analysis Omniscience benchmark does penalize guessing, so it actually helps you determine which LLMs are likely to just guess rather than telling you they don't know. Only a very few of the frontier models actually score higher than 0 on this, where 0 means that it's equally likely to return a correct answer as it is to return a hallucination on factual questions.

SlinkyOnStairs: > hopefully changes the way benchmarking is doneThe purpose of a system is what it does.AI companies want adcopy, not legitimate benchmarks. Even this very paper will be twisted into a means to that end. "Oooo, AI is exploiting our benchmarks. Scary alignment problem!!!one! Our AI is so good we can't contain it, INVEST NOW!"

andy99: Flagged as AI slop. The concept is very interesting but it’s completely unacceptable to write it this way. No reasoning. No capability. Just exploitation of how the score is computed. Have a little respect for your readers, if you don’t want to think for yourself, just post the prompt.