Why Do Most Coding Harnesses Not Matter?

Alex Zhang says the Harness Tax paper, which he thinks is by Arena, confirms a prior he already held: most harness choices do not matter because the harnesses are the same.

He thinks Grokbot is quite different, from his understanding, from how some other harnesses were designed, and he likes the difference. He is pretty sure Anthropic or OpenAI train exclusively on their own harnesses, and he assumes they do not train on a competitor harness, because he does not know why they would. Open models show the same requirement. Old Gemmas were notoriously bad at harness use, so a strong model still has to be trained in a harness. As models get smarter, the distinction becomes less important. A model that is already good enough will not go crazy inside a coding harness. The only benefit between different harnesses, for the most part, is cost.

Plug the same models into RLMs and they are still not that good. They are okay. Training has centered on one class of harness he calls trajectory as a prompt, which keeps the whole trajectory of the rollout as the context the main model uses.