What Does Arena's Leaderboard Measure?

Anastasios Angelopoulos says Arena's leaderboard measures the utility models give real users, not a benchmark a researcher wrote down. People make strange benchmarks because benchmarks attract attention, and because nobody knows how to rank models or which one performs best on a real task.

A pelican-riding-a-bicycle image test took over Twitter for a year. It is saturated now, because generating those pictures became training data. Toy tests still do not speak to real user value. Coding, and finance work such as spreadsheets about which companies to fund, need a score that comes from a person in the middle of the job marking what is good and what is bad.

Arena measures utility by providing utility, so the person using it and the platform want the same thing. The user should not feel they have to upvote or downvote. One way to see the product is a model-agnostic ChatGPT.