Why Is There No Good Eval for Finance AI?

Joseph Kim says there is no good eval today for financial work, and building one that actually measures the skill is much harder than it looks.

An eval is only a proxy for one dimension of the capability you care about. If the grading is wrong, the test is easy to reward-hack: a model can score high and still be poor at the job. OpenAI's GDP eval, a broader knowledge-work test, is completely saturated, and the parts that are not saturated are broken, so the team stopped reporting it and stopped tracking it. On GPT-6 Astra there was no good way to show progress on financial tasks. When that model was run on a set of financial evals, the numbers did not really change. A deep dive found the tests were already saturated, and the misses came from incorrect answer keys, poorly specified prompts, or some other flaw in the eval.

The team is trying to build an eval that predicts whether a strong score means you could automate a lot of a junior analyst's work in investment banking.