Joseph Kim
Related people
Episodes
Notes
Latent Space: The AI Engineer Podcast · ⚡ Building ChatGPT for Finance — Joseph Kim, OpenAI's Head of…
-
Why Is There No Good Eval for Finance AI?
Joseph Kim says there is no good eval today for financial work, and building one that actually measures the skill is much harder than it looks.
An eval is only a proxy for one dimension of the capability you care about. If the grading is wrong, the test is easy to reward-hack: a model can score high and still be poor at the job. OpenAI's GDP eval, a broader knowledge-work test, is completely saturated, and the parts that are not saturated are broken, so the team stopped reporting it and stopped tracking it. On GPT-6 Astra there was no good way to show progress on financial tasks. When that model was run on a set of financial evals, the numbers did not really change. A deep dive found the te
-
What Finance Work Can ChatGPT Actually Do Today?
Joseph Kim says the ChatGPT for financial services product OpenAI launched is a productivity tool for knowledge workers, and it assumes a human is always in the loop.
A financial analyst with a list of work for the day is meant to try ChatGPT first, the way a software engineer tries Codex. It might get the analyst 30% of the way there, or 80%, depending on the task. The person still reviews the work. That split made the product easier to explain, because the audit stays with the analyst.
The next version is meant to feel like working with a coworker rather than watching a tool. A company already knows how to check a person. At Google, a new software engineer needed readability before chang
-
How Is OpenAI Building ChatGPT for Finance?
Joseph Kim tells Richard MacManus that OpenAI built most of ChatGPT for financial services on the ChatGPT plugin architecture, and that 95% of the work is something a startup or another company could do without changing the model.
Codex gets a finance product about 80% of the way, and the missing 20% is what decides whether it wins. Launching Codex by itself as a finance product would fail. The team built that missing piece on the same plugin primitives any outside builder can use. A billion-dollar business can sit on a ChatGPT plugin by keeping that horizontal product and adding a secret sauce, which might be data, a UI change, or an integration.
The team inside OpenAI hits the same wall
-
What Finance Tasks Will OpenAI Take on in 2027?
Joseph Kim says OpenAI will start to move finance work up the stack in 2027, from the task a junior analyst can do, to some of the tasks an associate can do, to some of the tasks an MD would do. A lot of the low-hanging fruit is already solved. Higher on the chain, the objectives become much fuzzier, so training the model gets more difficult from a research perspective. The problem is not impossible.
The product has to match what the model can do today. It is quite good at concrete instructions: build a slide, make it look good, use a template. There is also a strong preference to make the model good at work humans are not good at or do not like. A lot of the people he talks to really do no
-
Why Does ChatGPT for Finance Bundle Earnings Transcripts?
Joseph Kim says ChatGPT for financial services bundles quarter earnings transcripts in-house because outside connectors failed too often. Some big names showed a 40% plus error rate, so the only workable fix at scale was to control the data end to end. The extra money paid on top to bundle the data is worth it.
The case looked weak at first. He had worked on the product for the last six months, and a connector already available in the store made bundling hard to defend. A customer might already hold a license and feel they were paying twice. Tests of the connectors showed the error rates were really high. Customers love the result, and the number one thing they call out is the data.
In ent