Why Do AI-written Kernels Fail in Real Systems?

Alex Zhang says almost all recent solutions on the GPU Mode leaderboard are generated by AI, including work from people with no background in kernel writing. The only kernel in the top 10 that stayed stable in an actual end-to-end system came from a regular GPU Mode member who steered the model.

On a lot of the recent GPU Mode problems, almost all the solutions are AI generated. A regular member, known for a long time as a very strong GPU kernel writer, used AI to help with the solutions. For the most part he prompted the model and moved it in certain directions. The kernel was basically the only one in the top 10 that stayed stable in actual end-to-end systems. GPU kernels have a verification problem, known since kernel bench was released, and a lot of reward hacking goes on. The lines of code are a lot smaller, and the difference is important.

There is still a lot of alpha in being good at bringing GPU kernels. A top leaderboard score and a kernel that stays stable in an end-to-end system are different results.