Smash Notes

Back to Academia is for Ambition — Alex Zhang, MIT, with Alex Zhang & Vibhu Sapra.

Why Does Matrix Multiplication Make a Speed of Light Easy?

Alex Zhang says matrix multiplication makes it very easy to compute a speed of light estimate of the fastest GPU kernel, while the same kind of estimate is not easy once the problem is more complex.

The estimate moves with where the data lives at the start. All the data might start on the CPU, or it might start in DRAM on the GPU, and the transfers change the numbers slightly. The estimate assumes perfect overlapping and transfer of data. In a lot of cases it is not clear whether the theoretical number can even be hit, because some bottleneck may be impossible to get around. Kernels people write are often not even close. They are not nearly close enough to the speed of light number to be meaningful at all.

Grokipedia·Alex Zhang

Transcript

Academia is for Ambition — Alex Zhang, MIT ·

Latent Space: The AI Engineer Podcast

More from this episode

  1. Why Do AI-written Kernels Fail in Real Systems?Alex Zhang says almost all recent solutions on the GPU Mode leaderboard are generated by AI, including work from people with no background in kernel writing.
  2. Why Can One Expert Erase a Trillion Tokens of Search?Alex Zhang says one person who knows a problem can erase a huge token search.
  3. Why Is a Puzzled Reaction a Good Sign for a Paper?Alex Zhang says a puzzled reaction is a good sign for a paper, because it means people are not thinking about what the work is for.
  4. What Is Academia for If Industry Has the Resources?Alex Zhang says academia is for big bets that an industry lab will not take. Doing that work in academia is hard.
  5. What Does the Language Model Game Benchmark Test?Alex Zhang's benchmark on language models playing video games asks whether vision language models are good enough at plugging into games with the latency…
  6. What Is a Language Model Harness?Alex Zhang treats a harness as the choices that let a language model solve a task, and raises whether a language model can just do the task on its own.
  7. Why Do Most Coding Harnesses Not Matter?Alex Zhang says the Harness Tax paper, which he thinks is by Arena, confirms a prior he already held: most harness choices do not matter because the harnesses…
  8. What Is a Recursive Language Model?Alex Zhang says a recursive language model, an RLM, is a harness whose only tool is code. He wishes that definition had been in the original paper.
  9. How Does Prime Agent Limit Tools to IPython?Alex Zhang describes Prime Agent as a harness on top of Pi Mono, the minimalist harness he uses as the reference for other coding harnesses.
  10. How Public Is OpenAI's Agent Spend?Shawn Wang says the amount OpenAI has spent on agents is semi-public.
  11. Why Is Sakana Different From OpenAI?Alex Zhang says Sakana is different from OpenAI because it is a research lab, closer to a PhD group than a company with a model everyone uses.
  12. Is the AI Capability Overhang Real?Alex Zhang agrees, in some sense, that the capability overhang is real.
  13. What Makes a Research Cold Email Fail?Alex Zhang answers a cold email when the writer shows up with a strong opinion, and he treats “I really like RLMs, I wanna work together” as a bad reach-out.
  14. How Do You Pick a Research Idea Worth Chasing?Alex Zhang chooses which research idea to chase from about 10 or 15 different ideas he wants to do, and most of them are bad.

People

More answers like this

More with Shawn Wang

Related episodes

Plays from this note through the rest of the episode.

Note

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.