Smash Notes

Back to Academia is for Ambition — Alex Zhang, MIT, with Alex Zhang & Vibhu Sapra.

Why Can One Expert Erase a Trillion Tokens of Search?

Alex Zhang says one person who knows a problem can erase a huge token search. The people who can look at a problem and see how to solve it are also the people who can use AI on it, because they act as a strong verifier. Shawn Wang's example is diagramming the code and refusing to continue until every part is understood. Agent swarms, given enough compute, can explore a lot of solutions. The burn might be a hundred billion tokens, or a trillion. Someone who already understands the problem can uncover a step for the model that wipes out that spend.

What that trend will be is not clear. There are still many problems to solve, and the work cannot afford to throw the maximum compute at every one of them. Efficiency still matters.

Transcript

Academia is for Ambition — Alex Zhang, MIT ·

Latent Space: The AI Engineer Podcast

More from this episode

  1. Why Do AI-written Kernels Fail in Real Systems?Alex Zhang says almost all recent solutions on the GPU Mode leaderboard are generated by AI, including work from people with no background in kernel writing.
  2. Why Does Matrix Multiplication Make a Speed of Light Easy?Alex Zhang says matrix multiplication makes it very easy to compute a speed of light estimate of the fastest GPU kernel, while the same kind of estimate is…
  3. Why Is a Puzzled Reaction a Good Sign for a Paper?Alex Zhang says a puzzled reaction is a good sign for a paper, because it means people are not thinking about what the work is for.
  4. What Is Academia for If Industry Has the Resources?Alex Zhang says academia is for big bets that an industry lab will not take. Doing that work in academia is hard.
  5. What Does the Language Model Game Benchmark Test?Alex Zhang's benchmark on language models playing video games asks whether vision language models are good enough at plugging into games with the latency…
  6. What Is a Language Model Harness?Alex Zhang treats a harness as the choices that let a language model solve a task, and raises whether a language model can just do the task on its own.
  7. Why Do Most Coding Harnesses Not Matter?Alex Zhang says the Harness Tax paper, which he thinks is by Arena, confirms a prior he already held: most harness choices do not matter because the harnesses…
  8. What Is a Recursive Language Model?Alex Zhang says a recursive language model, an RLM, is a harness whose only tool is code. He wishes that definition had been in the original paper.
  9. How Does Prime Agent Limit Tools to IPython?Alex Zhang describes Prime Agent as a harness on top of Pi Mono, the minimalist harness he uses as the reference for other coding harnesses.
  10. How Public Is OpenAI's Agent Spend?Shawn Wang says the amount OpenAI has spent on agents is semi-public.
  11. Why Is Sakana Different From OpenAI?Alex Zhang says Sakana is different from OpenAI because it is a research lab, closer to a PhD group than a company with a model everyone uses.
  12. Is the AI Capability Overhang Real?Alex Zhang agrees, in some sense, that the capability overhang is real.
  13. What Makes a Research Cold Email Fail?Alex Zhang answers a cold email when the writer shows up with a strong opinion, and he treats “I really like RLMs, I wanna work together” as a bad reach-out.
  14. How Do You Pick a Research Idea Worth Chasing?Alex Zhang chooses which research idea to chase from about 10 or 15 different ideas he wants to do, and most of them are bad.

People

More answers like this

More with Shawn Wang

Related episodes

Plays from this note through the rest of the episode.

Note

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.