Smash Notes

Back to Muse Wonder: Zuckerberg and Wang's $100B AI, with Mark Zuckerberg & Alexandr Wang.

Why Does Muse Wonder Lead on Legal and Finance?

Alexandr Wang says Muse Wonder is the state of the art model on a lot of the finance evaluations and a lot of the legal evaluations, and the domain skill starts in pre-training, where the team looks for a lot of domain data to include.

Mark Zuckerberg named what people care about: shopping, personal advice, personal legal questions, personal health questions, and personal tax questions. Muse Wonder is very good on personal advice questions, and no score was given. The same skills can be reinforced to some degree, but pre-training still needs a large amount of data. Health has a lot of data out there, and pre-training is where the model is meant to see it.

The philosophy is that the best personal agent should feel like a team of experts in a person's corner: a legal team, a finance team, and a medical team. Pre-training took a huge percentage of the compute spent training Muse Wonder and was the highest-stakes part of the research. The run brought troughs of sadness, troughs of uncertainty, and all seven stages of grief. Pre-training is also the most fundamentally scientific part of the work because there are very few samples, so inferring what will happen depends on reasoning hard about the whole process. A tweet pictures pre-training researchers chewing glass and staring into the abyss, and the picture holds.

Grokipedia·Alexandr Wang·Mark Zuckerberg

Transcript

Muse Wonder: Zuckerberg and Wang's $100B AI ·

Cheeky Pint

More from this episode

  1. How Does Mark Zuckerberg Use Muse?Mark Zuckerberg uses Muse to coach his mixed martial arts.
  2. Why Do Winning AI Products Look So Simple?John Collison says winning AI products look simple because the field collapses quickly onto a few designs you could describe on the back of a postcard.
  3. What Is Muse Wonder?Alexandr Wang says Muse Wonder is a new class of model at the frontier, and the lab has been working toward it since the start.
  4. What Is Active Memory in an AI Model?Alexandr Wang says Muse Wonder is state of the art, by Meta's own measurement, on a slate of personal and professional agent tasks.
  5. How Big Should a Pre-training Team Be?Mark Zuckerberg says he does not know how big a pre-training team is, and it does not need to be that big. Other labs are probably quite big.
  6. Why Doesn't Scaling One Part of an AI Model Work?Alexandr Wang's broad rule for AI is to scale everything, and to scale it together. Scaling one thing in isolation does not work.
  7. How Is Meta Scaling AI Past Five Gigawatts?Mark Zuckerberg says you can measure a lab's compute strength by how big its largest single supercomputer is.
  8. Why Is Meta Spending $100 Billion Before the Model Exists?Alexandr Wang says frontier AI is the opposite of a lean startup. A lean startup gets something working and only then doubles down. You cannot do that here.
  9. How Much Does Training a Frontier AI Model Cost?Mark Zuckerberg puts the figure you would see in public, to operate a gigawatt cluster for a year, on the order of $15 billion.
  10. Why Is Meta's AI Spend Less Risky Than the Price Tag?Mark Zuckerberg says Meta's huge AI spend is not as risky as the size of the numbers suggests, because Meta has a backstop for how it can use compute.
  11. How Does Muse Tell Mark Zuckerberg From His Daughter?Mark Zuckerberg says Muse voice mode segments the voices so it knows which one is him and which one is not.
  12. Why Are AI Agents Different From Chatbots?Alexandr Wang says an agent is a big step change because the mental model is different from a chatbot that only shares information.
  13. How Does Meta Plan to Pay for Its AI Buildout?John Collison asks how the AI buildout gets paid for.
  14. Why Hasn't One AI Lab Run Away With the Lead?John Collison recalls the 2023 story that one AI lab would get ahead, start recursive self-improvement, and pull away from everyone else.
  15. Why Did Researchers Join Meta Superintelligence Labs?Alexandr Wang says a reason people were excited to join Meta and Meta Superintelligence Labs is that they wanted to work somewhere focused on giving this…
  16. Why Did Mark Zuckerberg Choose Alexandr Wang?Mark Zuckerberg says he and Alexandr Wang had talked about once every few months for years, and he was always impressed by how Wang thought about strategy.
  17. Why Did Mark Zuckerberg Reboot Meta's AI Lab?John Collison names four moves he thinks even an opinionated founder would not make. Mark Zuckerberg treated the recent prior AI effort as not good enough.
  18. Why Did a Meta Lead Reject a Reasoning Model?Alexandr Wang heard a key lead on Meta's existing effort reject a reasoning model because the lead did not think consumers needed one and said a normal model…

People

More answers like this

More with John Collison

More with Mark Zuckerberg

Related episodes

Plays from this note through the rest of the episode.

Note

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.