How Big Should a Pre-training Team Be?

Mark Zuckerberg says he does not know how big a pre-training team is, and it does not need to be that big. Other labs are probably quite big. The models are doing well because the people Meta is betting on have a lot of resources. Compute stays with the key people instead of being spread across a lot of competing efforts, with a lot of compute per researcher so they can do the work.

Wonder was the next pre-train and climbed a ladder through a tiering system, and then came Spark. Avocado was the internal codename for the Muse Spark line, and Watermelon was the codename after it. The fruit names are used up. Durian is the only fruit that is technically larger, and it is heavier and spikier.

Alexandr Wang puts each stage at tens of people working on it at Meta versus hundreds. Meta is a huge company, and the research effort is really tiny. The research team is less than 200 people, very small, and intentionally very tight.