Why Doesn't Scaling One Part of an AI Model Work?

Alexandr Wang's broad rule for AI is to scale everything, and to scale it together. Scaling one thing in isolation does not work. Pushing a single dimension gets much less than scaling many things at once. Scaling laws only hold when the other factors move with them.

Muse Spark's point releases show the split. From version 1 to 1.1, 1.2, and 1.3, Meta did not change the pre-train. It kept doing more reinforcement learning on the same base. Early versions of a new class have a lower ratio of reinforcement learning to pre-training, and later versions raise that ratio. Muse Spark and Muse Wonder are different pre-trains. In reinforcement learning, less data is sometimes more, which is the opposite of pre-training.