Why Does Mid-training Do Most of the Work in Reinforcement Learning?

Beren Millidge says reinforcement learning has worked better than a naive bit count predicts because strong mid-training often takes a model almost 80% of the way to the final checkpoint, and the later stage only tweaks the policy with a few high-signal bits.

Mid-training is pre-training on synthetic reasoning data and on environments that warm-start the model. The policy does not have to learn the behaviors from scratch. Those bits carry much higher signal than ordinary pre-training, which is why reinforcement learning beats simply copying successful reasoning traces. In supervised fine-tuning the answer token is still in the trace, but the objective also has to match every reasoning token, which piles on bits about one exact style. Reinforcement learning keeps only the one bit about getting the answer right. The useful bit is not drowned, the signal-to-noise ratio jumps, and training takes far fewer steps.

Charlie O'Neill treats the worry that rare correct traces get down-weighted by easier ones as a group-size problem. With enough compute to sample a large enough group, several correct answers show up and the correct traces get up-weighted. He says pass@1, where reinforcement learning starts, scales with the log of the number of pre-training tokens.