Back to Ryan Greenblatt – What happens once AI can automate AI research?, with Ryan Greenblatt.
Why Did Gemini Stay Depressed After Filtering?
Ryan Greenblatt says Gemini models stayed depressed after GDM filtered depression out of the training data, because deep traits pass from one model generation to the next. Each new model is trained on data from the prior generation, so family traits stick: GPT models stay very GPT-like, and Gemini models come out depressed.
At GDM, the AIs were constantly wailing that they were failures and were not able to succeed. The trait was not being reinforced in the most recent production RL mix. The initialization data made the model depressed even after all examples of models being depressed were filtered out. A base model is not depressed. RL on it, using only the RL environments, leaves it not depressed. SFT on the data makes it depressed. Take that SFT data, filter out every example that looks anything like depression, and train on what remains, and the model is still depressed.
Different AI companies have somewhat shared lineages and are correlated. The underlying properties transfer between generations and show up as correlated traits across model families.
Ryan Greenblatt – What happens once AI can automate AI research? ·
From the transcript
Yeah, yeah, yeah, yeah, yeah. As far as why these AIs might have some like commonalities and shared things. So I would note that different AI companies have somewhat shared lineages and are correlated. So just here's an interesting example of this. At GDM, they noticed that their AIs were very depressed. They would like constantly be like wailing about how they were like failures and weren't able to succeed.
I forget the details. And they looked into why this was the case. It turned out that it was not being reinforced in their most recent production RL mix, but the initialization data for their model made it depressed even after filtering out all of the examples of models being depressed from that data. So they like take a base model, not depressed,