Why Did Later AlphaFold Models Learn Rotation Symmetry From Data?

Fabian Theis says later AlphaFold models found rotation symmetry in the data instead of building that rule into training. The first models went to extraordinary effort to make the molecules rotation invariant inside the learning process. Later versions let the model find the symmetry. A physical constraint put in by hand is not always rigid, so the later models were a bit more flexible, and letting the model learn from the data gained something.

He sets the AlphaFold story next to a longer result. Building a priori inductive biases in too strongly might cost predictive power. In cell modeling the number of cells is large, but the number of perturbations, the things done to a cell, is not, so he does not think the work is in a super data-rich regime yet. Hybrid models might still be the ones to use. The honest lesson he takes from large-scale machine learning is that at some point data beats human-made concepts added to a model. He expects the models people actually use to be the flexible, data-based ones, with the interpretation added afterward.