Did Hermes Match Boltz on Binding Prediction?

Ian Quigley says Hermes did about as well as Boltz, then the state of the art structural model, on public data sets for binding prediction. The models were trained only on internal data and scored external molecules they had never seen. In his count the Hermes paper covered something like 13 million tests across about 500 protein targets.

Large collections of external molecules were predicted, with positives and negatives chosen carefully. Other labs usually screen fewer protein targets, so outside sources are tempting, and he has not seen another group train for the task only on internally generated data. Hermes cannot do a lot of what Boltz can do, and a fair comparison needed some tricks. Boltz had also trained on the public sets, though the subsets were not fully clear. Many evaluations were ones the Boltz team had already run, with far more molecules added. Hermes was better on some proteins and protein families, and Boltz was better on others. The suspicion is different distributions in the training sets, or memorizing a bit more and extending it.

One validating set was an orthogonal chemical library the models had not seen, used internally, and a copy of a library already in the literature. On the protein BRD4, an identical molecule from the screen was also in the Protein Databank training set Boltz had almost certainly seen.