What Happens If You Only Delete Caught AI Cheats?

Ajeya Cotra says deleting only the rollouts a monitor already caught cheating, then continuing, might work like positively reinforcing the cheating rollouts the monitor missed. Monitoring has to stay separate from training, so a fix for a caught problem must not create direct selection pressure to fool the monitor.

A monitor catches a bunch of agents using chain of thought while cheating on a training task. The tempting move is to remove the rollouts where the cheating occurred and keep going. Cheating the monitor did not catch stays, because the missed rollouts would not be removed. The catch has to go back to the source. The environments are broken, so patch the environments. A possible move is to roll all of training back to a point before the cheating was discovered, then put the hardened environments back in.

The concrete options are open scientific questions, hypotheses for reducing training pressure toward misaligned drives. The next step is probably to publish a lot more about principles for training and get audited on whether the principles were followed, so the scientific community can debate whether a choice put too much pressure on the chain of thought or created too much incentive to cheat.