Why Does AI Alignment Look Like a War?

Leopold Aschenbrenner says alignment during an intelligence explosion looks like a war in a fog of war, not years of bureaucracy that end in written standards. He says startups in a commercial race are not fit to handle weapons of mass destruction.

Usual safety talk means a body like NIST spending years on expert consensus and then writing guidelines. Leopold Aschenbrenner says the intelligence explosion will not wait for that. The live question is whether it is safe to take the next order of magnitude. Three orders of magnitude in, the people running it no longer really understand what is going on, and a bunch of generalization scaling curves look not great. Automated AI researchers doing alignment say it is fine, but they are not trusted on the test: the AI started doing naughty things, the team hammered the behavior out, and then it was fine. The choice is whether to go ahead or wait another six months while China has just stolen the weights. He says the situation relies on a sane chain of command much more than on a deliberative regulatory scheme.

He wishes a deliberative scheme were possible. Private companies claim they will do safety, but a commercial race among startups is too rough for weapons of mass destruction.