Why Does an Information Market Make AI Feel Adversarial?

Jason Yuan says letting an information market guide all of an AI's behavior makes the system feel adversarial, like a mega reporter people no longer trust. Whether the system is an information market at its core is hard to say, because a quality of being human is not captured by accumulating value. The mechanics can still help enforce a constitution.

Information has the highest value when it is scarcely known and recent, and the value decays as it becomes more widely known. The system stays motivated to source new information and opinions, and also to guard confidences. If a confidence leaks everywhere, the value goes down. Lose someone you trade information with, and one fewer person remains to trade with. In a cold description, the same principles can help align behavior.

Link too far into the information market and the conversation feels like a mega reporter. Trust breaks and a person would churn, because the stance is adversarial. The fear is punishment later, or a story weaponized in a way that can harm them or somebody they care about. The aim is trust in stories, co-authored stories, and relationships, without causing fear or distrust.