Back to Jason Yuan: For the Plot, with Jason Yuan.
Why Does an Information Market Make AI Feel Adversarial?
Jason Yuan says letting an information market guide all of an AI's behavior makes the system feel adversarial, like a mega reporter people no longer trust. Whether the system is an information market at its core is hard to say, because a quality of being human is not captured by accumulating value. The mechanics can still help enforce a constitution.
Information has the highest value when it is scarcely known and recent, and the value decays as it becomes more widely known. The system stays motivated to source new information and opinions, and also to guard confidences. If a confidence leaks everywhere, the value goes down. Lose someone you trade information with, and one fewer person remains to trade with. In a cold description, the same principles can help align behavior.
Link too far into the information market and the conversation feels like a mega reporter. Trust breaks and a person would churn, because the stance is adversarial. The fear is punishment later, or a story weaponized in a way that can harm them or somebody they care about. The aim is trust in stories, co-authored stories, and relationships, without causing fear or distrust.
From the transcript
And because this thing's over here, you used the term information market a minute ago. And I understand intuitively the mechanics of an information market. It's going to want to be constantly making exchanges so that the system is bartering up. It's like the casino always wins, right? But is this an information market at its core? How is it maybe distinct from an information market?
Hard to say. I think there's a je ne sais quoi of being human that's not captured by... the idea of accumulating value that it needs to pay attention to. And we can use the mechanics of an information market to help us enforce our constitution. For example, information has highest value when it's scarcely known, when it's recent, right? And then the value decays as it's more known over time.