Why Does Taste Beat Evals in Creative AI?

Jon Noronha says taste beats evals when the question is creative. An eval is a fixed question used like a software test, and it works when the answer is objective. One Gamma eval says that "make a deck about tigers" should make the agent ask whether it is for kids or for adults. Scores compare prompts and models. Whether something looks cool, or sounds like a particular person, has no single objective test.

Gamma still checks proxies, such as whether the output follows instructions or sticks to a brand guideline. Founders compete by bringing their own sense of what looks good and by listening to users. The less verifiable the field, the more room there is for a point of view. He thinks clothing could support many harnesses, with Nike and Adidas each keeping their own. Because Gamma's users differ, the team runs large A/B tests. After a person generates, they watch how long the editing goes on, whether the work is shared or exported, whether a free user upgrades to paid, and whether that person comes back a week or a month later. Paying is his favorite signal, because it is hard to fake.