How Does Legora Test AI Agents Without Seeing Client Data?

Jacob Lauritzen says Legora tests its agents on stand-in matters and on coarse labels, because the company cannot read a trace of what the agent did for a lawyer.

The files belong to the law firm's clients, not to the firm, so an agreement with the firm still does not open the work. He calls that the frustrating kink in the business: the agent can do many things, and Legora does not know which ones people actually use. A consumer chat product can keep the full trajectory. Legora will not. Paraphrasing a real matter all the way down to a few buckets can be safe. Stripping the names and keeping the document is not safe enough, so the company picks the stricter rule.

What they can see is indirect. They watch whether people spend time in the product, whether they come back, and whether they renew. He says the return rate and the hours spent per month are extremely high for enterprise software. A cheap model scores whether a task looks solved, and a feature-flag rollout shows whether that score rose or collapsed. An evals team builds sets that approximate the work they think clients are doing. On live traffic they classify a run into the failure modes they already expect, which he puts at about 10 or 20, and into a task type and a practice area. That is a signal. It is not the document.