How Does Legora Test a New AI Model on Legal Work?

Jacob Lauritzen says Legora scores a new model on a heatmap of practice areas and tasks, because a release often gets better at some legal work and worse at the rest.

An evals team of three engineers and three lawyers builds those checks. They also pull in legal engineers, lawyers who spent three or four years in big law and often have a computer science degree. Lauritzen calls that mix the hardest hire and a force multiplier: a pool of 100 people who use the product, talk to clients, and can score the work. The tests are built like small matters. One is a 10,000-file data room for a due diligence. A good result names the red flags, leaves out the noise, and stays concise. The team splits that into whether the draft is sound and whether the red flags were found in the first place.

For a long stretch the team did less of this. Each new model just got better, so a better harness was the faster win. One citation failure shows the other pattern. The model would not cite in a way a lawyer could follow. The next release moved that from 70 percent to 99 percent, and Legora changed nothing.