Legal AI benchmarks require more than just a single score
Legal AI rankings should evaluate the entire human-in-the-loop workflow, rather than solely measuring task accuracy. In addition to quality, they should account for turnaround time, total cost to final deliverable, and the level of required human review. These metrics demonstrate the true return on investment for law firms and their clients, indicating whether an AI agent delivers genuine incremental capacity rather than merely boasting impressive benchmark results.
Legal AI rankings should evaluate the entire human-AI workflow, not merely task accuracy. In addition to quality, they should also disclose turnaround time, cost to completion, and the required human verification effort. These metrics demonstrate the actual return on investment that law firms and their clients receive, and whether an AI agent creates genuine additional capacity rather than merely boasting impressive benchmark results.
The Problem with a Single Score
Legal AI benchmarks should measure more than just accuracy. A high score may appear impressive, but it reveals nothing about the system's operational overhead required to achieve that result. One AI agent might achieve an outstanding result while consuming millions of tokens. Another may score marginally lower, yet complete the same task significantly faster and more cost-effectively.
What Rankings Should Disclose
This distinction is critical in legal practice. The true utility of a system depends not only on the quality of its output, but equally on cost, speed, and reliability.
A meaningful ranking should therefore account for four factors:
- Quality: How effectively did the AI agent resolve the task?
- Time to Completion: How long did the entire workflow take?
- Cost to Completion: What costs were incurred to achieve a viable work product?
- Human Verification: How much time and attention were legal professionals required to expend to review and verify the result?
Token consumption can serve as a helpful supplementary metric. However, for operational efficiency, total time to completion, cost, and verification effort are considerably more informative. What matters is the complete workflow combining human expertise and AI, not merely the model's compute time.
Benchmarks such as Harvey's Legal Agent Benchmark (LAB) or LEXam can, for instance, provide valuable insights into a system's capabilities. However, their scores should not serve as the sole evaluative standard. Only a comprehensive assessment of quality, time to completion, cost, and human verification effort provides an accurate picture of an AI agent's practical utility.
This does not mean that identical requirements must apply to every task. A tool ideally suited for a brief, low-risk initial draft may be inappropriate for a complex legal analysis requiring extensive authority checking. Disclosing all four metrics brings these distinctions to light, rather than masking them behind a single composite score.
What Value Clients Actually Receive
These metrics also show what value clients are actually getting for their money. A system that consumes vast amounts of tokens may incur higher operating costs or limit the number of tasks users can complete within their subscription allowances.
A balanced evaluation of quality, speed, cost, and human verification effort therefore helps corporate legal departments and law firms assess not just whether an AI agent can execute a specific task, but whether it delivers sufficient practical value relative to its cost.
The Verification Paradox and Its Inherent Opportunity
There is also the well-known Verification Paradox: the more content an AI system generates for a legal professional, the more time may be required to review it.
Even when more advanced models reduce overt hallucinations, it does not necessarily follow that their outputs are easier to verify. On the contrary, linguistically compelling and plausible-sounding text can increase cognitive load, as remaining errors become more subtle and difficult to detect.
Legal practitioners must verify not only whether a cited precedent or legal authority actually exists, but also whether it has been correctly construed and whether a court actually made the ruling or statement attributed to it.
Automation therefore does not automatically result in a commensurate reduction in legal labor. A portion of the time saved in drafting may simply shift to verification. This review effort must be factored into any evaluation, rather than treating the AI agent's raw output as a fully completed task.
At the same time, quality improvements hold independent value. An AI system can enable work that was previously cost-prohibitive or practically unfeasible. These enhanced capabilities should be captured just as diligently as time and cost.
Furthermore, AI can allow a law firm to expand its service offerings, rather than merely expediting existing tasks. This expansion of capabilities is also part of the value proposition that a robust evaluation framework should reflect.
What Constitutes Genuinely Useful Legal AI for Us
From this perspective, we evaluate whether an AI system genuinely creates value for legal professionals. We consider not just the quality of its output, but the entire workflow: the quality achieved, the time required, the costs incurred, and the human verification effort necessary before a practitioner can rely on the work product.
We strive to implement this approach rigorously within our own company. This is not about marketing. A high benchmark score is commendable, but on its own, it does not generate client value.
The decisive question is whether a system creates genuine additional capacity in legal practice and delivers greater value to clients.
Therefore, we are convinced that future legal AI rankings must report four metrics with equal weight and transparency: quality, time to completion, cost, and human verification effort.