🔍 Read the full analysis: Why Z.ai’s Legal AI’s Near-Top Ranking Needs Benchmark Context on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai says GLM-5.3-FlashX-Legal achieved a 19.2% all-criteria pass rate on Harvey’s Legal Agent Benchmark. Vals AI’s archived leaderboard records 19.17%, seventh among 81 models; the score is not a measure of partial correctness, and Z.ai’s “open-source” comparison claim lacks a stated comparison set.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh of 81 models on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, with a recorded score of 19.17%. The result is a task-level, all-or-nothing measure: a task passes only when every required criterion passes, so it does not mean the model gets 19% of legal work right or is generally reliable for legal practice.
Z.ai describes GLM-5.3-FlashX-Legal as a model for legal research, contract review and drafting, built on GLM-5.3-Flash. Its October 13 beta announcement says it was trained further for legal workflows and reports a 19.2% all-criteria pass rate on Harvey’s benchmark. Z.ai also claims this is a 12.5 percentage-point improvement over the base model and “state-of-the-art performance among open-source models in this evaluation.”
Vals AI’s archived Version 1 leaderboard lists the legal model at 19.17%, seventh among 81 entries, at a listed cost of $0.15 per test. The same table puts the base GLM-5.3-Flash at 6.67%; the difference is 12.50 percentage points, consistent with Z.ai’s improvement claim. Vals reports a separate 92.5% criteria pass rate for the legal model. The two figures describe different levels of the scoring system and should not be treated as interchangeable.
Harvey’s benchmark contains 1,250 tasks across 24 practice areas, with more than 75,000 expert-written rubric criteria. Vals says its leaderboard uses a private holdout of 120 tasks, with internet access disabled, and averages task pass rates from two AI judges. In this framework, one missed criterion can make an otherwise substantially correct work product fail the entire task. The leaderboard therefore measures end-to-end completion under a strict rubric, not how much partial credit a system might earn in routine use.
What the Score Means for Legal Teams
The result matters because law firms and legal departments need to know whether an AI system can produce work that is complete enough for review, not simply whether it can generate plausible-looking text. Harvey’s all-pass rule reflects that distinction: a report that misses a deadline, material risk or required citation may be unusable even if many other points are right. A 19.17% task pass rate signals a notable benchmark result, but also that most evaluated tasks did not clear every rubric requirement.
At the same time, the score alone does not establish how useful the model is with professional supervision. Vals lists its criteria pass rate at 92.5%, suggesting many individual requirements were met even when a complete task failed. That does not make the output safe to use without checking: partial success can conceal a critical omission or error. Firms considering the model will need to test their own workflows, confirm citations and verify all material details before relying on generated drafts.
Cost is another point of interest, not a verdict on value. Vals lists $0.15 per test, below the listed costs for several higher-ranked entries. That is a benchmark-specific figure, not a full estimate of deployment expenses, which can include human review, data controls and connected services. Z.ai’s pricing page separately lists token rates of $0.37 per million input tokens and $1.25 per million output tokens.
Products Worth Considering
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
Bose SoundLink Plus Portable Bluetooth Speaker, Wireless Outdoor Speaker, Up to 20 Hours Battery Life, Waterproof and Dustproof, Black
Samsung 27-inch Odyssey G5 QHD Gaming Monitor (G51F)
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Agent Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. Each task combines an instruction, a client matter’s relevant materials and a requested work product. Harvey says its binary criteria cover items such as facts, conclusions, citations, deadlines, dollar amounts and formatting. Under its all-pass approach, every criterion must pass for the task to count as complete.
The leaderboard result comes from Vals AI’s evaluation rather than a score disclosed only by Z.ai. Vals’ archived Version 1 table, updated October 18, places the model below several entries, including Muse Spark 1.2 at 25.42% and Gemini 4 Argon at 19.58%. Z.ai’s “state of the art among open-source models” statement is narrower than a claim to lead the entire leaderboard, but the company’s documentation does not identify its comparison set, and the supplied materials do not establish that the legal model’s weights are publicly available.
Harvey’s separate initial baseline, published May 26, reported a best overall all-pass rate of 7.1% among the frontier models it tested. That earlier result is useful as an indication that the benchmark is demanding, but it came from a different run and should not be treated as a direct comparison with Vals’ later leaderboard.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model, achieving state-of-the-art performance among open-source models in this evaluation.””
— Z.ai
What the Leaderboard Cannot Establish
The score does not show how the model performs across every legal matter, jurisdiction or firm workflow. Vals’ published result reflects a 120-task private holdout, not the benchmark’s entire 1,250-task public set. It also relies on two AI judges, GPT-5.5 and Claude Sonnet 4.6, whose task pass rates are averaged. The supplied materials do not provide enough detail to assess how the result changes with other judges, repeated runs or different task mixes.
A four-task spot check described in the source article found that GLM Legal cited rulings the reviewer could not locate; the same small check found no invented rulings from the OpenAI and Claude models tested, while two Claude models cited a real ruling for the opposite of its holding. This is an informal check, not a benchmark or representative study, and it cannot establish comparative error rates. The source material also says Z.ai’s pages describe outputs as drafts for legal professionals, but do not carry a “not legal advice” notice. It remains unclear what safeguards, citation checks or client-data terms apply to specific deployments.
Questions Before Firm Deployment
Z.ai’s AutoClaw announcement opened a beta by application, so access and product details may change. Firms evaluating the model will need to check current terms for client-data processing, retention and access, as well as which laws, cases or company records are connected through the services Z.ai recommends. The model’s benchmark result does not answer those procurement questions.
Further leaderboard updates could change the model’s position or provide additional comparisons. Readers should also look for clearer documentation of Z.ai’s “open-source” comparison set and whether the legal model’s weights are available. Until broader testing and deployment details are public, the defensible conclusion is limited: GLM-5.3-FlashX-Legal earned a strong relative placement on one evaluated leaderboard, while its all-pass rate and the benchmark’s scope call for careful interpretation and human verification.
Key Questions
What does GLM-5.3-FlashX-Legal’s 19.17% score mean?
On Vals AI’s leaderboard for Harvey’s benchmark, it means the model passed every required criterion on 19.17% of evaluated tasks. A task with even one failed criterion does not count as a pass.
Does the score mean the model gets only 19% of its answers right?
No. It is an all-or-nothing task completion rate, not a general accuracy score. Vals separately reports a 92.5% criteria pass rate, but that figure does not guarantee that any particular answer is complete or safe to use.
Was the 12.5 percentage-point improvement verified?
The difference in Vals’ archived table is 19.17% for GLM Legal minus 6.67% for the base model, or 12.50 percentage points. That supports the arithmetic of Z.ai’s claim for those entries.
Does the result prove the model is the best open-source legal AI?
No. Z.ai describes it as state of the art among open-source models in this evaluation, but the provided material does not define that comparison set or establish that the legal model’s weights are publicly available. It ranked seventh of 81 entries overall on Vals’ archived leaderboard.
Can law firms use the model without checking its work?
The benchmark does not support that conclusion. Legal professionals should verify citations, facts, deadlines and completeness before using or sending generated work, and assess data-handling terms before connecting client materials.
Source: ThorstenMeyerAI.com
