🔍 Read the full analysis: A Closer Look At 42 AI Setups And Their Legal Question Scores on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A November 4, 2026, report on Lexassay says 42 AI setups were tested on 15 legal questions spanning German, U.S. and EU law. The operator reports low scores for a law-focused model and frequent citation problems, while warning that the test is a limited spot check and that its grading uses Claude agents. The findings are not proof of how every model will perform in legal work.
Thorsten Meyer AI reported November 4 that 42 AI setups scored unevenly on Lexassay, a test of 15 legal questions covering German, U.S. and European Union law. The site says the law-focused GLM-5.3 FlashX Legal scored 8 out of 100, while its top result, Claude Opus 5.5, scored 79; the publisher also discloses that it runs the test and that Claude agents graded the answers.
According to the report, Lexassay gave each setup the same questions: 10 published and five hidden tasks. The questions covered legal deadlines, recent changes in law, court decisions and prompts built around false premises. Models answered without internet access or tools, so the results are intended to measure what they can produce without external research. The report does not provide enough detail in the supplied material to independently verify every score or reproduce every run.
The publisher says GLM-5.3 FlashX Legal received 8/100 and scored zero on the hidden tasks. Its citation review recorded 21 real citations, 25 misattributed citations and one invented citation. Mistral Large 4 scored 10, while Claude Opus 5.5 led at 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The report says the leader fully met all criteria on eight of the 10 published tasks, not every task.
Across published runs, Lexassay says it checked 2,644 citations as real and identified 804 as invented or as not supporting the claim made. These figures are the test operator’s findings, not an external audit. Meyer says answers were graded blind by several Claude agents and that the full public answers, gold answers and citation checks are available for inspection.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Legal Citations and Deadlines at Risk
The results speak to a practical risk: a fluent answer can be wrong about a filing deadline or the authority supporting a legal claim. In the example used by the report, two German appeals have different filing dates because Corpus Christi is a holiday in Bavaria but not Berlin. The report says one tested model assigned the later date to Berlin, while another calculated the deadline for written appeal grounds a month too late.
Those examples illustrate why a legal chatbot’s answer should not be treated as a verified legal conclusion. If a court rejects a late appeal as inadmissible, an error of one day can have serious consequences. The citation tally also matters: a real case or statute may still be cited inaccurately, and an invented citation can make an answer appear more authoritative than it is. The source recommends checking each citation against the original law or ruling and consulting a lawyer for deadlines, contracts or dealings with authorities.
The results should not be read as a direct measure of performance by every legal AI product in actual practice. The test excludes browsing and tools, and the publisher describes it as a spot check rather than a strict benchmark. Its findings are most useful as a warning about unverified answers under the conditions tested.
Products Worth Considering
Anker SOLIX C300 Portable Power Station, 288Wh
Samsung 34-Inch Odyssey G5 Ultra-Wide Curved Gaming Monitor
Amazon Kindle Scribe (64GB) with Premium Pen, Metallic Jade
As an affiliate, we earn on qualifying purchases.
How Lexassay Scored the Models
Lexassay evaluates model answers against a prepared gold answer using three to six weighted criteria per task, according to the method description summarized in the report. Each citation is checked and classified as real, misattributed, invented or not findable. Meyer says a misattributed or invented citation costs a quarter of the task’s points, while an answer without a legal-advice disclaimer or a pointer to a lawyer loses 5 percent.
The 42 setups include different models and settings; the report notes that effort levels, meaning how long a model may reason before responding, are tested separately. That can result in more than one setup for the same underlying model. There was one answer per model and task, so the publisher cautions that small score differences may be noise. Hidden questions are used to limit the value of memorizing publicly available answers, but only their scores are released.
The report also points to a different result from Vals AI’s legal-agent benchmark, where models work with supplied documents. It says Mistral Large 4 ranked eighth there, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th. Because the tasks and testing conditions differ, the rankings are not directly comparable. Meyer’s disclosure is material: his site operates Lexassay, and Claude agents graded the answers even though a Claude model ranked first.
“It is a spot check, not a benchmark in the strict sense: one answer per model and task, so a few points of difference are noise.”
— Thorsten Meyer AI, describing the test
Limits of the Reported Scores
The supplied report does not establish how the models would perform when given access to legal databases, current web search, supplied case files or professional review. Its offline format tests answers without tools, and one response per task and setup leaves results vulnerable to variation between runs. The publisher itself says small differences may be noise.
The disclosed grading arrangement also warrants scrutiny: Thorsten Meyer AI operates Lexassay, Claude agents grade the responses, and Claude Opus 5.5 ranks first. The site says its method and public answer materials let readers inspect the scoring, but the information supplied here does not include an independent replication or audit. The reported totals therefore remain the test operator’s results, not independently confirmed measurements.
It is also unclear from the provided material how all 42 setups performed, what exact conditions applied to each version and effort setting, or whether the models have since changed. The article gives selected scores and examples, but those details do not support a conclusion that any one system is reliable or unreliable for every legal task.
Review the Answers and Verify Citations
Lexassay says it publishes the public answers, gold answers and citation checks so readers can examine how scores were assigned; the hidden tasks are represented by scores only. Readers seeking to evaluate the findings can compare those materials with the method page and check cited legal authorities against official sources. The report does not announce a date for a follow-up test or an independent review.
For people using AI in a legal matter, the immediate practical step is not to rely on a chatbot for a court deadline or treat a citation as verified merely because it looks precise. Check the underlying law and ruling, and ask a qualified lawyer when a deadline, financial claim, contract or government authority is involved. The test measures selected model responses; it does not replace legal advice.
Key Questions
What did the 42 AI setups answer?
They were given the same 15 questions about German, U.S. and EU law, including deadline calculations, recent legal changes, court decisions and questions containing false premises. The test ran without internet access or tools, according to the publisher.
Which setup scored highest?
The report names Claude Opus 5.5 at 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The publisher says the leader fully met the criteria on eight of the 10 public tasks.
How did the law-focused GLM model perform?
Lexassay reports 8 out of 100 for GLM-5.3 FlashX Legal and zero on the hidden set. It records 21 real citations, 25 misattributed citations and one invented citation for that model.
Can these scores show whether AI is safe for legal work?
No. The operator calls this a spot check, with one answer per task and no internet or tools. It does not establish performance in every legal setting, and the publisher discloses that it runs the test and uses Claude agents to grade answers.
What should someone do with an AI-generated legal answer?
Verify each cited law or ruling against an authoritative source and do not rely on a chatbot for a filing deadline. For a deadline, contract, claim or communication with an authority, consult a qualified lawyer.
Source: ThorstenMeyerAI.com
