🔍 Read the full analysis: Claude Edited, GPT-6 Astra Checked: Our At-Home AI Experiment on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment edited the weights of local model Ornith 1.5 35B-A3B to reduce repetitive loops. It completed 12 of 15 legal benchmark answers, compared with 4 of 15 in the matched unedited run, but scored zero in both; recommended sampling settings let the unedited model complete 13 of 15.
A home AI experiment edited the weights of Ornith 1.5 35B-A3B to reduce repetitive loops, raising completed answers from 4 of 15 to 12 of 15 in a matched test. The model’s legal benchmark score remained zero, however, and the unedited model completed 13 of 15 answers when run with the sampling settings recommended in its model card.
The test, reported by Thorsten Meyer on November 5, ran from Wednesday evening to Thursday morning on a local model and a 15-question legal benchmark covering German, US and EU law. The model had no internet access or tools. The benchmark checks cited statutes and court decisions against primary sources. The published account says the edit improved completion, not the legal content of the answers.
A Claude Opus 5.5 session wrote code to run the model, measure an internal direction associated with looping, and remove that direction at half strength from the output weights in 27 of 40 layers. The process did not train the model or add legal data, and Claude did not edit the weights by reading them as text. The author describes the technique as directional ablation, a simpler method than the weight-space programming approach in a recent MIT CSAIL post.
In the matched comparison, the edited model completed 12 of 15 answers, against 4 of 15 for the unedited model. Both scored zero on the legal benchmark. The report says the edited answers contained 62 wrong or invented citations, an average of 5.2 per answer. Across 75 checked criteria, the edited model met seven, compared with three for the unedited run. With the model card’s recommended sampling settings and no weight edit, the original model completed 13 of 15 and also scored zero.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, 15 tasks, graded blind by three Claude agents
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Finished Answers, Same Legal Score
The result separates completion from correctness. A model that stops looping may produce more answers, but this test offers no evidence that the edit gave it legal knowledge or made its citations reliable. For readers evaluating AI for legal work, the distinction matters: a complete response can still contain fabricated authorities, and a zero benchmark score does not become useful merely because the system reaches the end.
The comparison also cautions against attributing a practical improvement to a weight edit alone. The unedited model completed more questions under the recommended sampling setup than the edited model did in the matched comparison. That does not prove sampling is a generally superior fix, but it means the reported result is not evidence that editing weights improved the model’s legal performance. The experiment is a small demonstration of a technical effect, not a validation of legal use.
The report says the benchmark’s public website still displays only the original Ornith result, not these new runs. Readers should treat the figures as findings from this experiment rather than as a revised published benchmark score.
Products Worth Considering
Apple MacBook Air 15-inch with M5 chip, 24GB RAM, 512GB SSD, Midnight
Samsung Galaxy Tab A11+ (6 GB RAM, 128 GB storage, expandable) – Gray
Apple MacBook Air 13.6-inch with M5 Chip, 16GB Memory, 512GB SSD, Sky Blue
As an affiliate, we earn on qualifying purchases.
A Home Test Inspired by MIT
The experiment followed an MIT CSAIL post by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, which described “weight-space programming”: using a language model to inspect and change another network’s weights directly, without a conventional training loop. The post’s reported Qwen example stopped looping after an edit, and its maths score rose from 7.9% to 48.1%. Those are the MIT team’s reported results, not results from Meyer’s home experiment.
Meyer says the home test did not replicate the MIT method. Instead, Claude-generated code ran the model and measured its internal signals before removing a direction from selected weights. The report says this kind of directional edit has been used in prior research, including work on removing refusals from chat models. Its test focused on whether addressing a looping problem would help a model whose legal answers were already weak.
Ornith was chosen because the initial checks reportedly found conspicuous looping in its benchmark answers. It is a mixture-of-experts model with about 35 billion parameters in total and roughly 3 billion active per token, according to its model card, and its publisher built it for coding and agent work rather than law. The report says other local models either lacked the same looping behavior or could not be tested under comparable conditions.
“Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked.”
— The MIT CSAIL researchers, as quoted in Meyer’s account
Limits of the Small Test
The results come from one run per setting, and the report identifies several limits: a small development set, a loop detector that changed during the experiment, and three graders from the same model family. Claude carried out the edit and Claude agents performed blind grading, creating a potential conflict of interest. OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, numbers and claims and reportedly found faults; the account does not describe this as a second experiment.
The local setup also changed during testing. According to the report, that change alone shifted the unedited model from 10 completed answers to four. The supplied account does not provide enough detail to establish how much each methodological or setup difference affected the result. Nor does the experiment show whether the weight edit would work on other models, benchmarks or repeated runs.
The report says the 12 edited answers contained 62 wrong or invented citations, but does not provide a separate breakdown of citation errors by question in the supplied material. The findings therefore do not establish the rate or type of errors that might arise in other legal tasks. The article itself states that the experiment is not legal advice.
Replication and Better Comparisons
The next useful step would be repeated tests with the same model, hardware and sampling settings, comparing the edit against an unedited baseline under controlled conditions. A larger evaluation set, fixed loop detector and graders with greater independence could help clarify whether the edit reliably reduces looping and whether that change affects answer quality.
For readers, the immediate practical takeaway is to distinguish fewer loops from better answers. The reported data show that the edit can coincide with more completed responses in one matched comparison, but they do not show improved legal performance. The author says the new runs are not yet reflected on Lexassay’s public results page. Whether they will be added, and whether further experiments will reproduce the effect, remains unclear.
Key Questions
What did the weight edit change?
Claude-written code measured a direction associated with looping and removed it at half strength from output weights in 27 of 40 layers of Ornith 1.5 35B-A3B. The report says there was no training and no new legal data.
Did the edited model become better at law?
Not by the benchmark’s score: the edited and unedited runs both scored zero. The edited run completed more answers than the matched unedited run, but the report also records wrong or invented citations.
Why compare the edit with recommended sampling settings?
The unedited model completed 13 of 15 answers using the model card’s recommended sampling settings, compared with 12 of 15 for the edited model in the matched test. That comparison makes it difficult to attribute a completion advantage to the edit alone.
Was this the same method used in the MIT study?
No. Meyer says the home test used directional ablation: code ran the model, measured an internal direction and removed it from selected weights. The MIT post described a different approach in which GPT-6 Astra reasoned about another model’s weights directly.
How reliable are the findings?
The report describes one run per setting, a small development set, a changing loop detector and grading by agents from the same model family. It also notes a local setup change that affected completion counts. The result needs controlled replication before it can support broader claims.
Source: ThorstenMeyerAI.com
