🔍 Read the full analysis: One Matrix Edit Moved Qwen From 49 To 70 On JSON Tab Escaping on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
An experiment led by OpenAI’s GPT-6 Astra edited one matrix in Qwen3-4B-Instruct-2507 to address a narrow JSON-escaping failure. On a frozen test, exact answers rose from 49 of 120 to 70, while a separate table-row extraction task remained at zero; the synthetic result has not received an independent review.
The model had to place short source texts inside JSON while preserving every character. A literal tab in the source needed to be represented as the JSON escape sequence \t, then decode back to the original tab. The answer counted only if parsing the JSON reproduced the source text exactly. The experiment report says a more explicit escaping prompt did not resolve the problem: the original model and the prompted version each passed 16 of 32 screening cases.
The Codex agent made a calculated rank-one update to the output projection matrix in the final multilayer perceptron layer. The report says it did not use gradient training and that the model’s other 397 stored tensors remained bit-identical to the original. On the frozen 120-case test, the edited model produced 70 exact answers versus 49 for the original: a gain of 21 cases, or 17.5 percentage points. The report records no cases fixed in one direction that became wrong in the other.
The gain did not carry over to every task. In a separate test extracting a table row, the score remained 0 of 30. On the final JSON test, 50 of 120 answers from the edited model were still incorrect. The source article describes the cases as synthetic and says a second AI has not yet checked this experiment.
Infographic 01 · The measured result
Where the edit helped, and where it did not
Qwen3-4B-Instruct-2507, original versus one edited matrix, on synthetic texts that must survive the trip into JSON without losing a character.
Final test · 120 cases by group
- Texts with tabsthe target failureOriginal19/30Edit30/30
- Line breaksalready perfectOriginal30/30Edit30/30
- Mixed special characterstabs, quotes, backslashesOriginal0/30Edit10/30
- Second table rowstill unsolvedOriginal0/30Edit0/30
Development, validation and controls
- Developmentused to fit the edit · not independentOriginal24/40Edit40/40
- Validationnew cases · 9 wins, 0 lossesOriginal10/40Edit19/40
- Random directioncontrol on validation · same strengthOriginal10/40Edit10/40
- Plain-text tabscontrol · real tabs must surviveOriginal10/10Edit10/10
Exact answer = after normal JSON parsing, the text field matches the source text character for character. Same prompt, temperature 0, seed and output limit for both models; saved and reloaded edit, no hooks, no grammar constraints, no repair. Final test, validation and controls frozen before any independent output of the edited model. Fifty of 120 final answers are still wrong.
Sources: our experiment records, 5 Nov 2026 (Lexassay research, internal): experiment 3 result report and final result by GPT-6 Astra working as a Codex agent (OpenAI). Synthetic test cases, not published.
Powered by Thorsten Meyer AI
A Narrow Gain in Exact JSON Handling
The result offers a small, task-specific example of changing model behavior by modifying weights rather than relying only on a prompt. That distinction matters for teams evaluating whether a model can reliably preserve special characters in structured data: a malformed or altered value can make an otherwise valid-looking response unusable. In this test, success required both valid JSON and an exact round trip of the input text.
But the numbers do not establish a general improvement in Qwen’s ability to follow instructions or produce structured output. The result covers one synthetic task and one model, with 50 errors still present and no gain on the table-row test. It is evidence worth checking, not a demonstration that a single-matrix edit broadly improves reliability or is ready for deployment.
Products Worth Considering
Amazon Kindle Paperwhite Signature Edition, 32GB (2024), Metallic Black
EcoFlow Delta 3 Portable Power Station – 1024 Wh, 1800 W
Logitech MX Master 4 Ergonomic Wireless Mouse with Haptic Feedback, Graphite
As an affiliate, we earn on qualifying purchases.
The Earlier Qwen Extraction Trial
This was the third experiment in a series described by ThorstenMeyerAI.com as tests of AI systems editing or reviewing other models’ weights. In an earlier trial, GPT-6 Astra attempted to improve the same Qwen model’s extraction of fields from synthetic business messages. The frozen final test in that experiment scored 97 of 200 for both versions, so the edit did not improve the final result.
The earlier trial did provide a lead for this one: both versions produced 47 invalid answers, mostly by placing raw tab characters inside JSON strings rather than escaping them. The source says the new experiment targeted that observed failure rather than searching broadly for weaknesses. The series also includes a separate experiment in which Claude edited a different local model; GPT-6 Astra reviewed that work and identified issues. Claude was not involved in the Qwen experiment described here.
According to the source, GPT-6 Astra, operating as a Codex agent, planned this experiment, inspected the weights, created and froze the tests, calculated the edit and produced the report. The site owner ordered the work on November 5, 2026. The source says the agent used extra-high reasoning effort. Those details describe the reported process; they do not constitute independent validation of its findings.
““One matrix out of 398, one very narrow task, and a test that was frozen before anyone saw the result.””
— ThorstenMeyerAI.com experiment report
Independent Checks Are Still Pending
The result comes from an internal experiment described by the site that commissioned it. The available source material does not provide an independent replication, an external review of the test design, or a second AI’s audit of this particular experiment. The source explicitly says that such a review has not yet happened.
The test cases are synthetic, and the material does not establish how the edit would perform on real-world JSON workloads, other models, or different types of structured-output tasks. It also does not detail enough of the test construction or scoring procedure to assess those questions independently. The reported 21-case gain should therefore be read as a result on this fixed test, not as a general performance claim.
The source says the other 397 tensors were unchanged, but does not provide independent verification of the stored model files in the supplied material. It also does not report whether the edit affects other capabilities beyond the separate table-row task, which remained at zero.
Replication and Broader Testing
The next useful step is an independent review of the experiment records and the frozen test, followed by a replication that checks whether the same matrix edit produces a similar gain. Testing on additional examples, including non-synthetic inputs, would help show whether the improvement extends beyond the cases used here.
Further evaluation would also need to test for regressions in other model tasks and measure how often the edited model returns invalid JSON or changes source text. No such follow-up schedule or results are stated in the source. Until those checks are reported, the confirmed finding remains a 21-case gain on one 120-case synthetic test, alongside the unchanged zero score on the separate table-row task.
Key Questions
What did the edit change?
The GPT-6 Astra Codex agent made a calculated rank-one update to the output projection matrix in Qwen3-4B’s final MLP layer. The report says the other 397 stored tensors were bit-identical to the original and that no gradient training was used.
What does the score improvement mean?
On a frozen test of 120 cases, exact answers increased from 49 to 70, a gain of 21 cases or 17.5 percentage points. An answer counted only when valid JSON decoded to exactly the original text, including tabs and other special characters.
Did the edit improve all the tested tasks?
No. The separate table-row extraction test remained at 0 of 30. The report also says 50 of the edited model’s 120 final JSON answers were still wrong.
Has another reviewer verified the result?
Not according to the source material. It describes the cases as synthetic and says this experiment has not yet received a second-AI review. Independent replication is also not reported.
Source: ThorstenMeyerAI.com
