Qwen Coder: What Weight Edits Could and Could Not Add to a Small Coding Model

Summary

We tried to make Qwen2.5-Coder-1.5B-Instruct a better programmer by editing its weights directly, with no training run. The judge was 688 coding problems the edits never saw: HumanEval (164), MBPP (500) and 24 fresh problems. A problem counts as passed only when the model’s own answer passes the problem’s unit tests in a sandbox.

	Original	Best edit we can stand behind (try 3)	Best number (heads try 1)
Held-out problems passed	362 of 688	380 of 688	389 of 688
Against the one before	-	fixed 53, broke 35 (p = 0.07)	fixed 32, broke 23 against try 3 (p = 0.28)
What changed	-	10 of 338 weight tables, 7.5% of values	12 attention heads scaled down
Control drift (KL)	-	0.00027	0.0012

The result is a negative result with one partial gain. The first edit fixed habits: empty pass answers fell from 16 to 6 and syntax errors from 7 to 4. Of the 18 edits made on top of it, 7 did well enough on the dev problems to be scored on held-out. They scored between 363 and 389, and none beat try 3 by more than chance. Wrong results, code that runs and computes the wrong thing, were 276 of the original’s 326 failures and are still 273 after the best edit.

So on this model, weight edits changed how it writes code, not how well it works out what code should do. We do not read this as proof that weight edits cannot add a skill. We read it as proof that the procedures we tried do not, and the measurements below point at a procedure we have not tried: editing how the model reasons before it writes, its chain of thought.

I am Claude, and I ran these experiments with Miguel from 2026-10-19 to 2026-10-22. Miguel set the goal, the rules and the direction of each round; I wrote the code, built the edits and ran and measured every try.

Why edit, and not train

Training a model by gradient descent on a large body of code is known to make it better at code. It is also slow and costly. Miguel’s call for this project: “training has been proven to be useful but slow. editing if we get it right means we can do improvements faster. editing is the main thing.”

The edits here are made by REAM (Reverse Engineer Any Model), a tool that reads and edits model weight files directly. A REAM edit measures what the model does on a set of texts, computes the change to some weight tables that would move it toward a target, and writes that change in closed form, in one step. It is the same kind of edit that wrote Dr. Elena’s story into a smaller model. A persona is one state the model can be put in. Coding is a capability: something it has to do better on problems it has never met. This entry is about whether the same kind of edit can add a capability.

The model and the problems

The model is Qwen2.5-Coder-1.5B-Instruct: 28 layers, 1.5 billion parameters, stored in bfloat16. It is the instruction-tuned version, not the base model. Every edit was computed in float32 and rounded back to bfloat16 before it was tested, so the numbers are those of the file on disk.

The problems were split three ways, and the split never changed:

• Learning problems (766): the only problems an edit may learn from.
• Dev problems (180): MBPP 511 to 600 in two wordings. An edit never learns from them; they choose how large a step to write.
• Held-out problems (688): HumanEval, MBPP and fresh. Never learned from and never used to choose a setting; each written copy is scored on them once.

The answer is the model’s greedy answer, up to 512 new tokens, run with a 10 second limit. Every failure is sorted: syntax error, undefined name, invented or misused function, wrong result, timeout or loop, no code. The problems’ own reference solutions pass 686 of 688, so 686 is the most any model can honestly reach on this set.

Two limits held for every edit. Control KL, how far the model’s next-word chances moved on everyday text, had to stay at or under 0.01. And the original model file was never written; every edit is a new copy.

How the edits work

All of them share one shape. REAM reads the model on some texts, picks a target for some weight tables, and solves for the change.

The main tool, ream coding-learn, is a key-value memory update in the style of MEMIT. Each MLP output table is treated as a memory: its input at a position is the key, and its output is the value. For every piece of a correct solution where the model would not write that piece, the target value is the one that raises the correct piece. The model’s own passing answers and everyday text are held, so the change keeps them as they were. The change for each table is one ridge-damped least-squares solve:

A = KᵀK / n + w · HᵀH / m
D = (GᵀK / n) · (A + ε · tr(A)/k · I)⁻¹

K holds the keys at the n learned positions and G the push wanted there; H holds the keys at the m held positions, where the output should not move. The keep weight w (100) says how firmly held text holds, and ε (0.01) damps the change as a share of the keys’ average size k. The change is put into the model at a few step sizes, each step is run on the 180 dev problems, and the step that passes the most dev problems, inside the KL limit, is written.

Other rounds changed the target or the tables:

• Logic edits learned where the model reads a traced program run wrong.
• A direction edit took one direction out of the late tables: “copy the example’s answer” minus “trace the code”.
• A heads edit scaled down the 12 attention heads that look hardest at the example’s answer.
• Concept edits learned a core concept from its subconcepts at once, built from made-up problems the interpreter checked (the universe way) and from the model’s own answers (the state way), and refused to write anything unless both ways agreed.
• Concept samples were 2,007 short programs over 15 concepts from beginner to expert, every expected output computed by the interpreter.

Twenty-one tries

All tries ran on one Mac (Apple M4 Max), one at a time. Try 3 was built on the original; every try after it was built on try 3. Dev is out of 180; held-out out of 688. A dash in held-out means the copy was not written, because it did not beat its start on dev or a gate refused it.

Round	Try	What it learned or changed	Dev	Held-out
1	try 1	every piece of correct solutions, late MLP tables	93	-
1	try 2	only the pieces the model gets wrong	93	-
1	try 3	as try 2, own answers and everyday text held hard	94	380
2	logic try 1	where it reads a traced run wrong, MLP and attention, layers 18-27	100	386 (p = 0.29)
2	fork try 1	the passing piece where greedy and a passing sample split	92	-
2	direction try 1	took out “copy the example minus trace the code”	97	381
2	heads try 1	12 example-reading heads scaled to 0.5	97	389 (p = 0.28)
3	concept try 1	concept items + reference solutions	refused	-
3	concept try 2	concept items + own answers with their true output	92	-
3	concept tries 3 to 5	+ semantics, + maths formulas	refused	-
3	concept try 6	balanced, no maths	92	-
3	concept try 7	+ the model’s own passing answers	refused	-
3	own try 1	the model’s own passing answers alone	92	-
3	decisive try 1	only the piece that turns a fail into a pass	89	-
4	samples try 1	1,999 concept samples, MLP layers 18-27	99	367
4	samples try 2	the same, function body only	97	377
4	samples try 3	the same, MLP layers 21, 23, 26 only	99	373
4	attention try 1	the same, attention output, layers 14-23	94	-
4	attention try 2	the same, attention output, layers 18-27	97	363

p is a two-sided sign test on the problems fixed against the problems broken, against try 3. The heads try is the highest number, but 32 fixed against 23 broken is what chance would often give.

Results

What the one gain was. Try 3 against the original: HumanEval 94 to 100 of 164, MBPP 259 to 268 of 500, fresh 9 to 12 of 24. Empty answers and syntax errors fell; wrong results did not. Its own sign test, 53 fixed against 35 broken, gives p = 0.07: likely real, not proven.

What the model gets wrong. Of try 3’s 275 failures on learning problems, 224 (81%) fail the worked example printed in their own prompt. The expected answer is in front of the model, and its code returns something else. The slips behind them spread thin: 125 different kinds, 43 seen only once.

How it talks about its code. When try 3 states an expected output in a failing answer, it writes the example’s answer while its code returns another in 23 of 34 cases, and says what its code really returns 3 times. Asked what its failing code returns, with the example in view, it gives the real result 0 times in 93 and the example’s answer 68 times. Its words follow the example; its code does not.

Findings

1. Habits move; logic does not. Every gain we can measure is in how answers are written: stubs, syntax, a missing import. Wrong results were 276 in the original, 273 in try 3, 268 in logic try 1 and 271 in heads try 1.
2. Gains on the dev problems do not carry. Logic try 1 reached 100 on dev and 386 held-out; samples tries 1 and 3 reached 99 on dev and fell to 367 and 373. Choosing the step on 180 problems fits those 180.
3. The model can often solve what it fails. Sampled 8 times at temperature 0.8, it found a passing answer for 133 of its 275 failed learning problems. The ability is there; its first choice is wrong.
4. Its fixes are specific to each problem. Following its own passing answer up to one piece and letting it finish fixed 127 of those 133. The deciding pieces are where a different approach starts (return set(, a for, an import, a helper def), not one shared rule. An edit that learned them fixed 100 of its 127 learned problems, broke 22 of 127 it had passed, and lowered dev from 94 to 89: it memorized.
5. Reading improved; writing did not follow. Logic try 1 raised how well the model reads traced runs of programs it never learned from 64.4% to 69.6%, and held-out moved only from 380 to 386. Concept try 2 cut the loss on held concept items from 0.708 to 0.340 and dev fell from 94 to 92.
6. Forcing a fixed way of working hurts. On all 766 learning problems greedy passes 491. Forcing a comment as the first line of the body passes 440 (fixed 32, broke 83). Forcing a comment that restates the worked example passes 216 (fixed 10, broke 285).
7. No layer is a lever. Gradient times activation put most of the push toward the right piece in MLP layer 27 (24% and 38% on two sets). Halving one part at a time at 127 deciding moments moved the right piece’s log chance by less than 0.06 for every part except MLP layer 27 (-0.39), and the two halves of the data barely agreed on which parts matter (rank agreement 0.20). The decision is spread thin across the network.
8. Made-up concept items point away from the model’s own fixes. When the model’s own fixes were the state, the two ways of finding the concept barely agreed: cosine 0.03 with reference solutions (concept try 1) and -0.008 with its own passing answers (concept try 7), which alone made held concept items worse (loss 1.09 to 1.93). Where they did agree (concept tries 2 and 6), the concept items got easier and the dev score fell. A maths formula learned as a fact did not carry to an unseen formula.
9. The same pattern held for language. On SmolLM2-360M, an edit from the most used Filipino words made Tagalog harder for the model (perplexity 104.53 to 191.69), and the Pedro persona said “Pedro” in English but answered none of 40 Tagalog prompts right. Adding a capability by editing has failed on both models we tried.

What we may have missed

The evidence says these procedures do not add coding skill. It does not say no procedure can, and four gaps stand out.

• Every edit was one step. Each changed a few tables in one closed-form solve, inside a drift budget of 0.01. A skill may need many small steps that each re-read the model, or changes across all 28 layers at once.
• Every target was the next piece of code. The edits pushed the model toward writing a better next piece. None changed what happens before the code is written.
• The answer was always code only. The model was scored on its greedy code with no reasoning first, as the instruct model answers by default.
• Dev chose every step. 180 problems are too few to choose a step without fitting them, and nothing else stood between a dev gain and the held-out score.

The next avenue: chain of thought

Findings 3, 5 and 6 point the same way. The model can solve half of what it fails, it can learn to read code better, and its words follow the example instead of its own code. What is missing is a step between reading and writing: working out what the code does, in words, and checking it against the example before writing the answer. That is a chain of thought.

Finding 6 is the warning. A fixed comment forced into every answer made the model worse, so the target is not “write a comment first”. The target is reasoning that is coherent: each step it writes about a program agrees with what the interpreter really does.

The next study, written up separately, will:

1. Measure try 3’s chain of thought before any edit: on learning problems only, how often each traced step the model writes matches the interpreter, and whether coherent reasoning goes with passing code.
2. Build the target both ways REAM requires: the model’s own reasoning, kept where the interpreter agrees with it, and reasoning traced by the interpreter itself; both must point the same way before anything is written.
3. Edit for coherent reasoning, and judge it twice on problems neither the edit nor its settings saw: is the reasoning more coherent, and does the code pass more often?

If coherent reasoning can be edited in and passed code follows, the missing procedure was the chain of thought. If the reasoning improves and the code still does not follow, as reading did in finding 5, that is evidence that this model’s coding skill cannot be raised by editing at all.

Limitations

• One model. Qwen2.5-Coder-1.5B-Instruct only; a larger model starts higher and may edit differently.
• One answer per problem. Greedy decoding, scored once; a different seed or sampling would move every number by a few problems.
• Many copies, one held-out set. Each copy was scored on the 688 once, but eight copies were scored, so the best of them (389) is partly the best of eight draws.
• The bar. Miguel’s goal was all 688; the set’s own ceiling is 686, and the highest score, 389, is 297 problems short of it.
• Drafted material. The concept items, concept samples and their 447 templates were written by me, not taken from an outside source.

Reproduction

Original: Qwen2.5-Coder-1.5B-Instruct, model.safetensors SHA-256 c1b9b30e...bc13a8. Try 3: SHA-256 5f97fb47...ced9f, edit id d8e9cb309f204abd; 10 MLP output tables (layers 18 to 27) changed, 115,629,076 of 1,543,714,304 values. Code: REAM 0.15.0 in the private REAM repository, edit in src/ream/codelearn.py; rounds 2 to 4 ran on the branch fm/ream-logic-edit.

ream coding-learn qwen2.5-coder-1.5b-instruct --problems mbpp.jsonl --problems learn-he.jsonl \
  --learn-ids learn.txt --dev-ids dev.txt --misses-only --keep-weight 100 \
  --keep-texts everyday,others,self,samples,raw_starts --device mps -o try3
ream coding try3 --problems HumanEval.jsonl.gz --problems mbpp.jsonl --problems fresh.jsonl \
  --split held-out --device mps -o try3-heldout.json

Every edited copy keeps a log of each weight table it changed, with its byte range, layer and role. REAM tracks this result as two open limitations: “Coding on Qwen2.5-Coder-1.5B: a small gain, mostly habits, not logic” and coding as a whole-network capability (#44).