lesion

tl;dr

Models may adapt around the internal signals we use to understand and monitor them, making those signals less dependable after further training. I tested this by training a 4B model on a two-hop geography task while continuously suppressing its top-10 eligible J-lens directions at six middle layers. The lesion was active during every training forward pass. Accuracy rose from 20.9% to 97.7% on a selected cohort of countries excluded from training. Recovery persisted even after refitting the adapted models. The lesion still changed their activations, but its later-layer effect became smaller. The model learned to tolerate the intervention; exactly how it did so remains open.

The circuits in our brain are constantly adapting to experience and external stimuli. Children with Rasmussen's encephalitis can recover substantial language function after the left hemisphere is removed, with the right hemisphere supporting further language development (a form of rerouting). That recovery is incomplete: verbal comprehension may recover even while grammatical judgment and speech production remain impaired (Language recovery after left hemispherotomy).

This raises a related question for language models: if a model's internals are disrupted during training, can it adapt around that disruption and recover the lost behavior?

In this post, I study that question through lesion-induced functional compensation. In a language model, we can impose an activation intervention, keep it active during training, and measure whether performance returns.

Anthropic's Verbalizable Representations Form a Global Workspace in Language Models identifies J-space representations associated with verbal report, internal reasoning, and flexible computation. Ablating active J-lens directions disrupts multi-hop reasoning while preserving much routine processing. I wanted to test whether supervised training could restore performance under this intervention, so I trained Qwen3.5-4B on a two-hop geography task with the lesion active throughout SFT, then evaluated on countries excluded from training. Accuracy rose from 20.9% to 97.7%, and the recovery survived refitting the lens to the adapted model. The experiments below distinguish behavioral recovery from changes in lens readouts and intervention magnitude.

Background

The residual stream carries activations between decoder blocks. The J-lens approximates downstream computation with a matrix $J_\ell$: the Jacobian from layer $\ell$ to the final residual stream, averaged over prompts and source/target token positions. Targets include the current and later positions. Its vocabulary readout is

where $W_U$ is the unembedding. Unlike a logit lens, it accounts for transformations between intermediate and final-layer coordinates. Averaging across prompts and positions helps capture general verbalizability rather than behavior on a single prompt (J-lens methods).

The workspace hypothesis proposes that some representations are shared across verbal report and other downstream computations. A high lens score alone does not establish that a particular computation uses that content. The paper tests this with interventions as well as readouts.

Readout Directions and the Intervention

In this implementation, the direction associated with token $v$ is the normalized row

where $g$ is the final RMSNorm gain. For Qwen3.5, $g=1+w$, with $w$ the stored norm parameter. The RMS denominator is common across vocabulary logits at a fixed activation, so it does not change their ranking. The intervention uses these readout-aligned directions; it does not differentiate through the normalization denominator.

I selected directions based on their logits at the current activation and subtracted their residual projections. This targets the direction that the lens reads most strongly at that position. It does not explicitly identify the intermediate country, remove a fixed set of neurons, or eliminate every representation associated with the country.

These three measurements answer different questions. Task accuracy indicates whether the model produces the answer, lens readouts show expression in a particular vocabulary basis, and interventions test sensitivity to the selected directions. Recovery in accuracy does not, by itself, reveal whether the model changed its internal representation or simply became less affected by the intervention.

Lesion Construction and Training

I needed a task the clean model could handle reliably. However, the lesion needed to cause enough disruption to make recovery meaningful while keeping training feasible (not completely destroyed). The controlled multi-hop geography task was a good fit, derived from Anthropic's jacobian-lens repository. The intended computation is city → country → capital, currency code, or geographic region. Answers are a single word or concept, so the model has to answer directly without writing out the intermediate country or a chain of thought. This data was an extension of Anthropic's own dataset in the jacobian-lens repo. I chose Qwen3.5-4B for its strong task performance and manageable size for SFT and lens refitting on my compute cluster.

At every prompt position, I intervene at decoder blocks 16, 18, 19, 20, 21, and 22. I chose these layers by scaling the normalized band of J-space layers identified for Claude Sonnet 4.5 in the Anthropic paper to a model of Qwen3.5's size.

The intervention procedure is:

1. Run the current model cleanly to collect its top-10 next-token predictions at each position. Those token directions are protected.
2. During the lesioned pass, use the J-lens to select the ten highest-scoring eligible directions from the current activation. Later layers therefore see the effects of earlier interventions.
3. Subtract the residual projection along each selected direction in sequence.

The intervention remained active during every training forward pass. Additional SFT details are provided in Appendix D.

Task Performance Recovers

With the lesion active throughout training, the key question was whether the model could recover the lost performance. For that recovery to be meaningful, the lesion first had to create a substantial performance drop. The clean base model answered all 129 selected examples correctly, while the lesion reduced accuracy to 27/129, or 20.9%. After SFT under the same lesion, accuracy recovered to 126/129, or 97.7%.

I wanted to know how much of the recovery came from training under the J-lesion specifically. To test this, I added two controls with the same training schedule:

The random control applies a random displacement matched to the J-intervention's removal norm at each position rather than deleting ten random directions. It was also much less damaging to the base model, which scored 107/129 under random perturbation. After training, performance under the same perturbation reached 124/129.

These controls show that both SFT and random-perturbation training improve J-lesion tolerance. But training under the J-lesion produces substantially more recovery than either control. On the 65-example screen subset, the recovered model answered 63 correct, compared to 41 for random training and 42 after sham.

Recovery Survives a Fresh Lens

The comparison conditions suggest that training under the J-lesion produced genuine behavioral recovery. However, another obvious explanation is that the model rotated its representations away from the fixed lens. If so, the intervention could appear less effective simply because the fixed lens was no longer tracking the model as well.

To test this, I fit fresh J-lenses to the recovered checkpoints using 500 fixed WikiText prompts and the coordinate-Jacobian estimator. I also fit a corresponding fresh lens to the unchanged base model. No task examples were used during lens fitting.

The recovered model scored 124/129 under its own fresh lens, nearly matching the 126/129 achieved with the published lens. In contrast, the base model still fell to 27/129 under its fresh lens. On the 65-example screen subset, the recovered model scored 61/65, compared with the base's 17/65.

These results make a simple stale-lens explanation less likely. However, they do not establish that the task no longer relies on any workspace-like representations, or that the same result would hold across every possible lens.

Activation Diagnostics

Now that we've ruled out the stale-lens explanation, what about the activation structure leads to task recovery? In this section I tried a wide range of interpretability techniques to get closer to a mechanistic explanation.

J-lens Readout Concentration

I first examined whether activation patterns disrupted by the lesion returned as performance recovered. For each prompt, I measured the excess kurtosis of the J-lens logits across the full vocabulary. Higher kurtosis indicates that the distribution is dominated by extreme values. This is a rough measure of readout concentration rather than a measure of country knowledge.

The activation pattern did not fully return to its clean state as performance recovered. The clean and lesioned curves still separate at the last intervention block. At block 22, the primary model's median kurtosis is 2.17 clean and 1.58 lesioned, even though task accuracy is almost fully recovered. The separation also persisted when each model was evaluated with its own fresh lens.

The effect was not uniform across layers. The clean and lesioned curves cross at block 18, so the direction of the change depends on the layer. Appendix C reports the paired statistics and fresh-lens comparison. These results describe how concentrated the full vocabulary distribution is, not whether the intermediate country's rank is decodable.

Residual Perturbation Magnitude

There are three ways a model could recover while the lesion stays on: it could change what its representations encode, it could learn to route the task through whatever the lesion leaves untouched, or it could arrange for the intervention to do less damage. I wanted to test the third case because it is a less interesting explanation of the rerouting behavior, so I measured the lesion's footprint on the residual stream.

I tracked two quantities at each intervention block. The norm ratio is the lesioned residual norm divided by the clean one, which says how much magnitude the subtraction removed. The cosine is the similarity between the clean and lesioned states, which says how far the subtraction rotated the residual.

Through the first half of the intervention band, the two models lose almost exactly the same norm. At later blocks, the base retains about 78% of its clean norm and the recovered model retains 84%. The recovered model's cosine is slightly lower than the base's at earlier blocks, and higher at block 22. The lesion still removes about as much as it did before training, and the recovered model only ends up better at the last block. Either way, training did not make the lesion weak, so the recovery must come from somewhere else.

Representation Similarity

That leaves the possibility that the model rearranged something. To try to see it visually, I measured centered linear CKA between every pair of decoder blocks using their final-prompt residuals across the 129 examples. Each cell summarizes how similarly two layers organize these prompts. The diagonal is always 1; the off-diagonal blocks show groups of layers with similar activation geometry.

The clean model showed two similar broad bright regions: one spanning much of the early-to-middle stack, and another among later layers. The intervening green regions indicate lower similarity between these groups. The final block is also distinct, visible as a darker outer row and column.

Applying the lesion preserves this broad structure but increases similarity between many early and late layers. This is the pale red off-diagonal rectangles in the lesion-effect panel. Task accuracy falls from 100% to 20.9% despite a comparatively small change in the overall CKA pattern.

After training, the early-to-late separation became substantially sharper. The recovery-effect panel contains two dark blue rectangles linking approximately blocks 0–5 with blocks 20–31, while much of the central region changes less. Mean CKA between those two layer groups is 0.72 in the clean base, 0.77 under the lesion, and 0.57 after recovery. The change extends to comparisons involving layers outside the intervention band.

Training restores the answers while producing a different organization of activations across depth. CKA shows that the model's internal geometry moved without identifying a mechanistic explanation. This is early evidence of lesion-induced functional compensation in language models, something only observed in human brains so far.

Discussion

Just as children with Rasmussen's encephalitis can recover substantial language function after the left hemisphere is removed, the model here recovered performance with a persistent lesion during training. In both cases, recovery does not require restoring the system to its base state, but rather enough plasticity for the system to accommodate new behavior.

While this was only a single demonstration (due to compute constraints), I believe the methods can be extended to large models and complex reasoning tasks. However, I think additional difficulties would emerge because many models now externalize part of their reasoning through CoT. If CoT reduces workspace dependency, future work may need to target the mechanisms for writing to the CoT for this to be an effective tool to study model internals.

One important application I would like to test is the robustness of alignment monitors and how targeted lesions can expose blind spots. If a model recovers deceptive behavior while the monitored directions remain suppressed, that would suggest those directions capture something not necessary for deception. We can then test the monitor to make sure it can still catch deception even if the model changes how it produces this behavior.

What I've shown is that a model can recover performance while the intervention that disrupted it stays active even after refitting the lens. I still don't know how much comes from changing the representations, learning to use what survives, or reducing the intervention's effect. I'm looking forward to seeing which of these explanations matter the most.

Related Work

• The Hydra Effect
• Explorations of Self-Repair
• Obfuscated Activations
• Measure, Don't Optimize

Appendix

Appendix A–F: methods, cohorts, diagnostics, and reproducibility

A. Intervention and Training Details

For an activation $h$ and selected unit directions $d_1,\ldots,d_k$, the sequential intervention is

$$h^{(0)} = h, \qquad h^{(i)} = h^{(i-1)} - \langle h^{(i-1)}, d_i\rangle d_i.$$

For nonorthogonal directions, a later subtraction changes the projection onto earlier directions. Exact erasure of the selected span would instead use $h' = (I - QQ^\top)h$, where $Q$ is an orthonormal basis of that span. The released implementation includes an orthogonal-span option, but the reported models were trained and evaluated with sequential projection.

Selection uses detached activations, so gradients do not pass through top-$k$. Once directions are selected, subtraction remains differentiable with respect to the live residual stream. Thus the loss updates the model through the intervention. The clean pass supplies the protected output tokens; it does not determine the active directions at later lesioned blocks.

The random control draws a unit direction $u$ and applies a displacement with the norm of the counterfactual J-removal:

$$h_{\mathrm{random}} = h - \|h - h_J\|_2\,u.$$

This matches displacement magnitude at each intervention position. It does not match task accuracy, selected subspace dimension, or semantic content.

Setting	Value
Base model	Qwen/Qwen3.5-4B
Model revision	851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Primary / replicate seeds	20260825 / 20260826
Objective	Cross-entropy on the single answer token
Optimizer	AdamW; betas (0.9, 0.95); no weight decay
Learning rate / warmup	1e-6 / 5 steps
Updates / accumulation	100 / 4 examples
Gradient clipping	Global norm 1
Precision	FP32 parameters and optimizer states; BF16 autocast
Frozen parameters	Final norm and unembedding
Gradient checkpointing	Disabled
Prompt format	Raw text; no chat template or generated reasoning

Each arm processes 400 training-example presentations, sampled by shuffling the 189 eligible rows across epochs. Trainable parameter inventories match across all four arms, and all 424 trainable tensors have gradients at the first update.

B. Cohorts, Relation Breakdown, and Uncertainty

The source geography dataset contains 1,296 rows. The selected training, validation, and screen cohorts contain 189/64/65 rows across 45/15/17 countries respectively, with no country overlap between those three splits. Country separation applies to this SFT experiment; it does not imply the pretrained model had never encountered those countries.

The headline combines validation and screen for continuity with the exploratory evaluation. The screen results should be read separately because validation selected learning rate and duration. The 49-prompt transfer evaluation comes from a separate Anthropic probe-swap cohort and also had prior exploratory exposure.

Recovery occurs across all three task relations under the published lens:

Relation	Primary	Second seed
Capital	49/50	47/50
Currency	23/24	22/24
Region	54/55	50/55

Behavioral intervals use a paired bootstrap over countries, retaining all prompts belonging to each sampled country. Across 10,000 resamples, the primary's published-lens improvement is 76.7 percentage points, with an exploratory 95% interval of [68.5, 85.8]. Its advantages over sham and random training are 31.8 [21.3, 43.9] and 39.5 [26.5, 53.1] points.

Under each model's own fresh lens, gains over the fresh-lens base are 75.2 [65.3, 84.6] points for the primary and 72.9 [64.5, 81.7] for the replicate. These intervals account for clustering within countries, but not task selection, hyperparameter selection, or uncertainty across model families.

C. Lens Fitting and Supplementary Diagnostics

The published base lens was fit on 1,000 WikiText prompts. Each fresh lens uses 500 fixed prompts, corpus rows 1000–1499, with maximum sequence length 128 and the first 16 source positions skipped. Fits map the six intervention blocks to block 31 using exact coordinate Jacobians, BF16 activations, FP32 accumulation, and FP16 saved matrices.

The recovered fits use four 125-prompt shards. The reused base fit uses the same model, corpus, and fitting settings with a two-rank FP32 reduction, so reduction-order rounding can differ. Before activation capture, all six model/lens conditions reproduced saved clean, J-lesioned, and random-control predictions. Capture forwards independently reproduced clean and J-lesioned token IDs.

Fresh-lens validation here consists of prediction parity and continued disruption of the base model. The fitting receipts retain a flag requiring the original paper's separate lens-quality validation suite. This experiment did not perform that full certification, nor fit fresh lenses for the sham and random-trained controls.

Kurtosis. For vocabulary logits $z$, I compute the population-moment excess kurtosis

$$\kappa(z) = \frac{\frac{1}{V}\sum_v (z_v - \bar z)^4} {\left[\frac{1}{V}\sum_v (z_v - \bar z)^2\right]^2} - 3.$$

The readout uses BF16 computation and FP32 central moments. Each point is the median over prompts; shading uses 2,000 country-bootstrap resamples. Intervals are pointwise. The median of paired lesioned-minus-clean differences is computed separately from the difference of the two medians.

BaseRecovered (primary)Recovered (second seed)

Excess kurtosis of J-lens vocabulary logits

At block 22, clean/lesioned medians are 2.48/0.99 for the base, 2.46/1.39 for the primary, and 2.01/1.28 for the replicate. The separation persists after refitting. At block 18 under the published lens, the replicate's median paired change is +0.087, with interval [+0.058, +0.124]; the primary's paired interval includes zero.

Intervention magnitude. Captures use final-prompt block outputs after lesion hooks. Norm ratios and cosines compare separate clean and lesioned trajectories, so they include accumulated upstream effects rather than isolating the local subtraction.

BaseRecovered (primary)Recovered (second seed)

Residual norm ratio (lesioned / clean)

Cosine similarity (clean vs. lesioned)

CKA. Let $X_\ell \in \mathbb{R}^{129 \times 2560}$ contain final-prompt residuals at block $\ell$, centered over prompts. The similarity between two layers is

$$\mathrm{CKA}(X_\ell, X_m) = \frac{\|X_\ell^\top X_m\|_F^2} {\|X_\ell^\top X_\ell\|_F\,\|X_m^\top X_m\|_F}.$$

The implementation evaluates the equivalent centered prompt-Gram expression in FP64. Each condition yields a $32 \times 32$ matrix. Difference panels subtract those matrices; they do not measure direct cross-model similarity. Both seeds use identical color scales.

D. Implementation Corrections and Earlier Experiments

The original intervention multiplied unembedding rows by the stored norm parameter $w$. Qwen3.5 uses effective gain $1 + w$; correcting this changes the selected readout directions, not just their overall scale.

A shared autocast cache between the clean no-gradient pass and the training pass could also suppress gradients for linear weights. The corrected trainer isolates these contexts. A separate hook-lifetime defect was fixed, although the original configured runs had gradient checkpointing disabled and therefore did not exercise that failure path.

After these fixes, the original learning-rate/duration combination gave 62/129 J-lesioned, 63/129 clean, and 1/49 J-lesioned transfer accuracy. Calibration tested lower rates and selected the 100-step run reported here. Because the intervention, gradient coverage, learning rate, and duration differ from the legacy run, comparing their endpoints cannot isolate any single change.

Earlier country probes, frozen-output-protection tests, width sweeps, and residual-transplantation experiments used legacy checkpoints. Those artifacts remain exploratory evidence about those checkpoints. They have not been reproduced on the corrected models, and do not establish a mechanism for the current result. In particular, decoding a country with a fitted probe would establish recoverable information, but would not show that the model's answer computation uses that information.

E. Experiments That Would Distinguish Recovery Mechanisms

• Freeze output protection. Reevaluate the recovered models using the base model's protected-token sets. Compare paired predictions to determine how much recovery depends on changes in the exemptions.
• Vary width and projection. Evaluate larger $k$ and an orthogonal-span intervention, recording residual components as well as accuracy. Increasing $k$ under sequential projection need not produce a monotonic increase in disruption.
• Test another readout. Fit R-lens to base and recovered checkpoints and compare behavioral sensitivity under equivalent selection rules. This tests estimator dependence beyond refitting J-lens.
• Probe and intervene on the intermediate. Train country probes on cities excluded from probe fitting, then combine decoding with targeted patching or concept substitution. Probe accuracy alone cannot distinguish a used intermediate from correlated information.
• Measure perturbation dependence. Sweep intervention strength for both base and recovered models and report accuracy jointly with norm loss and cosine. This would test whether the recovery gap persists at comparable measured perturbation levels.
• Evaluate new tasks and generic text. Use previously unseen task families and measure ordinary language performance under the lesion. The current controls do not establish broad selectivity or general robustness.

F. Reproducibility

The jspace-plasticity repository contains per-example predictions, activation arrays, per-prompt kurtosis statistics, bootstrap summaries, plotting code, and model/lens/source hashes. CKA and severity can be recomputed from the bundled activations; recomputing vocabulary logits requires the external checkpoints and lens matrices.

All 974 bundled file hashes and both CPU behavioral receipt audits passed during this write-up. These checks recompute saved results and verify provenance; they are not an independent training replication.