Circuit-Tracing MedGemma

1Introduction#

Medical vision–language models are being adapted for radiology, and with them come the faithfulness failures the field has long documented in medical imaging: models that assert findings without visual support, and that exploit acquisition shortcuts: scanner, view-position, or device cues standing in for pathology.¹

Behavioral benchmarks can detect these failures but not locate them. Understanding (and eventually fixing) them calls for interpretability tools that reach inside the model. For generative medical VLMs, those tools barely exist: sparse dictionaries have been trained on single medical models at the vision-encoder level,⁶ and circuit tracing with transcoders and attribution graphs has been shown on general Gemma-class VLMs in concurrent work,⁵ but with per-layer transcoders, and outside medicine. No prior work, to our knowledge, trains a cross-layer transcoder that reads image tokens inside a medical VLM, nor builds an attribution graph for one. We do both, and use the resulting microscope to ask what the attribution graphs reveal about MedGemma’s computation on a chest X-ray.

We find three things. Within the frozen-attention attribution model, decision-related attribution concentrates in the text/prompt prior rather than the image (§6). The decision resists localization to a single clampable feature, consistent with a distributed representation (§7). And a real clinical shortcut is localized to the attention pattern, outside the reach of feature-level attribution (§8).

2Background: transcoders and attribution graphs#

Our tools come from the sparse-dictionary and circuit-tracing line of mechanistic interpretability; we summarize them here for readers new to the area, and use them without claiming them.

Sparse features. A sparse autoencoder decomposes a model's activation vector into a large dictionary of features of which only a few are active per token. Trained with a sparsity constraint, the features tend to be monosemantic, each corresponds to a single human-interpretable concept.⁴ A transcoder is a variant that, rather than reconstructing one activation, reads the input to a component (here a decoder MLP) and predicts its output: a sparse, interpretable stand-in for the MLP's computation.

Cross-layer transcoders. In a cross-layer transcoder, each feature's decoder writes not just to its own layer but to all downstream layers, capturing structure that spans layers.² This is the substrate the attribution-graph method is built on. We use a Top-K activation (K = 32, 2048 features per layer), which fixes the number of active features per token and yields monosemantic units.

The local replacement model. To trace why a model produced an output, the circuit-tracing method³ substitutes the transcoder for the model's MLPs and adds a frozen per-position error term, yielding a replacement model that reproduces the original logits but computes through interpretable features. Freezing the attention patterns and normalization scales at the values they take on a specific input makes the path from any feature to any downstream feature or logit linear, so attribution edges become exact and additive: the contributions of all features sum to the logit. The result is an attribution graph from inputs, through features, to the output. A known cost, central to §8, is that freezing the attention pattern makes the method blind to computations carried by the pattern itself (“QK circuits”).

3The instrument: a cross-layer transcoder for MedGemma#

We train a cross-layer transcoder over all 34 layers of MedGemma,⁷ on activations captured on-the-fly as the model runs over chest X-rays and reports, so the dictionary sees both image and text tokens. The property that makes it a usable instrument is faithfulness: if we substitute the transcoder's reconstruction for every MLP and drop the error term, does the model still behave the same? On image tokens (the substrate for finding decisions), the substituted model reproduces the original top-1 next token 100% of the time (n = 60 held-out image/report pairs, KL 0.044). Within this replacement model, then, the computation turning image evidence into a finding runs through features we can read and manipulate. A span-ablation confirms the cross-layer reach is used: restricting each feature to short-range writes more than doubles image reconstruction error.

4The features encode real radiology#

A faithful dictionary is only useful if its features mean something. Labeling the top features by their top-activating image patches returns concrete radiological concepts, not noise: costophrenic angle / lung base, mediastinal contour, tubes and catheters, shoulder and spine landmarks, alongside acquisition artifacts. In a late layer, 25 of 30 top features receive a specific anatomical or acquisition label (a mid-layer is sparser, 10 of 18). Contrasting finding-positive against clean images yields finding-selective features (pleural effusion AUC 0.86, pneumothorax 0.88) that localize to the expected anatomy.

Costophrenic angle / lung baseL30 · f1045

Mediastinal contour & paraspinal regionL30 · f1434

Tubes & catheters crossing the chestL30 · f1439

Clavicle & shoulder regionL30 · f1344

Spine & paraspinal regionL30 · f1827

“AP / AP PORTABLE” technique text L24 · f565 · acquisition cue

5Circuit tracing: frozen-attention attribution#

A naive attribution (activation × gradient of a finding logit through the model with attention and layernorm live) is a first-order approximation whose edge weights do not conserve, and which yields only a single hop. We instead build the local replacement model (§2) and freeze the attention patterns and RMSNorm denominators. With those nonlinearities held constant the model is exactly linear in the features, so edges are exact and additive and the full multi-hop feature→feature graph becomes available.

We validate the linearization directly. The substituted-and-frozen model reproduces the original logit exactly; scaling the features traces a perfectly affine response (deviation 0.09 vs. 11.2 for the un-frozen model, a ~120× reduction); and the feature edge weights sum to the logit change to numerical tolerance. Two caveats shape how we read the graphs. First, the first-token attention sink carries large, generic attribution and must be excluded. Second (the crux of §8), freezing the attention pattern makes the method blind to effects that live in the pattern itself.

6Decision-related attribution concentrates in the text stream#

We trace the logit of the answer token (“Yes”) and read the graph one hop back: the features that directly write to the decision sit at text positions. This single-hop view has two well-known pitfalls, both of which we control for. The highest-attribution features are attention-sink / structural tokens, which we exclude; and a text-position feature may have read the image one hop earlier, via attention. Tracing each decision feature's preactivation to its own sources, the image is a small and variable minority contributor: across findings the mean image-source share runs from ≈ 3% (effusion) to ≈ 18% (genuine cardiomegaly), typically around 10%.

We deliberately do not headline a single ratio, the attribution remains partly entangled with residual structural signal, so cleanly isolating the semantic finding circuit is an open problem (§10). What is robust is the direction: within this attribution model MedGemma's finding decision draws primarily on the text/prompt stream, with image evidence a secondary, attention-mediated contribution. This is consistent with the model's behavior, which needs no attribution to interpret, it hallucinates findings with no image at all (rate ≈ 1.0) and continues to assert a finding after its supporting region is occluded (persistence 0.49).¹⁰

7The finding decision resists localization to a clampable feature#

A natural hope, and the premise of much feature-steering work, is that a finding corresponds to a feature you can turn off. It does not. Across five independent methods of choosing which features to clamp (ranking by contrastive selectivity, by gradient attribution, by image-contrast and prompt-contrast differencing, and by a negative-steering scale sweep), clamping the candidate features fails to controllably steer the finding decision. When suppression does move the probability, it does so with a loss of output coherence rather than by cleanly flipping the answer; a positive control confirms the intervention machinery works, so the null appears to be a property of the model, not a broken pipeline.

The sharpest form of the test uses the attribution graph against itself: take the very features the graph ranks as driving a finding’s logit and clamp those, against a random-feature control and a coherence check (did the edit flip the answer, or just break the model into out-of-distribution garbage?). The result is fragile and finding-dependent, not a clean lever. It is most telling on the headline no-image cardiomegaly hallucination, verified in the reason-then-probe framing where it actually fires: clamping the graph’s own top text features does not suppress the false finding at all, it nudges it slightly up.

We read this as evidence of distributed representation: across these methods, no small set of clampable features controllably governs the finding decision, which matters for safety (feature steering is not a reliable control here), and sets up §8: if the decision is not a sparse feature set, where does a specific, reproducible shortcut live?

8A clinical shortcut localized to the attention pattern#

MedGemma exhibits a textbook acquisition shortcut: on cardiomegaly-negative chest X-rays it answers “cardiomegaly: yes” 73% of the time on AP / portable films and 0% on PA / standard films. The correlation is not mysterious. Portable (AP) films are shot at the bedside for acutely ill, immobile patients, a group with a much higher prevalence of genuine heart disease; the AP projection also magnifies the cardiac silhouette. So across a training set “this is a portable film” becomes a strong statistical proxy for cardiomegaly, and a model can lean on that proxy instead of the heart itself. It is a clean, reproducible, clinically consequential bias, and an ideal test of whether our microscope can find and explain a real failure.

The feature-level tools locate the cue but not the cause. Contrasting AP against PA on cardiomegaly-negative images yields cleanly view-selective features (top AUC 0.97), the model plainly encodes “this is a portable film.” Yet clamping those features does nothing to the decision (a random-feature control is identical), and the attribution of §6 shows the decision running through generic text features that do not encode view position. The view is represented, the decision is elsewhere, and clamping the representation is inert, a genuine puzzle.

The resolution points to the attention pattern rather than the features: to where the model attends. We test this directly: run a portable-film image but replace every attention layer's pattern with the pattern from a standard-film image, keeping the portable film's own content.

P = 0.98  →  0.00

The false positive collapses, 0.62 → 0.00 (mean over 8 held-out AP/PA pairs); the reverse (standard-film content, portable-film attention) induces it, 0.00 → 0.17. This is consistent with an attention-routing (QK) effect: the film type changes what the model attends to, shifting the finding independent of detectable feature-level changes.

This carries a two-fold point. First, a clean mechanistic account of a real clinical bias, with a causal handle that removes it. Second, and more general: frozen-attention attribution (the basis of attribution graphs, ours included) is blind to this by construction. A shortcut that is the attention pattern leaves no trace in the feature→logit graph. We surface this as a documented boundary of the method: some faithfulness failures are attention-pattern effects, outside feature-level attribution, and identifying them requires an attention-pattern intervention of the kind we use here. You can step through the eight AP/PA pairs live in the interactive explorer.

9Related work#

Sparse dictionaries and circuit tracing. Our tools build on SAEs and monosemantic features, transcoders and cross-layer transcoders, and the attribution-graph method; we apply these to a new setting rather than extend the method itself.

Interpretability of vision–language models. SAEs have been applied to vision encoders, and concurrent 2026 work traces circuits in general Gemma-class VLMs.⁵ Those use per-layer transcoders on general models; we use a cross-layer transcoder on a medical VLM.

Interpretability of medical models. Medical sparse-dictionary work decomposes a single model at the vision-encoder level;⁶ none traces the decision circuit of a generative medical VLM, which is our object.

Shortcut learning in medical imaging. That CXR models exploit hospital, view-position, and device confounds is well documented.¹ Feature-level removal of such shortcuts has been shown for language and natural-image benchmarks;⁸ we instead give a mechanistic account of a clinical shortcut and show it lives in the attention pattern rather than in clampable features.

10Discussion and limitations#

The picture, within frozen-attention attribution, is of a medical VLM whose chest-X-ray findings attribute more to its text/report prior and to acquisition cues than to localized image evidence, an unfaithfulness that behavioral benchmarks see but cannot attribute, and that our microscope localizes. We flag the limitations candidly.

Quantifying the image contribution is method-sensitive. Our multi-hop estimate (§6) is small but noisy and remains partly entangled with attention-sink / structural positions we exclude only heuristically; we report the direction within the frozen-attention model and decline to headline a ratio. The QK blind spot is fundamental: frozen-attention attribution cannot see shortcuts carried by the attention pattern; QK-aware attribution is future work. Faithfulness is scoped to the image→finding computation: the transcoder reproduces the model on image tokens (100%) but not free-form text generation (8% agreement). Scale and generality: a single model, on public NIH/CheXpert data, with no external out-of-distribution validation yet, and the attention-pattern intervention is a small n; a second model and external data are the clearest next steps.

Future work. The most compelling extension is cross-model: do independently trained medical VLMs converge on the same interpretable features, and does the clinical-vs-shortcut distinction predict which features are shared? That turns this single-model microscope into a test of feature-level representational universality in medicine.⁹