Mark Oskin
Professor of Computer Science & Engineering at the University of Washington.
Current work
My current research is on transformer interpretability and design.
Interpretability is key to integrity. Without it, AI security is left up to the agent. With it, we can say precisely why a transformer makes a prediction. It turns out that the key to interpretability is understanding the geometry of how transformers compute, so to understand my work I would start with Off-Axis, On Purpose. Once the geometry is understood, you can identify the circuits that training builds in a transformer and edit them, as the follow-on paper, Through the Looking Glass, shows. The legible-by-construction series takes the design route to the same end: build the transformer out of named, bounded operations so that its units are readable from the start, at no cost in quality.
Beyond interpretability is performance. Once transformers are no longer opaque blobs of matrix math, their function becomes clear, and with it an abundance of performance opportunities.
Papers
-
September 2026 Reading & writing
Through the Looking Glass: Directly Reading and Writing Transformers
How many of a transformer's components decide a token? Counted by absolute contribution, thousands; counted by net contribution, dozens. On a 12-layer baseline, 53 components carry ninety percent of a prediction, 13 it cannot survive losing, and 8 suffice on their own, and across twelve outside models from 124M to 7B parameters the share of the model a prediction draws on stays at one to three percent. Every component can be named on both sides, what it reads and what it writes, straight from the weights and activations with nothing trained or fitted. A name can be acted on: an association the model does not hold installs into one spare unit for a fortieth of what a rank-one update costs.
Abstract
How many of a transformer's components decide a token? Counted by the absolute value of each unit's and channel's contribution to the logit, one prediction rests on thousands to hundreds of thousands of them. But contributions are signed, and across eighteen models the mass pushing away from the predicted token is a median of seven times the mass carrying it. Divide by the net and the count is dozens: on the baseline, 53 components carry ninety percent of a prediction, 13 it cannot survive losing, and 8 suffice to produce it alone. Across twelve models trained elsewhere, 124M to 7B parameters, the sufficient set runs from two components to sixteen, and what a prediction draws on, followed all the way back, is one to three percent of the model, a share that does not grow with size. Three quarters of a layer's update is a fixed linear map of the state it received. Everything is read from the model's own parameters and activations, with nothing trained or fitted, and it names a component on both sides: what it writes, from the predictions it drives, reaching close to half of every model; what it reads, from its weights in the frame of its own layer, at 58.9 percent above chance over its eight strongest inputs. Sorting the remainder by upstream source yields grammatical categories the embedding cannot see. A name can be acted on. An association the model does not hold installs into one spare unit, key and value read from the weights, for a quarter of a percent of held-out loss, a fortieth of what a rank-one update costs. An installed attention head and a unit two layers above it make an edit fire only where a token occurred earlier in the context, and a unit the model trained for itself is driven from two layers upstream, 86 percent of the effect passing through it. An order-preserving activation puts a unit's inputs at the instrument's ceiling, at the price of a two-part install.
-
August 2026 Geometry
Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So
A transformer's answer lives on one axis, the direction its unembedding reads, and nearly all of the work happens somewhere else, on purpose. Through the first of two phases every sublayer writes into a subspace held near-orthogonal to the read-out, and moving attention's values onto that axis is 64 to 84 times more damaging than a matched random rotation: the subspace insulates composition from the vocabulary. Flattening the structure, as early-exit training does, is invisible to perplexity, LAMBADA and BLiMP while shrinking the concept workspace from about twenty-five dimensions to fourteen. The geometry can be imposed by one fixed rotation at the phase boundary, and which rotation does not matter.
Abstract
A transformer's answer lives on one axis: the direction its unembedding reads. Its intermediate states largely do not, and that off-axis position is usually treated as an obstacle to interpretation. We show it is functional. A 12-layer model computes in two phases. Through the first, every sublayer writes into a subspace held near-orthogonal to the read-out, attention 75 to 96 degrees off it at every depth. Moving attention's values onto the read-out is 64 to 84 times more damaging than a matched random rotation, and the damage is entirely in cross-token mixing: the subspace insulates composition from the vocabulary. Beneath it the frame itself turns rigidly with depth. In the second phase the answer arrives on-axis, late, and by addition rather than by turning accumulated content onto the read-out. Pressing every layer onto the read-out instead, as training for early exit does, matches the baseline on perplexity, LAMBADA and BLiMP while cutting the concept-phase workspace from about twenty-five effective dimensions to fourteen, a change none of those benchmarks register. The geometry can also be imposed, though not by asking for it. Prescribing it through the loss is a lottery: six of eight seeds collapse, because a model told to null its read-out projection obeys most cheaply by discarding dimensions. Inserting one fixed rotation at the phase boundary lands it instead, at baseline quality. A sparse rotation the surrounding weights can absorb converges on all nine seeds, against five of nine for ordinary training. Which rotation is immaterial: twenty-five runs across thirteen distinct ones reach the same quality, and two baselines from different seeds hold their concepts in near-orthogonal frames while agreeing on their read-outs. That freedom is usable: a basis drawn at random and prescribed before training is adopted across the concept phase, with quality unchanged.
-
July 2026 Legible by construction
Training, Reading, and Editing Legible Transformers
The capstone of the series: how to train these models, read them, and edit them. The obvious crispness penalty collapses operators into dead constants; a per-channel variance floor, the legibility metric itself written as a loss, is the fix. The result is 78% of feed-forward operands and 50% of attention value channels reading as crisp, contextual detectors, edits that are 50 to 184 times more local in the deep layers, and a legibility dial that trades a circuit's reuse for independence at no quality cost. Quality holds at parity with a conventional baseline throughout.
Abstract
A transformer can be built from operators that are legible by construction — bounded, named units that read as fuzzy set operations rather than dense activations — but legibility must be pressed for during training, and the pressure has a failure mode. A crispness penalty meant to sharpen a bounded operator into a decisive detector instead collapses it into a dead constant. An identity,
E[v(1−v)] = μ(1−μ) − var, shows why — the penalty is a variance-minimizer blind to the difference between a live detector and a constant — and names the fix: a per-channel variance floor, the target legibility metric written as a loss, which recovers both legibility and quality. A learned per-unit fraction then retires the hand-set reserved-GELU partition of prior work: given the choice the model keeps no unit as pure GELU and routes 87% of its load-bearing computation through crisp operators. The result is the most legible transformer we have built — 78% of its feed-forward operands and 50% of its attention value channels are crisp-and-contextual detectors, and per-head legibility rises from 18% in shallow layers to 78% in deep ones. Read in the correct rotated per-layer frame, these units separate a clean detection (what a unit responds to) from a harder naming (what its output decodes to); and because the objective makes each unit crisp and sparse, edits to them are far more local — 50–184× in the deep layers where the edit sites concentrate — and can target explicit conjunctions a single neuron cannot express. Finally, a between-unit decorrelation pressure exposes a legibility dial: it trades a circuit's reuse for independence at no quality cost, turning concepts into single, surgically editable units and a prediction into a short explanation read off a handful of named operations. Quality holds at parity with a conventional baseline throughout. -
July 2026 Legible by construction
Legible-by-Construction: Attention and End-to-End Transformers
Carries the idea into attention and joins the two halves. Passing a head's value through a sigmoid turns each value channel into a readable detector of whether a feature holds at a token, with no new parameters. The load-bearing decision is to bound what a head detects while leaving what it writes free. Across five designs at 125M parameters, 44 to 62 percent of value channels become crisp, contextually selective detectors, and coupling this attention to the legible feed-forward layer yields an end-to-end legible language model at benchmark parity.
Abstract
A companion paper showed that a transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations — intersection, set-difference, and a self-forgetting sequence quantifier — so its hidden units read as named logical operators at no cost to language-model quality. That left the other half of the transformer opaque. Here we carry the same idea into attention and join the two into one model. The mechanism is minimal: a head's value is passed through a sigmoid, so each value channel becomes a readable detector of whether a feature holds at a token. This adds no parameters and leaves the standard head otherwise untouched. A Boolean variant goes further, restructuring the value into an explicit within-token intersection and negation-capable set-difference. In both designs the output projection is left free, not tied to the vocabulary, which is the load-bearing decision: bounding what a head detects while leaving what it writes unconstrained yields selective detectors, whereas constraining the write does not. A bounded value is shaped into a readable detector by two selectivity pressures — one for sparse firing, one for decisive firing at the rails — and which a design wants is not universal. Across five specialized-attention designs at 125M parameters, 44 to 62 percent of value channels become crisp, contextually selective detectors, and their legibility rises with depth rather than crystallizing only on punctuation. Language-model quality is at parity with a conventional baseline. Finally, we couple the Boolean attention to the legible feed-forward layer and train an end-to-end legible-by-construction language model at benchmark parity: its feed-forward units are named set and quantifier operations throughout, and we can take a token it generates and read the named units that compose to produce it.
-
June 2026 Legible by construction
Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors
Where the series begins. Each feed-forward hidden unit becomes an explicit fuzzy set operation over sigmoid-bounded memberships, intersection A·B and set-difference A·(1−B), a bounded negation that gated units lack. At 125M parameters on OpenWebText it ties the GELU baseline's perplexity with every unit carrying explicit logical form. A small block of self-forgetting sequence quantifiers closes the one robust grammatical deficit, and at the semantic layers the units read, without dictionary learning, as licensing detectors: each fires on a comparative, a passive participle or a negative-polarity item and carries that memory forward to predict than, by, or nor.
Abstract
A transformer's feed-forward (FFN) sublayer materializes the distinctions attention gathers, yet gives no account of what it computes. In a parameter-neutral replacement, each hidden unit is an explicit fuzzy set operation on sigmoid-bounded [0,1] memberships: intersection A*B and set-difference A*(1−B), the latter a bounded positive negation ("A but not B") that gated/bilinear units lack — a negation-capable FFN (NC-FFN). On N-bit parity they are the most parameter-efficient reasoning basis at shallow depth; at scale (125M, OpenWebText) NC-FFN ties the GELU baseline's perplexity, every unit carrying explicit logical form. Two limits share one cause: two-operand logic localizes to layer 0 and erodes under training, and the one robust grammatical deficit concentrates in licensing and quantifiers, beyond within-token operators. We resolve both with a small block of sequence quantifiers: a soft existential and a soft proportion, each with a per-unit learned forgetting rate from a sticky init. This recovers the deficit at epoch one (halving the wider epoch-two gap), modestly leads on LAMBADA, and makes the FFN legible: the structure now holds and migrates into depth; the decay un-learns its stickiness (median half-life ~1.5 tokens; zero latch units); and at the semantic layers the units read, without dictionary learning, as grammatical licensing detectors: each fires on a licensor (a comparative, a passive participle, a negative-polarity item) and carries its memory forward to predict the licensed word (than, by, nor). This legibility is localized and free only up to a partition (a fully Boolean FFN diverges in training), but the result is a parameter-neutral, language-model-quality transformer with a readable, interpretable-by-construction grammatical mechanism — an account not just of what a feed-forward layer represents but how it licenses.