Is RoPE Just a Recency Bias?

My question is whether RoPE is mostly a recency bias: the intuition is that rotating a distant key turns its match with the query into noise, so attention ends up favouring nearby tokens. I doubted models could decode much more than that from it. A semantic match would have to see past the rotation, which seemed hard, and attending to a specific relative position seemed to require undoing the rotation, for example by learning an inverse rotation with MLPs spread over several blocks.

This post is an attempt to find answers to a question I had, using Claude and Gemini Argon. The write-up, the simulations and figures were produced with Claude and Gemini and then edited by me for the blog. The question (and the intuition behind it) are mine, obviously. This is an experiment to post more actively again.

That intuition is partly right: rotation does wash out content matches as distance grows. However, trained models do more with RoPE than recency weighting. They concentrate most of their content in the slowest frequencies, which barely rotate, and use the fastest frequencies to build heads that attend to specific relative offsets—which, for a fixed offset, only requires learning a fixed rotation inside the query and key projections, not an MLP. At the same time, RoPE may matter mainly during training: late in pretraining it can be removed and the model recalibrated with almost no loss in in-domain perplexity, and whether its training-time benefit comes down to recency is still open.

Rotation dephases matches

At head dimension \(d = 128\), RoPE splits each query and key vector into 64 2D pairs and rotates pair \(i\) at token position \(n\) by the angle \(\omega_i n\), where \(\omega_i = 10{,}000^{-2i/128}\) ranges from \(1\text{ rad/token}\) down to about \(10^{-4}\text{ rad/token}\). In the attention dot product \(q_m^\top R(-\omega \Delta) k_n\), only the relative offset \(\Delta = m - n\) remains.

Because each 2D rotation is orthogonal, it preserves norms. If a query and key are unrelated—say, independent random unit vectors in \(D\) dimensions—rotating the key just yields another random unit vector, so the dot product has mean \(0\) and standard deviation \(1/\sqrt{D}\) at every offset (the grey chance band in Figure 1, with \(D\) the dimension of each band). To see any decay with distance, we have to look at a key that actually matches the query.

Take the strongest case, \(q = k\) drawn uniformly on the unit sphere within a frequency band. At offset \(\Delta\), each pair \(i\) contributes \(\|k_i\|^2 \cos(\omega_i \Delta)\). Because different pairs rotate at different rates, their contributions fall out of phase as \(\Delta\) grows, and how quickly the sum decays depends on the band:

The decay is deterministic: for a fixed key, the dot product at offset \(\Delta\) is fixed, and its average over keys is just the band average of \(\cos(\omega_i \Delta)\). Once a band has dephased, that average oscillates around zero at roughly the scale of chance rather than settling at zero (the fast band still swings to \(-0.31\) at offset 128), and the spread across keys is comparable to that of an unrelated key, so a matching key looks much like an unrelated one. A weaker match (\(\cos(q, k) = 0.5\)) scales each curve by half and reaches the chance band at most one doubling of \(\Delta\) earlier, because the drop-off is so steep.

Summed over all 64 pairs, this dephasing produces the smooth “long-term decay” originally used to motivate RoPE by Su et al.¹ Barbero et al.,² however, argue that this aggregate decay is not the main reason RoPE is useful in practice.

Most content lives in the slowest frequencies

When Barbero et al. inspect Gemma 7B, they find that most of the query and key norm is concentrated in the lowest RoPE frequencies, which they argue carry semantic (content) information.³ Even among the low frequencies, though, only the very slowest pairs stay nearly static across a full training context. For preserving a content match \(q \approx k\), the critical threshold is a quarter turn (\(\pi/2\) radians): beyond that, \(\cos(\omega_i \Delta) < 0\), so the pair’s contribution flips sign and works against the match.

With head dimension 128 and base \(10{,}000\), only the slowest 9 of the 64 pairs stay under a quarter turn across a 4K context. A match spread evenly across those 9 pairs keeps 89% of its strength at offset 2K and 59% at 4K (83% at 4K across the slowest 4), whereas across the slowest 16 it decays to chance by offset 4K (Figure 1). At 32K with the same base, no pair stays under a quarter turn, and even the slowest reaches \(\cos(\omega_{63} \Delta) \approx -0.8\). This is why long-context models raise the RoPE base (to \(500{,}000\) in Llama 2 Long and Llama 3, for example),⁴ slowing every frequency to restore near-static pairs.

This fits my original suspicion that rotation gets in the way of semantic matching: models put their content where rotation is smallest. Barbero et al. test it with \(p\)-RoPE, which removes rotation (\(\omega_i = 0\)) from a fraction of pairs, training Gemma 2B from scratch at an 8K context.⁵ Unrotating the slowest 25% costs nothing and, if anything, helps marginally (Wikipedia / FlanV2 perplexity 4.44 / 6.44, vs. 4.46 / 6.44 for RoPE and 4.86 / 6.64 for NoPE). Surprisingly, unrotating the fastest 25% barely matters either (4.46 / 6.47); only unrotating the slowest 75% (4.53 / 6.51) or everything clearly hurts. Partial RoPE in GPT-NeoX and Pythia,⁶ which rotates only a quarter of each head’s dimensions, and DeepSeek’s MLA⁷ (whose unrotated content keys can be compressed into a low-rank KV cache) build the same separation in by design.

Beyond the training length, a second threshold matters: one full turn (\(2\pi\)). A pair that completes a full turn within the training window only revisits angles it has already seen at longer contexts (even if the combinations across pairs are new). Pairs that complete less than a full turn (18 of 64 at a 4K training context; 4 at 32K) rotate into angles never seen in training. Context-extension methods rescale the slow frequencies for two overlapping reasons: YaRN interpolates exactly the pairs that turn less than once, so that none extrapolates to unseen angles,⁸ and compressing the slowest pairs keeps content channels as static over the longer window as they were in training. The second is the stated motivation for raising the base to \(500{,}000\) in Llama 2 Long: to reduce RoPE’s decay of attention to distant tokens.⁹

What RoPE adds that a recency bias cannot

A pure recency bias like ALiBi¹⁰ adds a linear penalty \(-\lambda \Delta\) (a fixed slope \(\lambda\) per head) to the attention logit, so its positional term always peaks at \(\Delta = 0\) and decays monotonically. RoPE acts multiplicatively, but because the logit is a sum over pairs, a head can partition the frequencies: if the projections put a nearly token-independent “carrier” vector in the fast pairs and content in the slowest pairs, the carrier pairs’ coefficients \(A_i, B_i\) are fixed (see footnote¹¹ for details), and the logit splits into a content match plus a positional profile \(\beta(\Delta) = \sum_i \big(A_i \cos(\omega_i \Delta) + B_i \sin(\omega_i \Delta)\big)\) over the carrier pairs.¹²

Symmetric weights (\(W_Q \approx W_K\) on the carrier, so \(B_i = 0\) and \(A_i \ge 0\); see footnote¹³) peak this profile at \(\Delta = 0\). On fast pairs that gives a sharp diagonal head; a smooth monotone decay comes from slower carrier pairs that turn less than half a turn across the context. To peak at \(\Delta_0 \ge 1\) instead, no MLP is needed to invert the rotation: on the fast pairs, \(W_Q\) just has to equal \(W_K\) rotated by \(\omega_i \Delta_0\) in each 2D block, a fixed linear map. Barbero et al. find that Gemma 7B builds such purely positional heads—including diagonal (\(\Delta_0 = 0\)) and previous-token (\(\Delta_0 = 1\)) heads, the first half of an induction circuit¹⁴—from its highest frequencies, mostly in the first and last layers.

Learning \(\Delta_0 = 0\), or a smooth recency decay, should still be easier than learning \(\Delta_0 \ge 1\). RoPE is applied only to \(q_m\) and \(k_n\) inside the logit, so unlike absolute position embeddings it never enters the residual stream: neither the embeddings nor the values \(v_n\) are rotated. The phases \(\omega_i n\) never appear as activations the network could compute with—only their effects do, through the attention pattern—so whatever the model knows about the frequencies must be stored in its weights, learned from the gradient through the attention logits. That gradient does carry the rotation (\(\partial (q^\top R(-\omega \Delta_0) k) / \partial q = R(-\omega \Delta_0) k\)), so a fixed offset is learnable, but the two targets differ in how specific they are. For \(\Delta_0 = 0\), any symmetric alignment \(W_Q \approx W_K\) on a shared subspace puts all pairs in phase at \(\Delta = 0\) (Figure 1), without matching the individual \(\omega_i\) or even knowing how coordinates are paired. A peak at \(\Delta_0 \ge 1\) needs \(W_Q^\top W_K\) to contain the rotation by \(\omega_i \Delta_0\) in every frequency block. Consistent with this, the logits of diagonal heads in Gemma 7B get much closer to their Cauchy–Schwarz bound \(\|q_m\|\|k_{m-\Delta_0}\|\) than those of previous-token heads do.¹⁵

Two consequences follow:

• Why frequency scaling is delicate: A head tuned to offset \(\Delta_0\) has the angles \(\omega_i \Delta_0\) baked into its weights, and rescaling \(\omega_i \mapsto \gamma_i \omega_i\) at test time shifts them to \(\gamma_i \omega_i \Delta_0\). Uniform Position Interpolation (\(\gamma_i = 1/s\))¹⁶ therefore perturbs local offset heads unless the model is fine-tuned. NTK-aware scaling and YaRN¹⁷ spare them by keeping \(\gamma_i \approx 1\) on the fast pairs and compressing only the slow pairs to keep their angles within the training range—which, Gelberg et al. argue, warps long-range semantic attention instead.¹⁸
• Why dynamic offsets are hard—and where my original worry holds: Because the phases never reach the residual stream, a query that wants to attend \(\Delta(x)\) tokens back based on content must synthesize \(\cos(\omega_i \Delta(x))\) and \(\sin(\omega_i \Delta(x))\) from weights that have memorized the \(\omega_i\): a linear lookup for a small discrete set of offsets, an MLP with high precision on the fast pairs for a continuous one. (Absolute encodings are no easier: a single attention layer cannot multiply a content-derived offset code by its own position features, which RoPE’s bilinear form does automatically.) In practice, transformers mostly sidestep this with induction circuits: a fixed \(\Delta_0 = 1\) head copies each token’s neighbour forward, and a later head matches on that content.

Where the recency story holds

Despite RoPE’s ability to express exact-offset heads, a pure recency bias recovers most of its in-domain perplexity advantage over NoPE at small scale and short context. In 120M-parameter models trained at context length 512 (Bianchessi et al.¹⁹), ALiBi and RoPE finish within 0.3 perplexity of each other, while NoPE trails both by 1.3 to 1.6:

Short-context perplexity cannot separate the recency and offset-head explanations, though, because ALiBi and RoPE both break attention uniformity from step 0. Beyond the training length, neither helps retrieval: at \(2\times\) the training context, Gelberg et al.’s 500M RoPE model scores 0.0 / 0.0 / 0.0 on RULER’s multi-query, multi-key and multi-value tasks and their ALiBi model 5.2 / 0.0 / 1.1, against 9.2 / 36.2 / 21.4 for NoPE and 28.0 / 41.6 / 23.3 for DroPE.²⁰ The authors attribute ALiBi’s failure to its penalty downweighting distant semantic matches.

RoPE as a training scaffold

Gelberg et al. (Sakana AI and Oxford, ICLR 2026) argue that RoPE’s main role is to speed up optimization during pretraining.²¹ Their Theorem 3.4 shows that in a causal NoPE transformer initialized with small-variance i.i.d. token embeddings, attention stays nearly uniform across arbitrary sequences and the gradients that would build positional structure stay tightly bounded across depth. With RoPE, the relative rotations break that uniformity from step 0.

To test whether RoPE is still needed once a network has trained, their DroPE ablation trains a 500M-parameter model for 16B FineWeb tokens (16K steps at context length 1,024) and removes RoPE at different points during training, continuing with NoPE for the remaining steps:²²

Dropping RoPE at step 14K of 16K has essentially no effect on final in-domain validation perplexity (21.73 vs. 21.72), so the finished model does not need RoPE for short-context language modeling. Dropping it halfway (step 8K), however, still leaves a 0.70 gap after 8K steps of recovery, which suggests RoPE keeps helping optimization through most of pretraining rather than only breaking symmetry at initialization.

Gelberg et al. also apply DroPE to pretrained SmolLM models and Llama-2-7B, recalibrating at the original context length, and beat zero-shot YaRN and NTK scaling on long-context benchmarks.²³ Recovery is slower at that scale: SmolLM-360M (pretrained on 600B tokens) regains 91.9% of its in-context performance after 5B recalibration tokens and needs 60B to match it, and there is no baseline that spends those tokens continuing with RoPE.²⁴

What changes during recalibration? In the 500M model, removing RoPE at step 14K makes the average gradient norm jump \(45\times\) (from \(0.18\) to \(8.19\) over the next 500 steps) before it settles at \(0.15\).²⁵ On 100 random 32-token sequences, the attention maps before and after correlate at \(r = 0.68\), staying similar in semantic heads and diverging in positional heads, which the authors say must now infer token order from the BOS token and cues like capitalization. A causal NoPE transformer can also recover position from the causal mask alone, starting in Layer 1 (Kazemnejad et al.²⁶).

Two external reports are consistent with this, but weak. Shravan (2026)²⁷ applied DroPE to a 9B hybrid DeltaNet and sparse-attention model before learning-rate annealing and saw a brief \(0.12\)-nat loss spike, though that backbone also carries position through its recurrence and position-basis embeddings. Africa’s (2026)²⁸ exploratory probe of 4-bit Llama-2-7B and Llama-2-7B-DroPE finds Layer 1 query and key norms over \(100\times\) larger after recalibration and ~60% fewer massive Q/K values in Layers 2–31, whose removal raises perplexity \(15\times\) for the DroPE model versus \(1{,}160\times\) for the original. That fits Layer 1 taking over positional bookkeeping, but the evaluations are noisy (IMDB accuracy ranges from 9% to 50% on a binary task) and the DroPE checkpoint was recalibrated on different data than Llama 2’s, so I read it as suggestive only.

Experiments that would separate them

Several experiments would separate the recency and exact-offset explanations:

1. Track when induction and offset heads form under RoPE, ALiBi and NoPE: Train a two-layer transformer on repeated random token sequences and record the step at which in-context copying loss drops, for standard induction (\(\Delta_0 = 1\)) and a skip-\(k\) variant (\(\Delta_0 \ge 2\), where ALiBi cannot place a peak in a single layer even if a token’s match with itself is suppressed). If RoPE’s training advantage is mainly recency, ALiBi should form standard induction heads as early as RoPE; if single-layer offset heads matter, RoPE should form both earlier.
2. Remove frequency bands from scratch and after training, at matched fractions: From scratch, Gemma 2B barely notices losing rotation on either its slowest or its fastest quarter of pairs. Post-training, the fast pairs matter more: when Ji et al. (MHA2MLA)²⁹ keep RoPE on only a small subset of pairs (1/8 by default) in pretrained SmolLM models and fine-tune, keeping the fastest pairs costs less accuracy than keeping the slowest (\(-0.85\%\) vs. \(-6.49\%\) at 135M; \(-0.76\%\) vs. \(-1.21\%\) at 1.7B). There is no from-scratch counterpart at that fraction (\(p\)-RoPE unrotated at most the fastest 25%), so the clean test removes the same bands at the same fraction in both settings. A late-stage variant is cheap: reviewer iW8i asked the DroPE authors to recalibrate under YaRN with \(s = 32\), which they did not run; at large \(s\), YaRN nearly stops the slow pairs (\(\gamma_i = 1/s\)) while leaving the fast ones untouched, which amounts to late-stage \(p\)-RoPE.
3. Swap the training scaffold across drop points: Gelberg et al. already train a 500M ALiBi baseline, so one can drop the ALiBi bias at steps 8K and 14K (not only 14K, where recovery is near the ceiling) and compare the NoPE recovery to the table above. If ALiBi matches RoPE at both drop points, a monotone recency bias is enough as a training scaffold.
4. Inspect early checkpoints without training anything: If symmetric alignment is easier to learn than per-block rotations, smooth recency and diagonal profiles should appear earlier in training than sharp off-diagonal heads. Pythia³⁰ publishes log-spaced early checkpoints (steps 0, 1, 2, 4, …, 512, then every 1,000), so each head’s attention profile by offset can be measured directly over early training, with the caveat that Pythia rotates only 25% of each head’s dimensions.

Conclusion

So is RoPE just a recency bias? For short-context perplexity, a surprising amount of its effect looks like one: rotation dephases content matches with distance, ALiBi matches RoPE’s perplexity at 120M scale, and training Gemma 2B without rotation on its fastest quarter of pairs barely moves perplexity. Mechanistically, though, trained RoPE models use the spectrum in two regimes rather than as a uniform decay: they pack content into the slowest pairs, where rotation is smallest, and use the fastest pairs for sharp offset heads, such as previous-token heads, that a monotone recency bias cannot express in a single layer—and pretrained models lose more when stripped of rotation on their fast pairs than on their slow ones.

DroPE shows that these circuits do not have to stay tied to RoPE: late in pretraining, it can be removed and the model recalibrated at the original context length, presumably recovering position from the causal mask and boundary tokens. That is the sense in which my original skepticism holds up best: RoPE’s lasting value may be as a training scaffold, one that keeps helping for most of training given the gap left by dropping it at step 8K. Whether that help comes from exact-offset heads or from a recency bias that keeps attention focused is what the experiments above would test.