Is RoPE Just a Recency Bias?

This post is an attempt to find answers to a question I had, using Claude and Gemini Argon. The write-up, the simulations and figures were produced with Claude and Gemini and then edited by me for the blog. The question (and the intuition behind it) are mine, obviously. This is an experiment to post more actively again.

My question is whether RoPE is mostly a recency bias: the intuition is that rotating a distant key turns its match with the query into noise, so attention ends up favouring nearby tokens. I doubted models could decode much more than that from it. A semantic match would have to see past the rotation, which seemed hard, and attending to a specific relative position seemed to require undoing the rotation, for example by learning an inverse rotation with MLPs spread over several blocks.

That intuition is partly right: rotation does wash out content matches as distance grows. However, trained models do more with RoPE than simple recency weighting. They concentrate most of their content in the slowest frequencies (which rotate the least) and use the fastest frequencies to build heads that attend to specific relative offsets—which, for a fixed offset, only requires learning a fixed rotation inside the query and key projections. At the same time, RoPE may matter mainly during training: late in pretraining it can be removed altogether with almost no loss in in-domain perplexity, and whether its training-time benefit comes down to recency is still an open question.

Rotation dephases matches; it does not add noise

At head dimension \(d = 128\), RoPE splits each query and key vector into 64 2D pairs and rotates pair \(i\) at token position \(n\) by the angle \(\omega_i n\), where \(\omega_i = 10{,}000^{-2i/128}\) ranges from \(1\text{ rad/token}\) down to about \(10^{-4}\text{ rad/token}\). In the attention dot product \(q_m^\top R(-\omega \Delta) k_n\), only the relative offset \(\Delta = m - n\) remains.

Because each 2D rotation is orthogonal, it preserves norms. If a query and key are unrelated—say, independent random unit vectors—rotating the key just yields another random unit vector, so the dot product has mean \(0\) and standard deviation \(1/\sqrt{d}\) at every offset \(\Delta\) (the grey chance band in Figure 1). To see any decay with distance, we have to look at a key that actually matches the query.

Take the strongest case, \(q = k\) drawn uniformly on the unit sphere within a frequency band. At offset \(\Delta\), each pair \(i\) contributes \(\|k_i\|^2 \cos(\omega_i \Delta)\) (and for a general query–key pair, \(A_i \cos(\omega_i \Delta) + B_i \sin(\omega_i \Delta)\)). Because different pairs rotate at different rates \(\omega_i\), their contributions fall out of phase as \(\Delta\) increases, and how quickly the sum decays depends on the frequency band:

Once the pairs in a band have dephased, the dot product with an identical key has the same mean and variance across keys as the dot product with an unrelated key, so the query can no longer distinguish the two. RoPE does not inject random noise—for any fixed key, the dot product at offset \(\Delta\) is completely deterministic—but across different content vectors, dephasing washes out the match in the same way. A weaker match (\(\cos(q, k) = 0.5\)) scales each curve down by half, which reaches the chance band at most one doubling of \(\Delta\) earlier because the drop-off is steep.

Summed over all 64 pairs, this dephasing produces the smooth “long-term decay” originally used to motivate RoPE by Su et al.¹ Barbero et al.,² however, argue that this aggregate decay is not the main reason RoPE is useful in practice.

Most content lives in the slowest frequencies

When Barbero et al. inspect Gemma 7B, they find that most of the query and key norm is concentrated in the lowest RoPE frequencies, which they argue are used to carry semantic (content) information.³ Even among the low frequencies, though, only the very slowest pairs stay nearly static across a full training context. For preserving a content match \(q \approx k\), the critical threshold is a quarter turn (\(\pi/2\) radians): beyond that, \(\cos(\omega_i \Delta) < 0\), so the pair’s contribution flips sign and actively works against the match.

With head dimension 128 and base \(10{,}000\), only the slowest 9 of the 64 pairs stay under a quarter turn across a 4K context. Content placed in those 9 pairs behaves almost as if there were no positional encoding (NoPE), whereas a match spread evenly across the slowest 16 pairs already decays to the chance floor by offset 4K (Figure 1). Extending the context to 32K without changing the base eliminates those near-static channels entirely: every pair exceeds a quarter turn, and even the slowest pair reaches \(\cos(\omega_{63} \Delta) \approx -0.8\). This is why long-context models increase the RoPE base (for example, to \(500{,}000\) in Llama 3), which slows all frequencies down and restores near-static pairs.

This also supports my original suspicion that rotation gets in the way of semantic matching: models concentrate content norm in the slowest pairs specifically to avoid the rotation, and the residual rotation on those pairs still hurts. Barbero et al. test this directly with \(p\)-RoPE, which simply removes RoPE (\(\omega_i = 0\)) from the slowest fraction of pairs.⁴ In Gemma 2B trained at an 8K context, turning off rotation on the slowest 25% of pairs slightly improves validation perplexity over standard RoPE (4.44 vs. 4.46 on Wikipedia, compared to 4.86 for NoPE), whereas turning off the fastest 25% degrades it. Architectural choices like partial RoPE in GPT-NeoX and Pythia⁵ and the explicit split between unrotated content dimensions and rotated RoPE dimensions in DeepSeek’s MLA accomplish the same thing by design.

When evaluating beyond the training length, a second threshold matters: one full turn (\(2\pi\)). Once a pair has completed a full rotation within the training window, longer contexts only revisit angles in \([0, 2\pi)\) that the model has already seen during training. Pairs that complete less than one full turn over the training window (18 of 64 pairs at 4K, and 4 pairs at 32K) instead rotate into angles never seen during training. Context-extension methods like YaRN avoid these out-of-distribution angles by scaling down the slow frequencies while leaving the fast frequencies intact, though Gelberg et al. show that compressing those slow frequencies perturbs long-range semantic attention at test time.⁶

What RoPE adds that a recency bias cannot

A pure recency bias like ALiBi⁷ adds a linear penalty \(-m \Delta\) to the attention logit, so its positional term always peaks at \(\Delta = 0\) and decays monotonically with distance. Although RoPE acts multiplicatively (\(q_m^\top R(-\omega \Delta) k_n\)) rather than additively, its fast frequencies allow a head to place a sharp attention peak at a specific non-zero offset \(\Delta_0\).

Because the full logit is a sum over the 64 2D pairs, a head can turn RoPE’s multiplicative rotation into an effectively additive positional bias by partitioning the frequencies: if the projections place a token-independent “carrier” vector in the fast pairs and token-dependent content in the slowest pairs, the sum across pairs decomposes into a positional profile \(b(\Delta)\) plus a content match. In bias-free architectures like Gemma and Llama, that token-independent carrier has to be extracted from a direction in the residual stream that stays nearly constant across positions; empirically, this appears as concentrated “massive activations” in specific query and key frequency pairs (Barbero et al.; Jin et al.⁸). For a fixed target offset \(\Delta_0\), the model does not need an MLP to invert the position rotation: on the fast pairs, the query projection \(W_Q\) simply needs to equal the key projection \(W_K\) rotated by \(\omega_i \Delta_0\) in each 2D block, which is a fixed linear transformation inside the weight matrices. Barbero et al. find that Gemma 7B constructs such heads—including diagonal (\(\Delta_0 = 0\)) and previous-token (\(\Delta_0 = 1\)) heads—using its highest frequencies, primarily in the first and last layers.

There is still a notable difference in how hard it is to learn \(\Delta_0 = 0\) (or a smooth recency decay) versus a specific non-zero offset \(\Delta_0 \ge 1\). Unlike absolute positional embeddings, RoPE is never added to the residual stream or to the value vectors \(v_n\)—it is applied only to \(q_m\) and \(k_n\) inside the scalar dot product \(q_m^\top R(-\omega \Delta) k_n\) and never exposed back to the model in the forward pass. The network therefore cannot read out the RoPE phases from the activations and can only discover the rotation angles through gradient descent on \(W_Q\) and \(W_K\) via the attention weights. For \(\Delta_0 = 0\), any symmetric alignment \(W_Q \approx W_K\) on a shared subspace automatically puts all frequency pairs in phase at \(\Delta = 0\) and lets them dephase as \(\Delta\) grows (Figure 1), producing a diagonal peak or recency decay without the weights needing to know the individual frequencies \(\omega_i\). Peaking at \(\Delta_0 \ge 1\), by contrast, requires \(W_Q\) (or \(W_K\)) to learn the explicit 2D rotation by \(\omega_i \Delta_0\) for each pair via gradients. Consistent with this, Barbero et al. compare each head’s attention dot product \(q_m^\top R(-\omega \Delta_0) k_{m-\Delta_0}\) against its Cauchy–Schwarz upper bound \(\|q_m\|\|k_{m-\Delta_0}\|\)—the maximum possible logit if the query and rotated key were perfectly aligned—and find that in Gemma 7B, diagonal heads (\(\Delta_0 = 0\)) come much closer to this bound than previous-token heads (\(\Delta_0 = 1\)) do.⁹

Because previous-token heads form the first half of an induction circuit,¹⁰ the ability to attend to an exact offset matters directly for in-context copying, not just local syntax.

This also explains why stretching RoPE frequencies to extend the context window is so delicate. A head that attends to offset \(\Delta_0\) has the training angles \(\omega_i \Delta_0\) baked directly into \(W_Q\) and \(W_K\). If we rescale the frequencies at test time (\(\omega_i \mapsto \gamma_i \omega_i\)), the RoPE rotation at offset \(\Delta_0\) becomes \(\gamma_i \omega_i \Delta_0\), which no longer matches the \(\omega_i \Delta_0\) stored in the weights unless \(\gamma_i \approx 1\) on the fast pairs. Uniformly dividing all frequencies by a scale factor \(s\) (Position Interpolation) therefore scrambles local offset heads without fine-tuning; methods like NTK-aware scaling and YaRN only preserve near-token offsets by leaving the fast frequencies untouched (\(\gamma_i = 1\)) and pushing the compression onto the slow frequencies—which in turn warps the long-range semantic channels discussed above.

Where my original worry about undoing rotations does hold is when the target offset \(\Delta\) depends dynamically on content. Because RoPE is not applied to the values \(v_n\), attending to a token does not pass its RoPE phase or relative offset into the residual stream either. To attend \(\Delta\) tokens back based on the current input, the query has to construct \(\cos(\omega_i \Delta)\) and \(\sin(\omega_i \Delta)\) for every frequency pair from scratch—a Fourier representation of \(\Delta\). If \(\Delta\) is selected from a small discrete set, that can be a simple lookup, but computing it from continuous features requires an MLP, and the fastest frequencies require high precision. Other positional encodings face the same or worse difficulties: ALiBi and T5 relative biases do not depend on content at all, and with absolute sinusoidal embeddings a single attention layer cannot even multiply a content-derived offset code by its own position features, whereas RoPE’s bilinear form \((R(\omega m) q_m)^\top (R(\omega n) k_n)\) provides that interaction automatically. In practice, transformers mostly avoid computing dynamic offsets and instead use induction circuits to turn position lookups into content lookups: a fixed \(\Delta_0 = 1\) head copies the previous token’s identity forward, and a later head matches on that content.

RoPE as a training scaffold

Gelberg et al. (Sakana AI and Oxford, ICLR 2026) argue that RoPE’s primary role is to speed up optimization during pretraining.¹¹ In a NoPE transformer at random initialization, small-variance token embeddings produce nearly uniform attention weights across the prefix, which tightly bounds the gradients that could break symmetry and build positional patterns. With RoPE, the relative rotations break attention uniformity from step 0, giving non-zero positional gradients even when the input sequence is constant.

To test whether RoPE is still needed once the network has trained, their DroPE ablation trains a 500M-parameter model for 16B tokens (16K steps) and removes RoPE at different points during training, continuing with NoPE for the remaining steps:¹²

Dropping RoPE at step 14K of 16K has essentially no effect on final in-domain validation perplexity (21.73 vs. 21.72), whereas training without RoPE from the start is 2.05 perplexity worse and dropping it halfway through (step 8K) still leaves a 0.70 gap after 8K steps of recovery. This supports the view that a trained model does not fundamentally rely on RoPE for in-domain language modeling, while leaving open whether RoPE’s training-time advantage comes from exact-offset heads or simply from breaking uniformity via recency (which ALiBi also does from initialization).

In their OpenReview rebuttal, the authors report that across 100 short sequences, the attention maps of the 500M RoPE model and the recalibrated DroPE model have an average correlation of 0.68, with high agreement in semantic heads and much lower agreement in positional heads. When RoPE is removed, the content-matching heads (which lived in the near-static slow frequencies) keep working, while the positional heads break and are rebuilt during recalibration.

How does a DroPE model rebuild positional heads without positional encodings? Theoretically, a causal NoPE transformer can infer absolute and relative positions in Layer 1 from the causal mask alone and write them into the residual stream for later layers (Kazemnejad et al.¹³). Inspecting the released Llama-2-7B-DroPE checkpoint, Africa (2026)¹⁴ found that recalibration learns precisely this mechanism: Layer 1’s query and key norms jump roughly \(100\times\) (from ~45–52 in standard Llama-2-7B to ~5,500–6,600 in DroPE), increasing Layer 1 attention’s share of the residual stream norm from 0.9% to 68.8%. Meanwhile, the concentrated “massive Q/K activations” in Layers 2–31 drop by 60% and become largely vestigial—zeroing them out destroys passkey retrieval in standard Llama-2-7B (100% \(\to\) 0%) but leaves Llama-2-7B-DroPE unaffected. At random initialization, before later layers have formed circuits that use positional information, Layer 1 receives very small gradients toward this causal-mask construction (Gelberg et al.’s Theorem 3.4); once a trained network already depends on positional features, removing RoPE creates a strong gradient signal that forces Layer 1 to take over the positional bookkeeping.

Two external checks bear directly on this in-domain recovery claim. On the positive side, a 120B hybrid model combining DeltaNet and sparse attention applied DroPE-style RoPE removal mid-training and saw the same behavior: loss spiked by about 0.12 nats and recovered quickly.¹⁵ On the cautious side, Africa’s probe of 4-bit quantized Llama-2-7B-DroPE found lower accuracy than the original Llama-2-7B on passkey retrieval (60% vs. 100%, \(n = 20\)), city facts (79% vs. 89.5%), and IMDB sentiment, alongside higher accuracy on sports facts. Those evaluations used small samples and 4-bit weights, but city facts and IMDB are short-context tasks where dropping RoPE is expected to come at no cost.

Where the recency story holds

At small scale and short context, a pure recency bias recovers nearly all of RoPE’s in-domain perplexity advantage over NoPE. In 120M-parameter models trained at context length 512 (Bianchessi et al.¹⁶), ALiBi and RoPE finish within 0.3 perplexity points of each other, while NoPE trails both by 1.3 to 1.6 points:

Perplexity alone, however, cannot distinguish the two explanations: ALiBi also breaks attention uniformity from step 0, so it provides the same early optimization advantage that DroPE attributes to RoPE.

Experiments that would separate them

One clean way to separate recency from exact-offset capability is to track when induction heads form during training in RoPE, ALiBi, and NoPE models at matched compute. Because an induction circuit relies on a previous-token head (\(\Delta_0 = 1\)):

• If RoPE’s main advantage during training is simply a recency bias (breaking uniformity in favor of nearby tokens), ALiBi should form induction heads just as early as RoPE.
• If being able to express a sharp peak at \(\Delta_0 = 1\) in a single layer is what matters, RoPE should form induction heads significantly earlier than ALiBi, with ALiBi closer to NoPE.

A minimal version of this test is to train a two-layer transformer on repeated random token sequences and record the training step at which in-context copying loss drops. A complementary ablation can reuse DroPE on a pretrained model: Barbero et al.’s \(p\)-RoPE already shows that training from scratch without the slowest 25% of pairs slightly helps perplexity while removing the fastest 25% hurts; repeating that selective removal late in training and measuring in-context copying after recalibration would show how much the trained model still relies on each band. (Incidentally, as a DroPE reviewer noted on OpenReview, applying YaRN¹⁷ with a large scale factor \(s\) and recalibrating at the original context length is nearly this exact experiment: for large \(s\), YaRN scales the slow frequencies down by \(1/s \to 0\) while leaving the fast frequencies at \(\gamma_i = 1\), which reduces to late-stage \(p\)-RoPE.)

Two further checks target the training-scaffold hypothesis directly:

1. Swap the scaffold: train from scratch with ALiBi instead of RoPE, drop the ALiBi bias at step 14K of 16K, and compare final NoPE perplexity to DroPE. If ALiBi works just as well as a removable scaffold, then RoPE’s training-time benefit is nothing more than what a linear recency bias provides.
2. Inspect early checkpoints without training anything: because any symmetric \(W_Q \approx W_K\) alignment produces a \(\Delta = 0\) peak and smooth recency decay automatically, whereas a \(\Delta_0 \ge 1\) peak requires learning each pair’s 2D rotation, recency heads should emerge earlier than off-diagonal heads in a RoPE model. Pythia publishes log-spaced early checkpoints (steps 0, 1, 2, 4, …, 512, then every 1,000), so one can directly measure each head’s attention profile by offset over early training to see whether smooth recency precedes sharp off-diagonal heads.

Conclusion

So is RoPE just a recency bias? Partly: content in its fast and middle frequency pairs does dephase with distance, and at small scale a linear recency bias like ALiBi recovers almost all of RoPE’s perplexity gain over NoPE. At the same time, trained RoPE models clearly use it for more than recency by splitting its frequencies into two regimes: Gemma 7B packs most of its query and key norm into the slowest pairs to keep content matches from rotating (and turning rotation off on those pairs with \(p\)-RoPE slightly improves perplexity), while using a token-independent carrier on the fastest pairs to build exact-offset heads like previous-token heads, which a monotonic recency bias cannot express in a single layer.

Even so, the DroPE result suggests that RoPE’s most important role may be as a training-time scaffold: late in pretraining, RoPE can be removed with virtually no in-domain perplexity cost as Layer 1 attention takes over the positional bookkeeping through the causal mask. That is the sense in which my original skepticism holds up best—nothing so far rules out that RoPE’s main value during training is simply a recency bias that breaks initial symmetry, and the experiments above would be a straightforward way to test that.