Are there alternatives to the transformer?

Why long texts are hard for transformers

A transformer compares every word with every earlier word. Double the text, and that work grows four times.

It also keeps notes on every word it has read, and the notes grow with the text. In the diagram above, Meta’s Llama 3.1 8B needs about 17 GB of notes for one conversation of 128,000 tokens.

A running summary

Older designs, called recurrent networks, read one word at a time and carry a running summary that they update as they read. In 2017 the transformer took over, partly because it trains on a whole text at once, which is much faster.

State space models bring the running summary back in a new form. Mamba, from 2023, decides what to keep and what to forget as it reads. Its memory stays the same size however long the text gets, and its authors report about five times a transformer’s speed when writing.

The weak spot, and the mix

A summary of a fixed size cannot hold every detail. Researchers found that transformers are far better at copying text or finding one exact fact in a long input.

So many new models mix both: mostly state space or similar layers, plus a few transformer layers for exact recall. IBM’s Granite 4.0-H and NVIDIA’s Nemotron 3 models work this way.

Not every maker agrees. MiniMax went back to full attention for its M2 model.

Two costs of attention

Self-attention compares every token with every other one, so its computing cost grows with the square of the input length. A transformer also stores a key-value cache, notes on every token, which grows as the text gets longer.

AI21’s own table gives a bigger example. Llama 2 7B keeps four times as many notes per token as Llama 3.1 8B, because it shares no key-value heads. At 256K tokens, the table gives it a 128 GB cache, against 4 GB for AI21’s hybrid, Jamba.

Designs with a fixed-size state

State space models carry a fixed-size state forward. S4 (2021) solved a long-range test with sequences of 16,000 steps. Mamba (2023) made the state update depend on the input, so the model can choose what to keep; its authors report about 5 times a transformer’s speed when writing.

Other routes lead to the same place. Linear attention rewrites attention so its cost grows linearly, which turns a transformer into a kind of recurrent network. RWKV and xLSTM are recurrent networks redesigned to train in parallel, like a transformer.

Where they fall short

A fixed-size state limits exact recall. Jelassi and others (2024) found that transformers far outperform state space models at copying and retrieving from the input. Another study traced about 82% of one quality gap to recalling earlier context.

At 8 billion parameters, an NVIDIA-led study found pure Mamba behind transformers on some tasks, such as phonebook-style lookup. A hybrid with a few attention layers beat the transformer on all 12 standard tasks it tested.

Hybrids in today’s releases

Most new models of this kind are hybrids. AI21’s Jamba (2024) uses one attention layer for every seven Mamba layers, and IBM’s Granite 4.0-H uses nine Mamba-2 layers for every transformer layer. Alibaba’s Qwen3.5 and NVIDIA’s Nemotron 3 models are hybrids too.

Not everyone is convinced. MiniMax dropped its hybrid design for M2, reporting “clear deficits” in multi-step reasoning at larger scale. The layer designs of today’s top closed models are not published.

Our stories touch this without naming it: Salesforce’s Koa is built on Nemotron 3 Super, and Cloudflare’s Clef-flash on Qwen3.5-9B, both hybrids according to their makers.

State space models

S4 (Gu, Goel and Ré, 2021) used structured state spaces and solved the Path-X task of Long Range Arena, at length 16,000. Mamba (Gu and Dao, 2023) made the state update input-dependent, or selective, with cost linear in length.

The Mamba authors report about 5 times the inference throughput of transformers. Mamba-2 (Dao and Gu, 2024) showed that state space models and attention are mathematically related, and its core layer runs 2 to 8 times faster than Mamba’s.

Mamba-3’s authors (2026) still note that many linear models trade quality for speed and fail at tasks such as state tracking. Merrill and others (2024) argue that state space models, like transformers, cannot express simple state tracking, so a recurrent state is no automatic advantage there.

Linear attention and new recurrent networks

Katharopoulos and others (2020) rewrote attention so that its cost grows linearly, showing that a transformer can run as a recurrent network. RWKV (2023) trains in parallel but runs with constant memory per step, and was scaled to 14 billion parameters.

xLSTM (2024) extends the 1997 LSTM with exponential gating and a matrix memory that trains in parallel. Newer releases use gated linear layers, such as the Gated DeltaNet in Qwen3-Next and Qwen3.5, and the KDA layers of Kimi Linear.

The recall problem

Jelassi and others (2024) showed that transformers far outperform state space models at copying and retrieving from context. The Zoology study traced about 82% of the gap between attention and gated-convolution models to recall of earlier context.

Waleffe and others (2024), an NVIDIA-led study with co-authors from state space model companies, found pure Mamba at 8 billion parameters behind on 5-shot MMLU, phonebook lookup and long-context reasoning.

Their hybrid of 43% Mamba-2, 7% attention and 50% feed-forward (MLP) layers beat the transformer on all 12 standard tasks, by 2.65 points on average. It is predicted to generate up to 8 times faster.

What the releases show

Ratios vary. Jamba has 1 attention layer per 7 Mamba layers, plus mixture of experts, and Granite 4.0-H has 1 transformer layer per 9 Mamba-2 layers. Qwen3-Next, Qwen3.5 and Kimi Linear have 1 attention layer per 3 linear ones. NVIDIA’s Nemotron 3 Nano has 23 Mamba-2, 23 mixture-of-experts (MoE) and 6 attention layers.

Efficiency figures are mostly makers’ own: IBM claims over 70% less memory, and Qwen claims 10 times the throughput above 32,000 tokens. In Google’s test, Gemma-7B ran out of memory at 8,192 output tokens, while RecurrentGemma-9B finished.

MiniMax dropped its lightning-attention hybrid for M2, citing “clear deficits” in multi-hop reasoning at scale and less mature serving software. Whether hybrids hold up at frontier scale is still open.