Model architectures explained

1. Architecture, weights and training are different levers

Think of architecture as specifying the operations and connections; weights as the learned numbers inside those operations; and training as the process that adjusts those numbers. Two checkpoints can share an architecture yet behave differently because their data, objectives or training histories differ.

Execution adds another layer. The same mathematical computation can be scheduled and implemented differently on hardware. FlashAttention is an important example of improving attention’s memory traffic without replacing its mathematical operation. We return to that distinction below. The original FlashAttention paper.

The original transformer combined an encoder and a decoder. Modern language-model designs are not all copies of its complete layout. Vaswani et al.’s transformer architecture. This guide uses a causal decoder as its main explanatory path, then shows where other arrangements diverge.

These distinctions matter when reading a release. “Uses MoE” identifies a conditional-computation choice. It does not disclose the training data. “Uses a long-context attention design” describes connectivity or storage, not whether the model successfully follows a complicated argument across that context. Treat an architecture label as the start of an explanation, not the conclusion.

2. Inside a transformer block

A text model first maps token IDs to numerical vectors called embeddings. Successive blocks transform those representations; an output projection produces vocabulary scores. In an autoregressive decoder, those scores support the next-token prediction. The repeated block structure is the important idea—not a sequence of human-readable reasoning steps. Raffel et al.’s transformer overview.

Residual connections add a sublayer’s result back to the representation flowing through the block. In a pre-normalised layout, normalisation acts before the attention or feed-forward transformation. LLaMA uses RMSNorm, rotary positions and SwiGLU: a useful concrete example, not a specification for every transformer. LLaMA’s architecture section.

Attention is only part of the model

The feed-forward network, also called an FFN or MLP, applies learned feature transformations at each position. A basic version projects into a hidden space, applies a nonlinearity and projects back. It uses the same layer weights across positions, but the input features differ.

Gated variants such as SwiGLU combine two projected feature streams through elementwise multiplication before the output projection. This is feature gating inside a network—not the top-k expert selection described later. Gated designs also change parameter and computation accounting; the GLU study adjusted hidden widths to make its comparisons approximately budget-matched. Shazeer’s GLU variants study.

A useful mental model is: attention mixes information between positions; the FFN transforms the features available at each position; residual paths carry the evolving representation onward. This is an explanation of computational roles, not a claim that knowledge and reasoning live in neatly separated compartments.

3. What attention actually computes

Attention forms learned queries, keys and values from representations. A query is compared with permitted keys. Scaled dot-product scores pass through a softmax, producing weights for a weighted sum of the values. The scale uses the square root of the key dimension.

Multi-head attention performs several such operations with different learned projections, then combines their outputs. Different heads provide different representation subspaces; a head need not correspond to a named human skill. The original scaled dot-product and multi-head definitions.

The “query–key–value” vocabulary is a helpful lookup analogy, but these are learned vectors, not database fields or explicit questions written in English. Understanding the arithmetic does not, by itself, tell us what a particular trained head has learned to represent.

4. Encoders, decoders and attention masks

A mask determines which positions may contribute to an attention result. It is a structural rule about visibility, not a filter that verifies an answer.

BERT’s model architecture explains the bidirectional encoder; LLaMA supplies the decoder example; T5’s architecture comparison explains the two-stack arrangement. These are design examples, not a ranking of the families.

During decoder training, known target tokens can be processed in parallel while the causal mask prevents later targets from leaking into an earlier prediction. During generation, the next token is not known yet. Parallel training therefore does not imply that an ordinary autoregressive decoder can produce its entire answer independently in one pass. T5’s masking and generation discussion.

Do not turn these examples into rigid task boundaries. A decoder can classify by generating a label; an encoder can support extraction through an appropriate output head. BERT’s downstream setups illustrate how the task interface sits on top of learned representations. BERT’s downstream task formulations.

5. Dense models and mixture of experts

In a conventional dense FFN, each token representation passes through the same feed-forward weights at a given layer. A sparse mixture-of-experts (MoE) layer supplies several alternative FFNs and a learned router that selects a subset for each token. Selection can differ between tokens and layers.

Mixtral’s disclosed design uses eight FFN experts per layer and selects two. Their outputs are combined using routing weights. This is not eight complete chatbots debating an answer. Its roughly 47 billion total parameters and 13 billion active parameters per token illustrate why the name “8×7B” should not be read as eight independent, fully duplicated 7B models. Mixtral’s architecture and parameter accounting.

“Expert” does not mean a named subject specialist

Mixtral’s routing analysis did not find obvious topic-based assignments across most examined domains; some patterns appeared more syntactic. That does not rule out specialisation in other designs, but it makes diagrams labelled “the maths expert” or “the law expert” misleading without specific evidence. Mixtral’s routing analysis.

DeepSeekMoE explores finer-grained experts and shared experts that always run, alongside selected routed experts. Its motivation is to separate common transformations from more specialised ones. Shared experts are additional FFNs, not another name for the attention layers shared by tokens. DeepSeekMoE’s shared and routed architecture.

6. Routing, balance and the real cost of MoE

The full expert collection still has to be stored somewhere. Different tokens may activate different experts, and distributing experts across devices introduces dispatch and return traffic. Uneven routes also create uneven workloads. MegaBlocks addresses variable expert batches with block-sparse computation; its analysis connects full-weight memory pressure to the batch sizes that fit. Arithmetic savings alone do not settle deployment cost. MegaBlocks’ execution and memory analysis.

Load balancing discourages routing most tokens to only a few experts. Switch Transformers uses top-one routing with capacity limits and an auxiliary balancing objective. In that implementation, overflow tokens can bypass expert computation through the residual path. “Token dropping” here does not mean deleting words from the user’s input. Switch’s capacity and balancing mechanism.

Dropping is not an inherent requirement of MoE. MegaBlocks demonstrates a no-dropping approach using variable block-sparse workloads. The relevant questions are which capacity policy is used and what happens when an expert receives more work than expected. MegaBlocks’ no-dropping design.

Recipes also differ substantially in granularity. DeepSeek-V3 reports 671 billion total parameters and 37 billion activated per token. Its MoE layers have one shared expert and 256 routed experts, selecting eight routed experts per token; the first three FFN layers are dense. These are specifications of that disclosed model, not default settings for MoE generally. DeepSeek-V3’s architecture configuration.

7. MHA, MQA, GQA and MLA: changing the attention memory

During autoregressive generation, an attention layer can retain earlier keys and values rather than recomputing them for every new token. This KV cache is sequence-specific working state, not the model’s learned weights. Reading it becomes an important part of decoding cost as the retained sequence grows. Shazeer’s analysis of incremental decoding.

The MQA paper establishes shared K/V; the GQA paper generalises the grouping; DeepSeek-V2 explains MLA. These are not four universal quality tiers.

GQA lies between the endpoints: one KV group gives MQA; as many KV groups as query heads gives MHA. Ainslie and colleagues also describe converting existing checkpoints through pooling and further training. It is not generally a lossless runtime switch applied to arbitrary weights. Actual memory savings can differ with the way heads are replicated across devices. GQA’s method and implementation considerations.

MLA compresses the representation, not just the head count

DeepSeek-V2 learns a low-rank joint representation for keys and values. Its execution can absorb some projection matrices into adjacent computations. The cache also includes a separate rotary-position key component: counting only the compressed latent would understate storage. This is learned architectural compression, not a generic lossless compression format for any existing transformer. Retaining one such state per token still makes cache size grow with sequence length. DeepSeek-V2’s MLA and decoupled RoPE sections.

8. Positions and context extension

Attention needs a way to account for order and distance. “The dog chased the cat” and “The cat chased the dog” contain the same words but different relationships; token identity alone is insufficient to express that distinction.

Rotary position embeddings, or RoPE, rotate pairs of query and key coordinates using position-dependent angles at multiple frequencies. Their dot product then incorporates relative displacement as well as content. RoPE changes the position-sensitive comparison; it is not a memory bank or an attention mask. Nor does its relative-position structure guarantee successful use of arbitrarily long inputs. RoFormer’s rotary-position derivation.

Stretching positions is different from learning to use them

Position Interpolation maps a longer sequence’s positions back into the original coordinate range. For original length L and extended length L′, position m becomes mL/L′. The coordinate mapping itself adds no model parameters; the reported recipe then fine-tunes existing weights on longer sequences. Merely raising a configured limit is not the same intervention. Chen et al.’s Position Interpolation method.

YaRN uses frequency-dependent scaling: it preserves short-wavelength components, interpolates long-wavelength components and blends the intermediate region. It also rescales attention logits. The authors’ rationale is to extend positional range while disturbing local distinctions less than uniform scaling would. This attention-softmax adjustment is not the temperature used to sample output tokens. YaRN’s interpolation and attention-scaling methods.

Both approaches need model-specific evidence. Their reported adaptation and evaluation results are not promises that any checkpoint can be extended by the same factor. A positional transformation, a long-sequence training recipe and a successful task test are three separate pieces of a context-extension claim. Position Interpolation’s experiments; YaRN’s long-context experiments.

9. Sliding windows and sparse attention

Another way to control cost is to change which attention connections exist. A sliding-window layer gives each position direct access to a bounded recent region. Newer states can carry influences from earlier layers, so information may travel farther through the stack than one layer’s window.

That distinction is explicit in Mistral 7B v0.1: the paper lists a 4,096-token window and an 8,192-token context length, while describing an approximately 131K theoretical attention span across 32 layers. The theoretical span is not a validated 131K usable context window, nor direct access to every original token. Its rolling cache overwrites old layer-specific entries while newer representations can retain their influence. Mistral 7B’s windows, architecture table and rolling buffer.

Longformer illustrates a different sparse design: local windows, optional dilated connections and selected globally connected positions. In its bidirectional encoder, global positions can gather information across the sequence and other positions can attend to them. A causal decoder must still respect its future-token restrictions; copying the encoder’s unrestricted connectivity would change the problem. Longformer’s attention patterns.

With sequence length n, a fixed local width w and g global positions, attention-pair counts scale on the order of nw + ng. Calling this “linear” assumes w and g remain bounded as n grows. Global tokens are also not the same as alternating local layers with fully connected attention layers. Longformer’s complexity and connectivity analysis.

The essential trade-off is direct availability versus bounded work. A computational path through intermediate states permits information flow; it does not guarantee that every detail survives or can be recovered accurately.

10. Faster attention, different sequence models and hybrids

FlashAttention: execute the same operation more efficiently

FlashAttention tiles the calculation and uses online softmax bookkeeping to reduce traffic to accelerator high-bandwidth memory. It avoids materialising the entire token-pair attention matrix there. Its dense version computes the same mathematical attention, subject to numerical precision; it does not make full-sequence dense attention’s arithmetic cease to be quadratic. Reducing intermediate storage is different from removing attention connections. FlashAttention’s algorithm and complexity analysis.

State-space models: maintain an evolving summary

A recurrent state-space model updates a carried state as inputs arrive. Mamba makes parts of that update input-dependent, so the input helps control what is carried forward. This selectivity comes from learned computations; it is not a weight-training step during every user interaction.

For a fixed model, its recurrent state can stay fixed in size as sequence length grows, unlike a full per-position KV cache. That changes the memory trade-off: past information must pass through a finite evolving summary. It does not create unlimited exact recall. Mamba’s hardware-aware parallel scan also shows why recurrent generation need not require purely sequential training. Mamba’s selective state-space and execution design.

“Constant-time decoding” in this setting concerns the per-token recurrence with fixed model and state dimensions—not reading an arbitrarily long prompt or generating an entire answer in constant time.

Mamba-2 develops a mathematical connection between particular structured state-space models and forms of linear attention. That does not make an ordinary softmax transformer identical to Mamba. Efficient reassociation or recurrence depends on the structure of the chosen operation; the softmax in standard attention cannot simply be ignored. Dao and Gu’s structured state-space duality.

Hybrids combine design choices

Jamba’s original architecture interleaves attention and Mamba layers, with MoE in selected feed-forward blocks. It demonstrates that “transformer versus state space” and “dense versus MoE” are different axes that can be combined. Fewer attention layers can reduce growing KV storage, but the remaining attention caches still grow and recurrent layers retain their own state. Jamba’s hybrid architecture.

These papers establish concrete alternatives, not a universal successor. Whether a hybrid’s memory savings are worth its quality or implementation trade-offs requires a matched evaluation.

11. Context capacity is not the same as usable context

Separate four questions: Can the system accept the input? Can information reach the relevant computation? Has the model adapted to this regime? Does it solve the task reliably? A “yes” to an earlier question is not automatically a “yes” to the next.

RULER tests more than finding a single planted fact. Its suite includes retrieval variants, multi-hop tracing, aggregation and question answering. The paper’s “effective length” is benchmark-defined: the largest tested length exceeding a chosen Llama2-7B-at-4K performance threshold. It is not an intrinsic constant of the architecture. RULER’s task design and effective-context definition.

The authors also identify limitations, including incomplete position control and an unverified correspondence to realistic tasks. Use such a suite as a diagnostic alongside application tests, not as the complete meaning of long-context understanding. RULER’s limitations.

Lost in the Middle moved answer-bearing information among distractors while keeping the task’s desired answer unchanged. Many tested models performed better when the relevant material appeared near the beginning or end. Those historical experiments motivate testing position sensitivity; they do not establish that every later model has the same curve, or that RoPE alone caused it. Liu et al.’s controlled position experiments.

An illustrative document-assistant test should therefore vary document length, answer position, distractor similarity and the number of facts that must be combined. Include cases where the requested fact is absent. Record task accuracy and evidence quality, not just whether the request was accepted. This is a proposed evaluation checklist, not a benchmark result.

12. How to compare architectures without being misled

A fair comparison begins with the question the model must answer and the constraints under which it must answer it. The table below is a practical synthesis of the mechanisms in this guide, not an empirical ranking.

In practice, compare quality at an acceptable cost, not parameter count in isolation. Also distinguish processing a long prompt from generating a long answer: their bottlenecks need not match. The companion inference guide follows these choices into serving and reasoning budgets.

The architectural lesson: MoE changes which transformations run; KV designs change what attention retains; sparse attention changes connectivity; state-space models change how past information is carried; efficient kernels change how an operation executes. These levers can reinforce one another, but none eliminates the need to measure the resulting system.

Glossary

Sources and research notes

This guide synthesises 24 original papers and technical reports, with public evidence reviewed on 6 September 2026. Examples retain their disclosed versions: they are explanations of mechanisms, not claims about the latest commercial systems or a current leaderboard. Undisclosed proprietary architectures are outside scope.

The numerical examples are original, hypothetical calculations. Named-model parameter counts are reported specifications, not measurements made for this guide. Historical benchmark findings are bounded to their tested systems. Simplified diagrams omit implementation details; no single diagram represents every model.