How frontier models work

1. What makes a model frontier?

Here, frontier means a model near the leading edge of broadly useful AI capability at the time it is assessed. It describes a moving position, rather than a particular architecture or parameter count. Leadership can differ across coding, scientific reasoning, language, visual understanding, speed and reliability. A model that excels at one demanding task may be an inferior choice for another.

A foundation model is trained broadly enough to support many downstream uses. A large language model, or LLM, specialises in processing and generating language-like sequences. A frontier system may combine language with images, sound and tools. These terms overlap, but they answer different questions: how broadly a model can be reused, what it processes, and how capable it is.

This guide explains the publicly documented approach behind many leading language and multimodal systems. Exact architectures, training mixtures, parameter counts and costs are not fully disclosed for many commercial models. The named papers are evidence for particular techniques and historical examples; they are not a claim that every current lab follows an identical recipe.

The distinction between the model and the product is essential. A chat interface may choose between models, search the web, retrieve files and execute code. An impressive result can reflect all of those components working together. Evaluating the underlying model requires knowing which help it received.

2. Tokens, weights and context

A language model does not receive a sentence as a single object. A tokeniser converts it into numbered pieces called tokens. A piece might be a word, part of a word, punctuation or a fragment of code. Token counts depend on the tokeniser, language and content; a token is not a fixed number of words.

Each token is mapped to a vector, an ordered list of numbers called an embedding. Position information helps the model distinguish sequences such as “the dog chased the cat” and “the cat chased the dog”. As the vectors pass through the network, their representations change in response to the surrounding context.

Weights are the learned numerical parameters that govern these transformations. Activations are the temporary values computed for the current input. A model’s knowledge and behaviours are distributed across its parameters; it is not generally looking up a stored answer in a neat internal database. It can nevertheless memorise some training material.

The context window is the sequence capacity available to a request. Instructions, conversation history, retrieved passages, tool results and generated output use that capacity according to the system’s implementation. Providing a document changes what the model can work with during the request. It does not ordinarily rewrite the model’s weights.

Public training reports make these layers tangible: Meta’s Llama 3 report describes tokenisation, model architecture, data preparation and the later adaptation into an assistant. Llama 3 technical report.

3. Inside a transformer

A transformer repeatedly updates representations using attention and other learned transformations. Many language models use a causal decoder: a position can use the preceding sequence and its current input, while future target tokens are hidden.

Attention mixes information from accessible positions. Learned projections create queries, keys and values. Query–key comparisons produce scores; normalising them produces weights for combining the values. Several attention heads can learn different relationships. The feed-forward part then transforms each position’s representation. Residual connections and normalisation help information and learning signals pass through many layers. Attention Is All You Need.

At the output, scores over the vocabulary become probabilities. The decoding method selects or samples a token. Temperature changes how concentrated that sampling distribution is; it is not a truthfulness setting. A highly likely continuation can still be wrong.

Consider the unfinished phrase “The capital of France is”. The model assigns probabilities to possible next tokens. Predicting that continuation is easy compared with writing a correct program, but both use learned representations to condition a sequence of outputs. The training objective is simple to state; the computation learned to satisfy it can be complex. Attention itself should not be mistaken for human awareness or a complete explanation of why an answer was produced.

4. How pre-training works

For a conventional autoregressive language model, pre-training repeatedly asks the network to predict the next token in training sequences. A loss function penalises predictions that assign too little probability to the observed continuation. Backpropagation calculates how parameters contributed to that loss; an optimiser updates them. Repeating this across many batches produces a reusable model.

During training, the example sequence is already available, so predictions at many positions can be computed together with causal masking. During ordinary generation, future output tokens have not yet been selected. This helps explain why learning from a passage and writing a new answer have different computational patterns. Transformer training architecture.

The data mixture influences what the model learns. Code, multilingual text, mathematical material and general web documents make different contributions. More tokens can also mean more duplication, low-quality material or contamination of evaluation sets. The Llama 3 report documents extensive filtering and curation; its authors attribute improvements to data and training scale as well as architecture. Llama 3: pre-training data.

A frontier training programme includes failed experiments, smaller trials, evaluations, data preparation and engineering. The compute reported for a final successful run therefore does not establish the full cost of developing the model or the business.

5. What scaling actually buys

Empirical scaling laws describe how measures such as predictive loss change with model size, data and training compute. They help labs plan experiments and allocate resources. They do not guarantee that every downstream capability improves smoothly, or that a particular level of spending produces human-level general intelligence. Scaling Laws for Neural Language Models.

The Chinchilla research demonstrated that the balance matters. Under its experimental conditions, spending a fixed training budget on a smaller model with more training tokens could beat a larger model trained on fewer tokens. The broader lesson is to optimise a combination of resources. A parameter count by itself is an incomplete measure of capability. Training Compute-Optimal Large Language Models.

Training efficiency and lifetime economics are different objectives. Extra training of a smaller model can be worthwhile when it reduces the cost of serving a very large number of future requests. Conversely, the highest possible benchmark score may justify an expensive model for a narrow set of valuable tasks. The right trade-off depends on quality requirements, usage and latency.

Large training jobs also require coordinated hardware. Data parallelism divides examples across model replicas. Tensor parallelism divides large calculations; pipeline parallelism divides model stages. Communication and failures can leave accelerators waiting, so useful progress depends on networks, software and reliability as well as arithmetic capacity. DeepSeek-V3 documents the engineering needed to train an expert model across a large cluster. DeepSeek-V3 technical report.

This connects the model to the site’s compute, networking and memory research: the model’s design helps determine which physical resources become constraints.

6. Teaching useful behaviour

A model trained to continue text is not automatically a dependable assistant. Post-training shapes how it follows instructions, explains answers, uses tools and handles uncertainty. Different labs combine methods in different sequences; there is no mandatory three-step recipe.

In the classic RLHF approach, people rank outputs, a reward model learns those preferences, and reinforcement learning adjusts the assistant to obtain higher rewards. InstructGPT showed how this could improve instruction following even without simply making the base model bigger. Its authors also reported continuing errors. InstructGPT research.

Direct preference optimisation, or DPO, uses preference pairs to update the model directly, without the same separate reward-model-and-RL loop. It is an alternative way to train from preferences, not a synonym for every form of reinforcement learning. DPO paper.

AI feedback can help generate critiques and preference labels at scale. Constitutional AI demonstrated a process in which stated principles guided model critiques, revisions and preference judgements. Human decisions still enter through the principles, data and evaluation criteria. Automated feedback reduces some annotation work; it does not create an independent ground truth. Constitutional AI.

7. Reasoning and test-time compute

Reasoning models are trained or configured to spend additional computation working through a task before producing a final answer. That can involve longer intermediate sequences, trying candidate solutions, checking results or using tools. Writing more words is not itself evidence of better reasoning.

Reinforcement learning is particularly useful where an answer can be checked. A mathematical result can be compared with a known solution; a program can be tested. The reward encourages behaviours that improve measured success. DeepSeek-R1 provides a public example of reasoning training and of using a stronger model’s generated solutions to train smaller models. Its results do not establish that the same rewards solve every kind of open-ended task. DeepSeek-R1 research.

Test-time compute is the computation spent answering a request, after training. A system may use it to explore several answers and select one, or revise a candidate using feedback. Research by Snell and colleagues shows that the useful allocation depends on the problem and the underlying model. More effort can help, but gains are neither uniform nor unlimited. Research on test-time compute.

The economic consequence is that two requests to the same model can consume very different resources. A short customer query and a difficult debugging job are different workloads even when their final answers have similar lengths.

Visible explanations also need care. Experiments have found that a plausible chain of thought can omit influences on the model’s answer or rationalise a biased result. Intermediate text can be useful for solving and inspecting tasks, but it is not a guaranteed faithful transcript of all internal computation. Research on unfaithful explanations.

8. Mixture of experts: capacity without using every weight

A dense transformer uses the same main computational blocks for each token. A mixture-of-experts, or MoE, model replaces selected components with a collection of expert networks and a router that selects a subset for each token. Much of the model can remain shared. The experts are learned numerical modules; they are not necessarily separate subject specialists or independent chatbots.

The Switch Transformers research illustrates how selective activation can increase model capacity while limiting computation per input. Switch Transformers.

MoE separates total parameters from active parameters per token. DeepSeek-V3’s original report describes 671 billion total parameters with 37 billion activated per token. That is a documented architectural example, not a description of every frontier model. DeepSeek-V3 architecture.

The design can increase capacity without paying the arithmetic cost of activating every expert on each step. However, the weights still have to be stored or moved, and routing creates communication and load-balancing challenges. Fewer active parameters do not imply that the entire system fits in the memory required by a dense model of that smaller size. The business question is delivered quality and throughput at a usable latency, after those system costs.

9. How images, audio and video enter the model

Multimodal models transform different kinds of input into representations the network can process together. Images may be divided into patches or represented by a vision encoder; audio may use acoustic features or learned units; video requires both visual content and temporal information. Architectures differ in where these representations meet and how jointly they are trained.

A model asked to interpret a chart must connect visual marks, labels and the user’s question. A voice system must manage sound and language, potentially including timing and turn-taking. Converting everything into a text transcript can discard information such as visual layout or tone. The Gemini technical report is a public example of research spanning text, image, audio and video understanding. Gemini multimodal report.

Input and output capabilities must be assessed separately. Understanding an image does not automatically imply native image generation. A product may call a separate image, speech or video model. The visible interface can conceal that division of work.

For buyers, modality claims should become practical tests: can the system read the actual charts, noisy recordings or scanned documents in the workflow, at an acceptable cost and error rate?

10. What happens when the model serves an answer

Inference runs the trained model. For a conventional autoregressive transformer, serving is often discussed in two phases: prefill processes the input sequence, and decode generates subsequent tokens. Reading a long document and producing a long answer therefore create different demands.

The key-value cache, or KV cache, retains intermediate attention data for previously processed tokens. This saves repeated work but consumes memory, especially across long conversations and many simultaneous users. PagedAttention and the vLLM serving system show why managing that memory efficiently can raise useful throughput. PagedAttention research.

Prefill often offers substantial parallel computation. Small-batch decode can be constrained by moving model weights and cached data through memory. The actual bottleneck depends on the architecture, hardware, sequence lengths, batching and concurrency. Peak chip FLOPS alone cannot determine the user experience.

FlashAttention improves the organisation of attention computation to reduce transfers between levels of GPU memory. It is an example of faster exact attention, rather than a change that makes the model attend to fewer tokens by definition. FlashAttention.

Quantisation uses lower-precision representations for weights or other numerical values. It can reduce memory needs and sometimes improve speed, but the quality impact and hardware benefit require measurement. GPTQ is one demonstrated approach to quantising model weights. GPTQ.

Speculative decoding lets a cheaper drafting process propose tokens for a larger model to verify. Correct acceptance and correction rules can preserve the target sampling distribution. The speed benefit depends partly on how often the proposed tokens are accepted. Speculative decoding research.

Other improvements include reusing identical prompt prefixes, batching compatible requests and routing simpler tasks to smaller models. Each changes the balance between cost, latency and quality. A provider’s price per token is a commercial offer, not a direct disclosure of its serving cost.

11. Retrieval, memory and agents

A model’s training cannot contain every current company document or live database value. Retrieval-augmented generation, or RAG, finds relevant external information and supplies it as context for generation. Retrieval may combine keyword search, vector similarity, structured queries and reranking. It can improve access to evidence without retraining the model, but a bad retrieval result can still produce a bad answer. Retrieval-augmented generation research.

An agent adds a loop around model calls: observe the task, choose an action, execute an allowed tool, inspect the result, and continue or finish. The application executes the tool; a model-generated description of an action does not mean it happened. Anthropic’s account of agent design distinguishes predefined workflows from systems that decide how to proceed more dynamically. Building effective agents.

Persistent “memory” usually involves stored information being selected and reintroduced into later context. It is distinct from both learned weights and the temporary KV cache. Fine-tuning changes parameters; RAG changes the evidence supplied to a request; external memory carries selected state across requests. Those mechanisms solve different problems.

Long context is useful but not a guarantee that every detail is used well. The 2023 Lost in the Middle experiments showed sensitivity to where evidence appeared in the models tested. Current models need fresh testing, but the evaluation principle survives: measure successful use of information, not just advertised window size. Lost in the Middle.

12. Why models fail, and how to evaluate them

Language generation can produce convincing unsupported claims. Incomplete knowledge, ambiguous questions, unreliable source material, faulty reasoning and incentives to give an answer can all contribute. Fluent language and high token probability are not equivalent to factual verification. Research examining internal representations and truthfulness also distinguishes the probability of a sentence from the truth of its claim. Research on truthfulness and model representations.

Tool use changes the risks as well as the capabilities. A mistaken answer can become an incorrect action. Retrieved content can contain instructions that conflict with the task. Access controls, restricted tool interfaces, external checks and human review for consequential decisions are therefore part of system design; they cannot all be replaced by a better prompt.

Evaluation should cover more than headline accuracy. HELM established a framework spanning accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. The appropriate balance depends on the use case. A model that performs well on one test suite may still fail on a different language, document type or business process. Holistic Evaluation of Language Models.

For agents, inspect the final environment and run repeated trials. Saying “the issue is fixed” is different from leaving working code that passes relevant tests. Record the tools, prompts, time allowance, attempts and grading rules: these affect the result. Anthropic’s evaluation guide explains why tests must examine both the sequence of actions and the actual outcome. Demystifying evals for AI agents.

A credible model comparison should disclose the exact version, task set, available tools, reasoning budget, success criteria and cost. Tests should include fresh tasks that are unlikely to have entered training, plus ordinary failures found in production. Passing a benchmark is evidence about those test conditions; dependable deployment requires evidence about the intended workload.

13. The economics: cost per successful task

Our analytical view is that the most useful economic unit is the cost of an accepted outcome. Token prices are inputs to that calculation. They do not capture how many attempts, tool calls or minutes of review are needed.

Consider an illustrative workload of 100 tasks. System A costs £20 in model and tool usage, then £80 in review, and produces 80 accepted outcomes: £1.25 each. System B costs £40 in usage and £40 in review, and produces 95 accepted outcomes: about £0.84 each. The more expensive model usage can produce the cheaper completed work. These numbers are hypothetical, not vendor pricing or performance estimates.

For a model developer, the corresponding questions are whether revenue covers serving costs, whether customers keep using the service, and whether enough cash remains to fund research and subsequent model generations. A reported training-run cost leaves out many expenses. A profitable serving margin does not by itself demonstrate that the whole company is profitable.

Open weights change who can host and adapt a model, subject to its licence. They do not necessarily include the training data or a fully reproducible development process. Hosting still requires hardware, operations, updates and security. An API can simplify those responsibilities while introducing supplier dependence. Buyers should compare the full operating arrangement.

Distillation trains a student using information from a stronger teacher, often generated answers or solution traces. It can make useful capability cheaper to deploy without reproducing every teacher capability. DeepSeek-R1’s released distilled models provide a concrete example. DeepSeek-R1 distillation.

Our investment interpretation is that technical leadership becomes commercially durable only when it supports repeat customer value. Advantages can come from better models, lower serving cost, distribution, proprietary task feedback or strong integration with real workflows. Competition can transfer efficiency gains to customers, while more demanding reasoning and agent tasks can increase consumption. The net effect depends on adoption, prices and the amount of work attempted.

14. What to watch

Frontier progress can come from better data, architecture, learning signals, inference software and the environment in which a model operates. Following those mechanisms makes it easier to judge whether an announcement changes useful capability, the cost of delivering it, or both.