Inference & reasoning systems

1. The request lifecycle

It helps to separate three actors. The model computes predictions from its current input. The inference engine manages the calculations, memory and shared hardware. The application decides what information to provide, whether to call tools, how many attempts to allow and what to return. A product can improve any of these without changing all the others.

For a simple rewrite, the middle of this workflow might be one generation. For a coding task, it might be a succession of file reads, proposed edits, test runs and corrections. The user sees one task, but the system handles several dependent model calls. MLCommons’ 2026 agentic workload explicitly represents conversations as ordered turns rather than treating every prompt as an unrelated request. MLCommons’ agentic inference design.

During ordinary inference, the model’s learned weights remain fixed. New instructions, generated reasoning and tool results change its context and intermediate state; they do not, by themselves, constitute a training update. Keeping this distinction in mind prevents “the model remembered” from becoming an explanation for several very different mechanisms.

2. How the next token is chosen

An autoregressive language model predicts a distribution over possible next tokens, conditioned on the tokens already available. A token can be a word, part of a word, punctuation or another encoded unit. The system selects a token, appends it to the sequence and repeats until a stopping condition is reached. The choice rule is called a decoding policy.

Greedy decoding chooses the highest-probability next token. Sampling draws from the distribution instead, allowing different continuations. Neither procedure searches a database for a complete prewritten answer. Both build a continuation from conditional predictions. Hugging Face’s generation-strategy documentation.

Temperature changes how sharply probabilities are concentrated: lower values favour the leading options more strongly. Top-p, or nucleus sampling, keeps a high-probability set whose cumulative mass reaches a threshold, then samples within it. These controls influence variety; they are not settings for factual accuracy or intellectual effort. The generation reference defines these mechanisms.

Why not always take the most probable token? In the open-ended generation experiments behind nucleus sampling, likelihood-maximising approaches could produce repetitive or bland text. That was a finding about particular models and tasks, not evidence that randomness universally improves reasoning. It shows that output quality depends on the selection rule as well as the weights. Holtzman et al.’s nucleus-sampling paper.

A lower-temperature answer can still be confidently wrong. And “deterministic decoding” is not necessarily identical to reproducibility across an entire service: numerical execution, batching, hardware and software versions can matter. vLLM documents additional requirements for reproducible execution rather than promising that a seed alone is sufficient. vLLM’s reproducibility guidance.

3. Why answers take time

The conventional split is prefill and decode. During prefill, the system processes the available prompt, often handling many positions in parallel, and prepares the first output token. During decode, later tokens depend on earlier generated tokens. Cached attention state avoids recomputing the entire preceding sequence at every step. MLCommons’ prefill and decode explanation.

These phases put different pressure on hardware. A substantial prefill can supply large parallel calculations. Small-batch decode often spends much of its time moving model weights and attention state through memory. Larger batches share some weight reads across requests, changing the balance. Neither “prefill is always compute-bound” nor “decode is always memory-bound” is a safe universal rule. The JAX scaling book’s inference analysis.

Three clocks that should not be confused

• Time to first token (TTFT): time until the first token at a specified measurement point. A client measurement can include queueing, transport and scheduling, not just prefill.
• Time between tokens: the spacing of streamed output. An average time per output token can conceal occasional long pauses.
• Time to useful answer: time until the user receives enough of the actual answer to proceed—or until the complete result arrives, if completion is required.

For reasoning systems, the first generated token might belong to an intermediate reasoning sequence. If that sequence is not exposed as the final answer, fast internal generation does not imply a fast first visible answer. A progress indicator, a reasoning summary and an answer token are different events. Reports need to say which one starts or stops the timer.

4. What extra thinking actually means

Test-time compute is computation used when answering a request, rather than when training the model. It can extend one attempt, produce several attempts, or support a search that repeatedly scores partial solutions. These methods can be combined; a commercial “reasoning effort” label does not disclose which combination is running.

Snell and colleagues studied the first three approaches with specially adapted PaLM-2 models on competition mathematics. The effective allocation depended on model-relative problem difficulty. Their result supports conditional, adaptive allocation—not a rule that every difficult question becomes solvable by generating longer. Snell et al.’s test-time compute study.

Training teaches behaviour; runtime chooses how much to run

In DeepSeek’s disclosed research, R1-Zero applied reinforcement learning to an already pretrained base, using rule-based correctness and formatting rewards. The released R1 used a different, multi-stage combination of supervised fine-tuning and reinforcement learning. “No supervised fine-tuning” for the Zero experiment does not mean no pretraining, and Zero’s reward setup was not a neural process verifier. The DeepSeek-R1 technical report.

The s1 project illustrates another route: fine-tuning on 1,000 curated examples, then controlling when a trained model stops reasoning. Its “budget forcing” intervenes in decoding to end or extend the reasoning phase. This is more specific than asking an arbitrary model to “think harder”; both the learned behaviour and the runtime intervention matter. The s1 paper.

The same distinction applies to self-correction. SCoRe trained improvement across successive attempts with reinforcement learning; its baseline models could sometimes turn correct answers into incorrect revisions. Successful trained correction is evidence against both extremes: that self-correction is impossible, or that it automatically follows from adding “check your answer.” The study’s correctness reward during training was not a ground-truth oracle available at deployment. SCoRe’s self-correction research.

5. Generating a correct answer is not the same as selecting it

Imagine asking a model the same question several times. One candidate might be correct while another is more persuasive. A system returning one answer must solve a second problem: which candidate should it trust?

Self-consistency samples different reasoning paths and aggregates their final answers, often through voting. It can use one model repeatedly; it does not require a committee of separately trained models. Agreement can be useful, but shared misconceptions can produce the same wrong answer across many paths. The original study evaluates this approach on tasks with answers that can be extracted and compared. Wang et al.’s self-consistency paper.

A learned outcome verifier scores an answer or complete solution. A process verifier scores intermediate steps. In Let’s Verify Step by Step, process supervision improved selection among a fixed generator’s mathematics solutions in the studied setting. It did not establish perfect verification or universal superiority across open-ended tasks. Lightman et al.’s verifier study.

Checking plausibility is different from checking the world

A compiler, test suite or calculation can provide evidence outside the generator’s own verbal judgement. But tests cover only what they test, and a correct calculation can use an incorrect input. For an evidence-based answer, a useful check is whether its source actually supports the claim, including date, units and scope—not whether a second model finds the prose convincing.

Nor is a readable chain of thought a complete audit trail. Anthropic’s controlled hint experiments found that reasoning models sometimes followed an inserted hint without acknowledging that influence in their written reasoning. The evidence is bounded to those experiments; it does not make all reasoning traces useless. It does show why a fluent explanation is not guaranteed to reveal the causal basis of an answer. Anthropic’s faithfulness research.

6. Budgets, stopping and diminishing returns

A useful reasoning budget is more than a maximum token count. It can include an elapsed-time deadline, a candidate limit, a tool-call allowance and a stopping rule. A system might answer a routine extraction directly, spend more on an uncertain calculation, or escalate when the available evidence cannot support a reliable conclusion.

That is an operating policy to test, not a claim that models know their own difficulty perfectly. Snell et al.’s headline compute-efficiency comparison excluded the cost of estimating difficulty. A production router must pay for its own classification, extra attempts and verification. Savings should be measured after those costs, not borrowed from an idealised allocation. The study’s difficulty-estimation caveat, §3.2.

Longer reasoning can reach a plateau. In s1, repeatedly forcing continuation eventually produced flattening gains and repetitive loops. Separately, Chen et al. found redundant solution rounds in the reasoning models and mathematics tasks they studied. Their retrospective identification of the first correct solution does not mean a deployed system can recognise that exact stopping point without an additional reliable check. s1’s scaling limits; the overthinking study.

Extra work can also have negative returns: an incorrect revision can replace a correct answer, or search can exploit weaknesses in its scoring model. Snell et al. observed verifier over-optimisation at higher search budgets in some settings. More search increases the importance of a good objective; it does not repair a poor one automatically. The verifier-search results, §5.3.

A practical evaluation should compare several budgets on the same held-out task set. Measure the marginal change in accepted answers, failure types, completion time and total expense. Stop increasing the budget when the next increment no longer earns its cost—or when a deadline requires escalation. Missing evidence may call for retrieval or a human decision, rather than a longer continuation.

7. Four meanings of memory

“Memory” can mean learned knowledge, text supplied to the model, computed attention state or information stored by an application. They have different lifetimes and failure modes. Increasing one does not automatically improve the others.

The original retrieval-augmented generation work explicitly combines learned, parametric information with a separately retrieved information store. Retrieval selects evidence to make available for generation. It can supply newer or more specific material without treating every update as a model-weight change, but retrieval quality and source quality still matter. Lewis et al.’s RAG paper.

The KV cache solves a different problem: reuse of previous attention computation. Under ordinary causal attention, a token’s state depends on its preceding context. A matching phrase somewhere else is therefore not necessarily reusable KV state. The cache is computational working memory, not a semantic filing cabinet. The JAX scaling book’s attention-state analysis.

A longer window is capacity, not comprehension

A context window describes how much input and generated sequence a configuration can accommodate. It does not measure whether the model can find the right fact, combine distant details or resist a misleading passage. Lost in the Middle found strong position effects in the models and retrieval tasks it evaluated: information near the middle could be used less effectively than information near the ends. This is a historical warning to test effective context use, not a blanket verdict on every current model. Liu et al.’s long-context study.

For a long-running task, the application must choose what to retain verbatim, summarise, retrieve again or omit. Summarisation can preserve a useful overview while losing exact wording or constraints. As an editorial design principle, retain the original evidence and its identifiers outside the summary when exact provenance matters. “The conversation was compressed” should not be mistaken for “the model learned all its contents.”

8. Managing the KV cache

Long prompts, long reasoning sequences and many simultaneous requests can all increase demand for runtime state. Memory capacity can therefore limit concurrency before the system reaches its advertised arithmetic peak. The architecture guide’s memory-budget examples show how model and cache footprints differ.

Paging reduces waste; prefix caching avoids repeated work

PagedAttention maps logical KV blocks to physical memory blocks rather than requiring one large contiguous allocation for each sequence. Blocks can be allocated as a request grows and shared for compatible branches, with copying when necessary. This reduces allocation waste and enables sharing; it does not make the architecture’s required attention state disappear. Kwon et al.’s PagedAttention design.

Prefix caching skips recomputation when compatible prompt-prefix state is already retained. Repeated instructions or a shared document prefix may benefit. But a hit accelerates the reused prefill work, not the generation of all subsequent answer tokens. A decode-heavy answer can therefore see much less end-to-end benefit than its prompt-cache hit rate suggests. vLLM’s prefix-caching explanation.

Compatibility is stricter than similar wording. vLLM’s design identifies blocks using exact token IDs, ancestor-block identity and additional information such as adapter IDs, multimodal hashes and isolation salts. It caches full blocks, and eviction can remove them. Moving the same paragraph after a different prefix is not ordinary exact-prefix reuse. The prefix-cache design documentation.

Quantisation and offloading make different trades

KV quantisation stores attention state at reduced numerical precision. This can leave room for more context or concurrent requests, but scales, kernels and numerical effects matter. Validate answer quality and latency together: fewer stored bytes do not guarantee a faster response. vLLM’s quantised-cache documentation.

Offloading places eligible cached blocks in another memory tier and reloads them when useful. It trades scarce device capacity against transfer time and bandwidth. Asynchronous movement can overlap other work; it does not make movement free. Reusing offloaded prefixes is also distinct from streaming active attention state from host memory on every decode step. The KV offloading guide.

These techniques belong in separate boxes on an architecture diagram. Allocation, reuse, reduced precision and storage placement address different causes of memory pressure. Their gains should be measured together, because improving one can expose a different bottleneck.

9. Serving many users at once

A single fast demonstration is not a capacity test. A production service must accept requests with different prompt lengths, output lengths, arrival times and deadlines, while keeping enough useful work on expensive hardware.

Continuous batching changes batch membership between generation iterations: finished requests leave and new ones can enter. This avoids waiting for the longest answer before reusing every available slot. It is a scheduling technique; PagedAttention is a memory-management technique, even when they are deployed together. The PagedAttention paper’s scheduling background, §2.3.

Long prompts can interrupt short token steps

A large prefill can occupy resources that ongoing decodes need for their next tokens. Chunked prefill divides prompt processing into smaller scheduled pieces and interleaves them with decode work. Sarathi-Serve chooses a token budget around the desired time between tokens. Oversized chunks can cause pauses; undersized chunks can add overhead and inefficient repeated reads. Chunking schedules the prompt differently—it does not silently shorten it. The Sarathi-Serve paper.

Current vLLM documentation describes prioritising pending decodes and using the remaining token budget for prefill. Different budgets change the trade-off between starting new answers and keeping existing streams smooth. The correct setting depends on the model, hardware, workload and latency target. vLLM’s chunked-prefill guidance.

Separating prefill and decode can improve isolation

Disaggregated serving puts the two phases into different resource pools. DistServe’s design maintains model replicas in both pools and transfers KV state from prefill to decode. This lets each phase use a different allocation, but creates transfer and placement costs. Long prompts and high arrival rates can make that boundary expensive. The DistServe system design.

This is not an unconditional throughput multiplier. DistServe optimises goodput: requests meeting latency requirements with a given resource allocation. Current vLLM disaggregation documentation emphasises independent latency control and explicitly cautions that its feature does not improve throughput. The statements concern different objectives and implementations. Ask whether separation improves the service’s qualified capacity after accounting for both pools and KV transfer. vLLM’s disaggregated-prefill caveat; DistServe’s goodput objective.

A queue is where these choices become visible to users. A configuration may achieve attractive aggregate throughput while making an interactive request wait too long. Batch services and live conversations can therefore rationally choose different operating points.

10. Speculative decoding: faster generation, not deeper reasoning

Leviathan, Kalman and Matias give an exact scheme preserving the target’s configured sampling distribution. With target probability p and draft probability q, acceptance uses min(1, p/q). Rejection requires a correction drawn from the normalised positive residual of p − q. The original speculative-decoding paper.

This guarantees a distribution, not identical random sequences or better reasoning. Speed depends on draft cost, acceptance, verification and available parallel compute. Fewer serial target calls can coexist with more arithmetic and wasted drafts. Approximate variants need separate quality checks.

Executing a policy faster and changing that policy to improve answers are different interventions.

11. Measuring a complete system

A useful performance claim is a contract: which model and policy, which tasks, which hardware, which load, which quality threshold and which timer? Without those boundaries, two tokens-per-second figures can describe different services.

Output length matters because a system could appear faster by doing less generation. MLCommons’ 2026 reasoning benchmark additions pair performance requirements with quality checks and output-length compliance in specified workloads. Such controls make comparisons more meaningful, but a benchmark remains evidence about its defined tasks and settings—not every possible user request. MLCommons’ 2026 reasoning benchmark update.

The measurement boundary matters just as much. MLCommons’ agentic design preserves inter-turn waits for realistic pacing without counting those waits as model-serving latency. A person waiting for a tool-assisted answer still experiences the full workflow. Both measurements can be valid if their boundaries are explicit. The agentic workload’s timing rules.

The August 2026 end-to-end RAG benchmark goes beyond a single generation to include document ingestion and an iterative retrieval-and-answering pipeline. Its compliance discussion also states a limitation: an output-length check on the answer generator does not guard every intermediate pipeline stage. “End to end” is therefore a scope to inspect, not a substitute for reading the methodology. MLCommons’ end-to-end RAG benchmark.

For practical testing, retain the task set, model version, decoding policy, budgets and cache conditions alongside the result. Report latency distributions rather than just a mean: a 95th-percentile latency identifies a slower tail that an average can hide. Include rejected requests, failed tool calls and retries; excluding them rewards systems for dropping difficult work.

12. Cost per successful answer

A token price measures one input to an application’s economics. A successful answer may require several candidates, a verifier, retrieval, external tools, retries and review. Conversely, spending more on generation may avoid expensive downstream work. The useful denominator is the result the application actually values.

Cost per accepted outcome = total in-scope cost of all attempts ÷ number of accepted outcomes

Define “accepted” before measuring it: for example, a support response that satisfies a review rubric, or a code change that passes the required checks. Do not count each candidate as a separate successful task. For self-hosted systems, include the relevant serving resources and operating overhead; for purchased services, distinguish the bill from the provider’s underlying hardware cost.

A practical decision order

1. Define success and the deadline. Decide what failure means and what must be checked before returning a result.
2. Measure a simple baseline. Include complete-task latency, failed attempts and downstream review.
3. Diagnose the limiting factor. Missing evidence, weak reasoning, poor selection, queueing and cache pressure call for different changes.
4. Change one policy or mechanism at a time. Compare retrieval, reasoning budgets, verification and serving improvements on the same tasks.
5. Choose the smallest sufficient operating point. Retain extra computation where it demonstrably improves worthwhile outcomes, and re-test when the model or workload changes.

The core question is not “How much can the model think?” It is “Which additional work makes the delivered answer more useful, and is that improvement worth its full cost?” That connects inference engineering to model capability without confusing speed with intelligence.

Glossary

Sources and research notes

This guide synthesises original papers, official serving documentation and MLCommons benchmark methodology available by 6 September 2026. Research examples illustrate mechanisms; they are not current model rankings. Mathematics and coding findings are scoped to their evaluated settings, and implementation documentation describes that implementation rather than every provider.

All worked calculations are explicitly hypothetical. The workflow diagrams and decision framework are explanatory synthesis. Public evidence does not establish the hidden reasoning, routing or memory design of every commercial frontier system. Living documentation can change after the review date.