AI-chip system architecture

Frontier models · The physical computing stack

A frontier model runs on a distributed machine. Its performance depends on how well arithmetic, memory, networks and software work together—not simply on how many AI chips are installed.

Deep research guide · Evidence checked 6 September 2026 · Approximately 25 minutes
Scope: data-centre training and inference, from accelerator internals to rack-scale systems.

The essential idea

An AI accelerator is a fast calculator surrounded by a data-delivery system. More arithmetic capacity helps only when the workload can feed it.

Training, reading a prompt and generating a response stress different parts of that system. Memory capacity determines what fits; bandwidth determines how quickly data moves; interconnects determine how efficiently chips cooperate.

The useful comparison is the cost of meeting a workload’s quality, speed and reliability requirements. A chip’s peak FLOPS is only one input.

This guide connects the mechanisms in How frontier models work to the hardware that executes them. It focuses on publicly documented designs, using NVIDIA, AMD and Google as architectural examples rather than a league table. Semiconductor fabrication and lithography are separate subjects; here, packaging matters because it changes the machine’s data paths.

1. From model to distributed machine

A model describes calculations and learned parameters. A system architecture decides where those parameters live, which processors execute each calculation, and how intermediate results travel between them. These are different kinds of architecture: changing the hardware does not necessarily change the model, but it can radically change its operating cost.

The system, in four layers

Model & workloadLayers, tokens, precision, batch size and response-time requirements.
Accelerator & memoryMatrix engines, local SRAM and high-bandwidth memory.
Connected systemHosts, accelerator links, switches, network adapters and storage.
Operating environmentScheduling, power delivery, cooling, monitoring and recovery.

An explanatory map, not a fixed physical layout. Software coordinates all four layers; data movement crosses them in both directions.

Physical labels need care. A die is one piece of silicon. A package can contain several dies and memory stacks. A server or compute tray combines processors and supporting components; racks and clusters join more of these units. Software can expose the result differently: Blackwell joins two compute dies into a logical GPU, while Ironwood exposes its two chiplets as separate devices in JAX. NVIDIA’s Blackwell brief and Google’s Ironwood documentation illustrate why “per chip” needs a definition.

Consider a document-summary request. The system admits the request, prepares tokens, processes the prompt, retains attention state, and generates output. A training run adds a different loop: calculate a loss, propagate gradients, combine distributed updates and change parameters. The chips execute pieces of these workflows; the surrounding system determines how much time they spend waiting.

2. Training, prefill and decode are different workloads

“AI compute” covers several jobs. A useful first question is not which chip is fastest, but which phase needs to become faster.

Same model family, different system pressures
Phase	What happens	Common pressure
Training	Forward calculations, backward gradients and parameter updates across batches.	Arithmetic, training-state memory, activation storage and distributed communication.
Prefill	The available prompt tokens are processed together to prepare the first output.	Often arithmetic-intensive; long-context attention and prompt queues also matter.
Decode	Successive output tokens use the model and accumulated attention state.	Often memory-bandwidth-sensitive at small batches; latency and growing KV state matter.

The training-state distinction is documented in DeepSpeed’s ZeRO explanation; the prefill/decode split is central to DistServe’s serving-system analysis. These are tendencies, not laws. Larger decode batches reuse weights across more requests and can make parts of the calculation compute-bound. Long contexts increase attention work and memory traffic, while short operations may be limited by launch or communication latency.

In ordinary autoregressive decoding, the next token depends on previous output, so one request cannot simply generate every position independently. Parallelism across requests improves total throughput, but does not automatically shorten one user’s response. A system serving many conversations and a system optimised for one very fast conversation can therefore favour different configurations.

Reasoning increases the importance of this distinction: generating more intermediate tokens consumes more decode steps and extends the lifetime of request state. Multimodal inputs can also expand the sequence or introduce additional encoders. Hardware demand depends on the actual computation path, not merely the length of the answer visible to the user. The underlying generation mechanics are developed in the JAX scaling-book inference chapter.

3. Inside an AI accelerator

CPUs coordinate; accelerators execute parallel work

A CPU remains useful for control flow, application logic and coordinating device work. A GPU runs many parallel threads organised into execution groups. NVIDIA calls its larger execution blocks streaming multiprocessors; they include arithmetic resources, registers and shared local memory. The CPU launches GPU programs called kernels, and the two processors can work concurrently. NVIDIA’s CUDA programming model documents this host–device arrangement.

“GPU cores” are not interchangeable with CPU cores. Their organisations and capabilities differ, so comparing raw core counts across products says little about model performance. The more useful questions concern supported operations, precision, data reuse and the workload’s ability to occupy the available execution resources.

Matrix engines do the repetitive arithmetic

Many neural-network operations multiply matrices: an array of activations is transformed by an array of weights. Dedicated matrix engines perform many multiply-and-accumulate operations efficiently. General-purpose vector and scalar resources handle other work, including activations, reductions, indexing and control.

A matrix multiplication is normally split into tiles. A kernel loads a manageable block, reuses its values near the arithmetic units, and writes results back. Larger tiles can improve reuse but also consume more local storage and reduce the number of simultaneously active blocks. This is why the shape of a matrix and the selected kernel can matter as much as its total operation count. NVIDIA’s matrix-multiplication guide explains this trade-off.

GPU versus ASIC is not a simple flexibility switch

A TPU is a machine-learning application-specific integrated circuit, or ASIC. Its matrix units use systolic arrays, passing data through neighbouring arithmetic elements to perform repeated multiply-accumulate work. It also has vector and scalar units. Google’s TPU architecture guide describes this division of labour.

Specialisation does not mean a device can execute only one model. It means its design spends silicon and engineering effort on a particular workload domain. GPUs also contain highly specialised matrix engines. The practical distinction involves the supported operations, compiler, memory organisation, deployment model and software ecosystem—not simply “programmable” versus “fixed.”

4. Memory is a hierarchy, not one pool

Arithmetic units need a continuous supply of values. Different memory tiers trade capacity, locality and transfer speed.

Where model data can live
Tier	Typical role	Architectural constraint
Registers & on-chip SRAM	Small, frequently reused working values and matrix tiles.	Limited capacity; allocation affects how much work can run concurrently.
HBM	Weights, activations and attention state close to the accelerator.	Capacity and sustained bandwidth are separate limits.
Host memory	CPU working data, staging and offloaded model state.	Reaching it adds a transfer path; it is not equivalent to local HBM.
SSD & network storage	Datasets, model files, checkpoints and cached state.	Useful persistence and capacity, with a different access-time and bandwidth budget.

The specific tiers and access paths are visible in DGX B200’s hardware guide and Ironwood’s memory hierarchy. Remote memory may be addressable without being equally fast. Likewise, adding together all GPU memory in a cluster does not create one unrestricted local allocation; placement and communication still matter.

Why HBM and packaging belong in the same story

High-bandwidth memory is stacked DRAM. Multiple memory dies connect vertically through tiny interconnections, including through-silicon vias. The stacks sit alongside processing logic in an advanced package, allowing a short, wide data interface. It is physically and functionally different from on-chip SRAM. Micron’s HBM technical overview explains the construction.

Packaging lets designers assemble a processor from specialised pieces. AMD’s CDNA 4 design combines eight compute chiplets, two I/O dies and eight HBM stacks. Compute and I/O use different manufacturing processes: the fastest logic and the memory/interface functions do not benefit identically from process scaling. The CDNA 4 white paper makes this explicit.

The implication is that a smaller advertised process node cannot, on its own, describe a better AI system. Logic density, memory stacks, package connections, cooling and software must work together. This is the bridge between model architecture and the semiconductor value chain.

5. What actually fits in memory?

Start with weights, then add the working state

The raw weight-storage calculation is simple: parameter count multiplied by bytes per parameter. It is only the beginning of a deployment budget.

Worked example: a hypothetical 70-billion-parameter model

At 16 bits per stored weight, the raw weights occupy 140 GB. At 8 bits, they occupy 70 GB. At 4 bits, the raw payload is 35 GB.

These are decimal gigabytes and arithmetic illustrations, not complete model-file sizes. Quantisation scales, padding, unquantised tensors and runtime allocations add overhead. The figures do not establish that a model will fit on a particular GPU.

Training also needs gradients, optimizer state and activations. The amount depends on precision, optimizer and implementation. ZeRO-style techniques partition training state across devices; activation checkpointing instead saves memory by recomputing selected intermediate results during the backward pass. Both alter the time–memory trade-off. See DeepSpeed’s state-partitioning explanation and Korthikanti and colleagues on activation recomputation.

The KV cache is the second major inference budget

An attention-based decoder retains keys and values for previous positions so later tokens can use them without recomputing all earlier projections. Unlike weights, this KV cache grows with the number and length of active requests.

Conventional full-attention KV bytes

2 × layers × stored tokens × KV heads × head dimension × bytes per value

The factor of two represents keys and values. Sum across requests; this is logical storage before implementation overhead or replication.

Worked example: context competes with concurrency

Assume 80 layers, 8 KV heads per layer, a head dimension of 128, 16-bit cache values and 8,192 stored tokens. The formula gives 2.5 GiB per request. Thirty-two such requests need 80 GiB of KV storage, before weights and other allocations.

At 131,072 stored tokens, the same architecture needs 40 GiB per request. These are illustrative inputs, not a claim about a named frontier model. Here GiB means 2³⁰ bytes, while GB means 10⁹ bytes.

The formula follows the JAX inference chapter. It is not universal: grouped-query attention shares KV heads, sliding-window layers retain less history, and latent-attention architectures use different representations. The GQA paper explains head sharing; DeepSeek-V3’s technical report documents a latent-attention design.

The operational consequence is straightforward: a longer advertised context window does not imply that every user can occupy that window simultaneously at unchanged cost. Capacity planning needs a distribution of real request lengths, not just the model’s maximum.

6. Why peak FLOPS can mislead

FLOPS measures floating-point operations per second. Bandwidth measures bytes moved per second. A processor can have ample arithmetic capacity and still wait for its inputs.

Arithmetic intensity is the amount of computation performed for each byte moved across a specified memory boundary. The roofline model gives a useful simplified ceiling:

Attainable FLOPS ≤ the smaller of:

Peak compute throughput
Memory bandwidth × arithmetic intensity

This is a diagnostic model, not a performance forecast. Small kernels, unfavourable access patterns, communication and scheduling can lower achieved performance further. NVIDIA’s GPU performance guide describes these limiting cases.

Worked example: a fast chip waiting for data

Imagine a device with a peak of 1,000 TFLOPS at the chosen precision and 4 TB/s of relevant memory bandwidth. A kernel performing 50 operations per byte has a bandwidth roof of 200 TFLOPS: 4 trillion bytes/s × 50 operations/byte.

Doubling the chip’s peak arithmetic rate would not raise that roof. Increasing useful data reuse, reducing bytes transferred, or increasing sustained bandwidth could.

For a separate, simplified decode illustration, reading 140 GB of weights through a 4 TB/s path takes at least 35 milliseconds. This assumes one full weight read per step with ideal bandwidth; it excludes KV traffic, arithmetic and other overheads. It is not a measured token rate or a general bound for every architecture.

Batching can reuse a weight tile across multiple requests, increasing arithmetic intensity. But every request still brings its own state and latency requirement. Software may also reduce traffic directly: FlashAttention reorganises exact attention into tiles to avoid unnecessary transfers between HBM and on-chip SRAM. It does not remove all long-context attention computation or promise the same speedup on every system.

The useful troubleshooting question is therefore: which resource is limiting this operation at this batch size? An average GPU-utilisation number is too coarse to answer it.

7. Precision and quantisation change the machine’s workload

AI systems use several numerical formats. FP16 and BF16 store 16-bit floating-point values; FP8 uses eight bits; various FP4 and integer formats use fewer. A lower bit count can reduce stored data and memory traffic, and supported matrix hardware can execute more low-precision operations per second.

But three questions must be separated: How are weights stored? At what precision is multiplication performed? At what precision are partial results accumulated? A compact checkpoint does not prove that the hardware executes every operation at that same precision. Some tensors or operations remain at higher precision.

Scaling metadata also occupies space. NVIDIA’s NVFP4 format stores an FP8 scale for each group of 16 four-bit values, plus a tensor-level FP32 scale. That is approximately 4.5 bits per value before the tensor-level scale, not exactly four. NVIDIA’s NVFP4 technical explanation provides the format details.

Quantisation can change model quality. Validation should cover the intended workload, difficult cases and long contexts, rather than assuming that a small change on one benchmark applies everywhere. The benefit is useful only if the resulting model still meets its quality requirements.

Structured sparsity is another separate assumption

Some advertised compute rates assume that weights contain a supported pattern of zeros, allowing the hardware to skip work. That is not the same as a dense computation, nor the same as selecting experts in a mixture-of-experts model.

For example, AMD lists MI355X BF16 matrix throughput at approximately 2.5 PFLOPS dense and 5 PFLOPS with structured sparsity. Both are disclosed peaks under different conditions. AMD’s specification table labels them separately. Any comparison must match format, accumulation rules, sparsity assumptions and device count.

8. Connecting the accelerators

Once work crosses device boundaries, moving intermediate values becomes part of executing the model. A fast local GPU cannot compensate indefinitely for slow or congested communication.

Scale-up and scale-out describe domains

Here, scale-up means a tightly connected accelerator domain, while scale-out means joining those domains through a broader network. These are useful operational definitions, not universal physical boundaries. One domain may span multiple hosts or racks, and vendors sometimes use the labels differently.

NVIDIA’s rack-scale systems use NVLink and NVSwitch for tightly connected GPUs. InfiniBand or suitable Ethernet-based fabrics connect larger systems. The DGX networking guide distinguishes the in-rack fabric from the surrounding networks. A rack-scale multi-GPU system remains multiple processors with distributed memory, even when marketed as acting like one large GPU.

RDMA allows supported devices to transfer data to or from remote memory with less CPU involvement in the data path. GPUDirect RDMA can avoid a host-memory copy when a supported network adapter accesses GPU buffers. RoCE provides RDMA over Ethernet; it is not a capability of every ordinary Ethernet installation. The GPUDirect RDMA manual describes the supported arrangement.

The network must match the traffic pattern

Distributed programs use collective operations. An all-reduce combines values and returns the result to every participant. All-gather assembles distributed pieces. Reduce-scatter combines values but leaves each participant a shard. All-to-all exchanges different pieces with different peers. NCCL’s collective-operation definitions explain the distinction.

Latency matters for repeated small exchanges; sustained bandwidth matters for large transfers. The topology determines possible paths. Bisection bandwidth describes capacity across a cut that divides a network into two halves; adding endpoint bandwidth does not necessarily increase that cross-network capacity proportionally. Routing, congestion and collective algorithms determine how much of the physical fabric is useful.

There is no universal conclusion that “Ethernet is slow” or “InfiniBand always wins.” Meta reported using both RoCE and InfiniBand designs for large training clusters and tuning them to comparable performance for its particular workloads. That is a documented deployment result, not a guarantee for another cluster. Meta’s 2024 engineering account explains the surrounding work.

Read the bandwidth denominator

NVIDIA documents NVLink 5 at 1.8 TB/s bidirectional per GPU. Multiplying by 72 GPUs gives approximately 130 TB/s, the advertised aggregate for the NVL72 rack. That is not 130 TB/s available to each GPU, and it is not a measured application-throughput figure.

The per-GPU definition is in the DGX networking guide; the rack aggregate appears in the GB300 reference architecture. Also check bits versus bytes: 800 Gb/s is 100 GB/s before protocol overhead.

9. Dividing the model’s work

More devices provide useful capacity only when the program has a way to share work between them. The common strategies divide different dimensions of the calculation.

What is divided—and what must communicate?
Strategy	Division	Main trade-off
Data parallelism	Different examples run on model replicas.	Training replicas must combine gradient information.
State sharding	Parameters, gradients or optimizer state are partitioned.	Less replicated memory; state must be gathered or exchanged when needed.
Tensor parallelism	Pieces of a layer run on different devices.	Frequent communication inside the model’s forward and backward calculations.
Pipeline parallelism	Different groups of layers form stages.	Transfers between stages; idle gaps when the pipeline is not balanced.
Context parallelism	Sequence positions are distributed.	Less local sequence state; attention needs cross-device information.
Expert parallelism	MoE experts are placed on different devices.	Tokens travel to selected experts and outputs return.

The division of work is documented in Megatron Core’s parallelism guide; state sharding is developed in DeepSpeed’s ZeRO documentation. These techniques can be combined. The best placement keeps frequent exchanges on a suitable fabric while allowing less tightly coupled work to extend across larger domains.

Mixture of experts saves arithmetic, not every resource

A mixture-of-experts model activates only selected experts for each token. It can therefore have many more total parameters than active parameters. DeepSeek-V3, as a documented example, reports 671 billion total parameters and 37 billion activated per token. Its technical report also discusses the communication and deployment design.

The inactive weights do not vanish. They must remain available somewhere, and routing introduces data movement and load-balancing work. Different tokens may favour different experts, making some devices busier than others. The architectural opportunity is to reduce useful computation; the challenge is to avoid replacing it with excessive transfers or waiting.

More parallelism can also become counterproductive. Splitting a matrix too finely reduces work per device while retaining communication and scheduling overhead. A model that already fits and meets latency targets may benefit more from additional serving replicas than from distributing each request across still more GPUs.

10. Software is part of the architecture

The model does not directly issue instructions to every arithmetic unit. Frameworks, compilers, libraries and runtimes turn its operations into executable kernels and coordinate their data movement.

From model code to useful output

DescribeA framework expresses layers, tensors and the computation.
TransformA compiler selects layouts, fuses operations and generates executable work.
ExecuteKernels and communication libraries run on the accelerator system.
Serve & measureThe runtime batches requests, manages state and tracks response quality and speed.

An illustrative software path. Implementations combine compiled, library and custom kernels; no single compiler owns every step.

PyTorch’s compiler stack captures model operations and generates optimised execution through backends such as TorchInductor. XLA is another compiler infrastructure for machine-learning workloads and multiple hardware targets. See PyTorch’s compiler overview and OpenXLA’s documentation.

A change of accelerator therefore involves more than converting a checkpoint. Kernel availability, numerical behaviour, compilation, distributed execution and operating tools need to work together. An application running successfully on two platforms does not establish that it runs equally efficiently on both.

Serving is also a scheduling and memory problem

Requests arrive at different times and have different lengths. Continuous batching lets the runtime admit new work as earlier requests finish, instead of waiting for a fixed group to complete together. The scheduler must balance throughput against waiting time and smooth output delivery.

PagedAttention, introduced with vLLM, manages the KV cache in blocks rather than requiring a large contiguous allocation per request. It reduces fragmentation and enables suitable sharing. It improves how memory is used; it does not erase the underlying state of distinct conversations. The original PagedAttention paper explains the design.

Some systems separate prefill and decode onto different resources. This can reduce interference and let each phase scale independently, but requires transferring KV state and coordinating two resource pools. DistServe studies this trade-off. Whether separation helps depends on prompt lengths, arrival rates, network paths and latency targets—not simply on having enough GPUs.

These optimisations reinforce the same principle: software can change the amount of data moved, the time devices spend idle and the number of requests served within a deadline, even when the silicon is unchanged.

11. Three documented designs, three useful lessons

The examples below explain different architectural choices. They are selected disclosed systems, not an exhaustive list of the newest products or a ranking. Specifications were checked on 6 September 2026; listed bandwidth is a vendor peak, not measured model throughput.

Compare the stated unit before comparing the numbers
Named configuration	Disclosed memory	What it illustrates
NVIDIA DGX B200 Eight-GPU system	1,440 GB GPU memory and 64 TB/s aggregate HBM bandwidth.	System totals are not single-GPU allocations. Dividing by eight gives 180 GB and 8 TB/s per GPU in this configuration.
AMD Instinct MI355X One accelerator	288 GB HBM3E and 8 TB/s peak memory bandwidth.	A chiplet-based package combines compute, I/O, cache and memory; capacity is distinct from arithmetic throughput.
Google Ironwood / TPU7x One chip/package	192 GiB in Google’s specification table; 7,380 GB/s HBM bandwidth.	The two-chiplet design exposes two devices in JAX, with dedicated memory spaces.

Sources: DGX B200 specifications, MI355X specifications and TPU7x architecture. Keep the units as published: Google’s table says GiB, although its accompanying prose says GB. We retain the table’s unit rather than silently treating the two as identical. B200 specifications also depend on the named system; the DGX figures above should not be substituted into every B200 product variant.

Rack versus pod is another change of scale

NVIDIA’s GB300 NVL72 reference design integrates 72 GPUs, 36 Grace CPUs and nine NVSwitch trays. The tightly connected domain is rack-scale. The reference architecture describes the components and their connections.

Google’s Ironwood system instead uses a three-dimensional torus, with a pod footprint of up to 9,216 chips. Its broader design uses optical circuit switches to connect groups of chips and a data-centre network beyond the tightly connected domain. Google’s Ironwood stack article explains these layers.

A torus can execute all-to-all communication without giving every chip a dedicated direct connection to every other chip. Likewise, a larger chip count does not establish lower latency or better economics for a particular model. Topology, partition size, memory locality and software placement belong in the comparison.

12. Power, cooling and reliability are architectural limits

An accelerator deployment is constrained by the power and heat-removal capacity that can actually be delivered. Chip power, board power, server power, rack power and facility power describe different boundaries. A peak specification is not the same as average workload consumption.

For a concrete example, NVIDIA’s GB300 NVL72 reference design is liquid-cooled, includes leak detection and specifies a full-rack requirement of up to 142 kW. This is a requirement for that reference design—not a universal AI-rack number or a measured average. The hardware reference documents the scope.

Workloads also have changing power profiles. Coordinated computation across many GPUs can create correlated power swings; enforced power limits can change achievable performance. NVIDIA’s power-and-thermals guidance discusses these operating constraints.

Reliability turns installed capacity into productive capacity. A long distributed job needs fault detection, checkpoint storage, spare resources and a recovery strategy. Checkpoints cost time and bandwidth; insufficient recovery planning can waste completed computation after a failure. Meta’s training-infrastructure account describes checkpointing, fast restart and reliability engineering at scale.

Our analytical conclusion is that the relevant asset is available, usable computing capacity. A rack awaiting commissioning, a throttled system or a job repeatedly restarting may own the same number of GPUs while delivering very different amounts of work. The wider physical context is covered in AI infrastructure & data centres.

13. Compare delivered work, not the largest headline number

A benchmark needs a workload contract

A meaningful inference comparison states the model and quality threshold, input/output lengths, numerical formats, hardware count, software version, concurrency and latency constraints. It should distinguish total throughput from the experience of one request.

Time to first token includes the wait before the first output arrives. Time per output token describes subsequent generation speed. Tail measurements matter: a good average can conceal poor experience for the slowest requests.

MLCommons illustrates this discipline through MLPerf. Its defined workloads include quality targets and distinguish offline throughput from server scenarios with token-latency constraints. A result belongs to its specified model, dataset, scenario and rules; it does not establish a universal winner. MLCommons’ explanation of its LLM benchmarks describes the measurement choices.

Translate that contract into economics

For owned systems, account for hardware depreciation, networking, storage, power, cooling, software, operations and financing where relevant. For rented systems, start with the actual bill and avoid adding costs already included in it. Include idle and unavailable periods in the cost horizon.

Cost per million useful output tokens

Total system cost over a period ÷ qualifying output tokens over that period × 1,000,000

“Qualifying” means produced under the specified quality and latency requirements. Keep input lengths, output lengths and the workload mix comparable.

Worked example: a higher hourly cost can still be cheaper

Assume two hypothetical systems serve the same workload at the same accepted quality and latency. System A costs £60/hour and sustains 6,000 qualifying output tokens/second: £2.78 per million. System B costs £75/hour and sustains 9,000: £2.31 per million.

These invented inputs illustrate the denominator; they are not market prices or measured product results. If B cannot sustain that rate under the required constraints, the comparison changes.

Token economics are an intermediate measure. Applications ultimately need accepted answers or completed tasks. Extra reasoning, retries, tool calls and human review can change the cost of that outcome even when token-serving efficiency improves.

For training, compare time and total cost to reach a specified quality target, including failed work and recovery—not simply the number of GPUs purchased. For infrastructure suppliers, the corresponding analytical questions are which bottlenecks remain scarce, how readily customers can substitute alternatives and how much of the resulting value suppliers retain. Architecture informs those questions; it does not by itself prove a company has durable pricing power.

14. Questions that reveal the bottleneck

When assessing an AI-chip announcement or deployment, use the following sequence:

What is the workload? Training, prefill, decode, or a mixture—and at what model quality?
What exactly is being counted? Die, accelerator, server, rack or pod?
What fits? Include weights, request state, activations and training state where relevant.
What is limiting speed? Arithmetic, memory capacity, memory bandwidth, communication, scheduling or power?
Which assumptions support the peak? Precision, sparsity, batch size and software.
What happens at real traffic? Variable prompts, tail latency, uneven expert routing and failures.
What is the delivered cost? Include the whole relevant system and measure work that meets the requirement.

The strongest AI-chip system is the one that matches its workload while keeping expensive resources productive. That is why the competitive unit is increasingly the combination of accelerator, memory, fabric, software and operating environment—not the arithmetic engine in isolation.

Glossary

Accelerator: A processor designed to execute selected workloads efficiently alongside a host system.
Activation: An intermediate value produced as data passes through a model.
Arithmetic intensity: Operations performed per byte moved across a specified memory boundary.
HBM / SRAM: Stacked high-bandwidth DRAM near an accelerator / small, fast static memory used on the processing chip.
Kernel: A program performing a particular operation or fused group of operations on an accelerator.
KV cache: Stored attention keys and values reused during generation.
Collective: A coordinated data operation involving a group of devices.
Fabric: The interconnects and switches that carry traffic between components.
Quantisation: Representing values with a more limited set of numerical values, often to reduce storage and computation costs.
Goodput: Useful completed work per unit time under stated requirements, rather than all activity or raw traffic.
GB / GiB: 10⁹ bytes / 2³⁰ bytes. A lower-case b denotes bits; an upper-case B denotes bytes.

Sources and research notes

This guide draws on original research, official architecture and software documentation, and MLCommons’ benchmark explanations. The sources were checked on 6 September 2026. Vendor specifications describe disclosed designs and peak capabilities; they are not independent measurements of application performance.

The arithmetic examples are our own simplified calculations with explicit assumptions. Public information does not establish a universal best accelerator, private customer pricing, actual fleet utilisation or the undisclosed internals of every frontier model. Those limits are intentional; this is an architectural explanation, not a procurement recommendation.

Two specification ambiguities matter: named B200 configurations have different published memory figures, and Google’s Ironwood table and prose use different capacity units. The comparison preserves the named configuration and labelled units. Research was stopped once the explanatory claims were supported and these consequential differences were bounded.

View the 31 research references