AI Infrastructure Landscape

The vocabulary

Vertical AI

Software built around one industry's workflow, where the model is a component and the domain data is the moat.

Agentic workflow

A task the software completes end to end, rather than a suggestion a person then acts on.

System of record

The place the work's data actually lives, which decides who can be displaced and who cannot.

Outcome pricing

Charging per unit of work finished instead of per user seat.

The vocabulary

MCP

Model Context Protocol: an open standard for how a model reaches tools and data.

Agent memory

Storage that carries state across runs, as opposed to a context window that resets each time.

Harness

The layer that decides which tool an agent may call, with what arguments, and what happens when it fails.

Non-human identity

Credentials belonging to an agent or workload rather than a person.

The vocabulary

Post-training

Everything after pre-training: fine-tuning, reinforcement learning, distillation. Where most differentiation now happens.

Eval

A repeatable test of model behaviour, used as a release gate the way a test suite is.

Distillation

Training a smaller model to imitate a larger one, to cut the cost of serving it.

Context window

How much input a model can hold in view at once.

The vocabulary

RAG

Retrieval-augmented generation: fetching relevant documents at query time instead of baking knowledge into weights.

Vector database

Storage indexed by embedding similarity rather than exact match.

Lakehouse

A warehouse and a data lake in one system, so analytics and training read the same tables.

Synthetic data

Generated training examples, used where real data is scarce or restricted.

The vocabulary

Token

The unit a model reads and writes. Inference is billed by it, so cost per token is the unit economics.

Throughput vs latency

Tokens per second across every user, against the wait a single user actually feels.

Speculative decoding

Drafting tokens with a small model and verifying with a large one, to cut latency without changing the answer.

Cold start

The delay while a model loads before it can serve its first request.

The vocabulary

Cluster

Thousands of accelerators wired together tightly enough to train one model as a single job.

Checkpoint

A saved snapshot of weights, so a failed run resumes instead of starting over.

Interconnect

The network between accelerators, which is usually the real limit on how far a run can scale.

RL pipeline

The loop that turns a base model into one that follows instructions and refuses what it should.

The vocabulary

Neocloud

A cloud built specifically for accelerators, competing with hyperscalers on price and availability.

Utilisation

The share of a paid accelerator actually doing work. The industry's largest source of waste.

Bin-packing

Fitting jobs onto hardware so that little capacity sits idle between them.

Multi-tenancy

Letting several customers share one cluster without leaking data or performance into each other.

The vocabulary

GPU, TPU, ASIC

General-purpose, Google's tensor chip, and fixed-function silicon. Specialisation buys efficiency and gives up flexibility.

HBM

High-bandwidth memory stacked beside the processor. Usually the component in shortest supply.

Wafer-scale

Building one very large chip from an entire wafer instead of cutting it into many small ones.

FLOPs per watt

The efficiency figure that decides what a given building can actually run.

The vocabulary

Behind-the-meter

Generating power on site, which sidesteps the queue to connect to the grid.

Interconnection queue

The wait to attach a large load to the grid, now measured in years rather than months.

PUE

Power usage effectiveness: total facility power divided by the power that reaches the computers.

Direct-to-chip cooling

Bringing liquid to the processor itself, because air stops working above a certain rack density.

The vocabulary

Non-human identity

Credentials held by a service, workload or agent rather than a person. They now outnumber human accounts.

Prompt injection

Instructions hidden inside content an agent reads, which the agent then follows as if they came from its operator.

Model provenance

Evidence of where a model's weights came from and what has been changed in them since.

Runtime guardrail

A check that approves, modifies or blocks an agent's action in the moment before it runs.

The vocabulary

Tracing

Recording each step of a model or agent run, so a failure can be reconstructed afterwards.

Eval harness

A fixed suite a model must pass before it ships, and again after every change.

Drift

Behaviour changing over time while the code stays the same.

Groundedness

Whether an answer is actually supported by the sources it cites.

The vocabulary

East-west traffic

Traffic between machines inside a data centre, which now dwarfs traffic in and out of it.

Scale-up vs scale-out

Connecting accelerators inside one rack, against connecting racks to one another.

InfiniBand and RoCE

Two routes to low-latency networking: purpose-built, and built on Ethernet.

Optical interconnect

Using light rather than copper between chips, to move more data per watt.

The vocabulary

Throughput per dollar

The real metric for training storage, in place of capacity per dollar.

Checkpointing

Writing model state periodically. A write pattern most storage systems were never designed for.

Object vs parallel file system

Cheap and vast, against fast and tightly coupled to the compute.

Tiering

Moving data between fast and cheap media automatically, by how often it is read.

The vocabulary

Risk tier

How a regulation classifies a system, which in turn decides the obligations attached to it.

Model card

A published description of what a model is, how it was trained, and where it is known to fail.

Audit trail

A record complete enough to reconstruct why a particular decision was made.

Conformity assessment

Demonstrating before deployment that a system meets a standard, rather than after an incident.

The vocabulary

Licensed corpus

Training data acquired under contract, as opposed to scraped from the open web.

Crawler signal

A publisher's machine-readable instruction about crawling and training on its content.

Provenance

The record of where a piece of training data came from and on what terms.

Synthetic data

Model-generated examples used in training where real examples are scarce or restricted.

The vocabulary

PPA

Power purchase agreement: a long contract to buy electricity, often the thing that finances new generation.

SMR

Small modular reactor. Factory-built nuclear, sized for a single large load rather than a region.

Uprate

Increasing an existing plant's output without building a new one. The fastest capacity available.

Curtailment

Power that is generated but cannot be used or moved, and is thrown away.