People say “the network” about an AI cluster as if it were one thing. It is six, and they are built on different physics, fail in different ways, and are tested with different tools. Here they are, innermost first, each with what one machine can say about it. Every number below was measured this week on a 36-core Hetzner box; where a tier can only be modelled, the post says so and gives the model’s number instead.
The six
- Scale-in — die to die inside one package. Marvell’s term. Terabytes a second over millimetres.
- Scale-up — accelerators inside one rack doing loads and stores to each other’s memory. NVLink, UALink, and since March OCP’s ESUN, which puts it on Ethernet.
- Scale-out — the lossless Ethernet fabric between racks that carries the collectives. RoCE today, Ultra Ethernet next.
- Scale-across — the DCI between buildings or sites, so one job can span them. Routed, with an overlay.
- Scale-above — regions a public transit apart. Our name; the industry has not settled one.
- Scale-edge — many small sites on bad links, where inference actually runs. Also ours.
The first three are the industry’s own tiers; the last two we named because the shapes exist and nobody had. The exercise: take one 1 MB all-reduce across 16 accelerators and follow it out.
Scale-in and scale-up: modelled, in cycles
Nothing on a server without the silicon can emulate a die-to-die link or NVLink. What it can do is model them, and ASTRA-sim’s analytical backend takes a hierarchy of exactly these tiers: a topology, a count and a bandwidth per dimension. Two dies per package fully connected at 5 TB/s a direction, four packages per node on a switch at 900 GB/s, two nodes on a fabric switch at 50 GB/s. Blackwell-class figures from the public specifications, not a validated model — the validated one it ships is an 8-GPU HGX H100 — and that caveat is the point of this tier: the number is only as good as the inputs.
topology: [ FullyConnected, Switch, Switch ]
npus_count: [ 2, 4, 2 ] # dies, packages, nodes
bandwidth: [ 5000.0, 900.0, 50.0 ] # GB/s per direction
latency: [ 100.0, 936.25, 2000.0 ] # ns
1 MB all-reduce, 16 NPUs
baseline 46,544 cycles
die-to-die cut to 500 GB/s 46,984 cycles (+0.9%)
fabric cut to 25 GB/s 48,912 cycles (+5.1%)Read that table once and you have the whole argument for why scale-out gets the attention: halving the fabric costs five times what cutting the die-to-die link by ten does, because that is where the all-reduce spends its time once it leaves the package. The model tells you how much of the collective left the rack. Everything from here down is about what happens to it after it did.
One correction to what we wrote on the fabrics page a month ago. ESUN’s 1.0 specification keeps the lower layers Ethernet — PFC, credit-based flow control, link-level retry — and includes UEC 1.0 by reference. So the lossless behaviour the validation talks at OCP this week are about is the same thing the next tier tests, and it runs on a veth. The compact ESUN header that replaces IP and UDP has no open implementation yet. UALink still has neither.
Scale-out: real packets, and a real ceiling
This is the tier where the box stops modelling and starts forwarding. Two things were measured. The first is what the virtual NIC itself can carry, because every figure above it is bounded by that. Two Ubuntu guests, one link, iperf3 between them, with the one setting that changes the answer: how many queues the virtio NIC has and whether vhost moves the packets in the host kernel.
1 queue, no vhost 4 queues + vhost
TCP, 1 stream 27.7 Gbit/s 35.9 Gbit/s
TCP, 4 streams 26.3 Gbit/s 69.0 Gbit/s
UDP 64 B, 4 streams, rx 0.31 Mpps 0.84 MppsSingle-stream TCP barely moves because TSO already hands the kernel 64 KB segments; the queues matter when there are flows to spread, and for small packets, which is what a collective looks like on the wire. nic_queues: 4 on the node is the whole change. The second measurement is a device under test in the path: FD.io VPP as the router, Cisco TRex as the generator, both as container nodes on af_packet, 64-byte UDP both ways.
offered per direction forwarded where the loss was
0.5 and 1 Mpps 100 % -
2 Mpps 98.7 % the veth's ring into VPP
3 Mpps 99.3 % same; VPP's counters show
4 Mpps 99.0 % every frame it received went outThe honest sentence about that table: the ceiling is the software port’s, not VPP’s. Three things had to be found before it came out at all. TRex’s default 512-frame rings lose packets at 150 kpps. VPP’s interrupt receive mode overflows at a megapacket because the worker sleeps between bursts. And a first run showed loss at 500 kpps that turned out to be a second VPP still running on the same pinned cores — neither reported anything. Each of those is now written next to the config it fixed, because a lab that gives you a number without the three ways the number was wrong first is not worth much.
What a run like this tells you is whether the device forwarded what it was handed, and vppctl show runtime tells you which graph node took the cycles. Line rate is a question for a NIC, and we say so on the node’s own palette card.
Scale-across: the overlay survives
Two sites, a slow hop between them, slower one way than the other because real DCI is not symmetric. The question is not bandwidth here; it is whether the fabric’s overlay still converges and still forwards when the path underneath changes. The 931-node fat tree from the previous post is the large version: 182 of 182 ordered pod pairs over a BGP EVPN overlay, 0% loss. The pod that ships is the eight-node version of the same shape, seeded so it converges on boot.
Scale-above: BGP picks, and the directions differ
Three regions, two transit routers, every region on both. Each transit a step further than the last, and every return leg a step longer than the outbound, so the route between two regions is BGP’s choice and the reverse path need not match. Stop a transit and traffic re-routes; it does not partition. This is the tier the industry has not named because it is the internet, and an AI job that spans it inherits every asymmetry the internet has.
Scale-edge: the same request, four ways
One core with an inference API behind an access switch, and four edge sites, each on a different last mile: satellite at 550 ms and 2 Mbit, 3G at 150 ms and 1% loss, Wi-Fi with jitter, a lossy WAN dropping one packet in twenty. Real tc on real veths. The same request arrives four different ways, and what a model serving at the edge does about that is a software question the lab can now ask.
What the box can and cannot say
Two tiers modelled in cycles, four emulated with packets, and the line between them stated at every step. The modelled tiers give you a number whose provenance you can audit in a 30-line YAML file. The emulated tiers give you behaviour — did it converge, did it forward, where did the time go — up to the rate a software port carries, which on this box is about 3 Mpps a direction through a full VPP graph and 69 Gbit/s of TCP between two guests. The expensive tester, the real 800G NIC, the silicon: those answer the question above that line. What a lab answers is everything below it, before a cable exists, and that is most of the questions.
Every pod named here ships with Labtris and loads with one command; the numbers are reproducible on your own box with the fabrics page as the map.