Learn the AI-fabric stack on hardware you already own.
Generate the topology a GPU cluster is wired in, make one priority lossless, change what the switch does, run real RDMA verbs over soft-RoCE, and put Ultra Ethernet frames on a wire you can tcpdump. Five rungs, one machine, no ConnectX and no Tomahawk. Every section says where the model stops.
This is the part of Labtris with no equivalent elsewhere. You cannot do any of it on containerlab, GNS3, EVE-NG or CML — not because they are worse, but because none of them are trying to.
Six scales, one front door. Labtris emulates four and models two.
Scale-in, up, out and across are how the industry names the tiers: the package, the rack, the fabric, the sites. Above and edge are our names for the two shapes past that — regions over transit, and edges on a last mile. Each carries different traffic, fails differently, and is built by a different team. Four run real packets on real wires here. The two inside the rack cannot, by anyone, on a box without the silicon; those are modelled, and labelled as such.
Scale-in
modelled, no wireDie to die inside one package — the chiplet interconnect. Marvell's term; the tier below scale-up.
Modelled only, as the innermost dimension of the astra-sim node's hierarchy. The image ships a three-dimension example — 2 dies × 4 packages × 2 nodes, a 1 MB all-reduce — and the die-to-die bandwidth moves the answer (46,544 cycles at 5 TB/s, 46,984 at 500 GB/s). The numbers are Blackwell-class figures from public specifications, not a validated model; nothing on this tier can be, on a box without the silicon.
Scale-up
modelled, no wireMemory-semantic load/store between accelerators inside one rack — NVLink and NVSwitch, UALink, and since March 2026 OCP's ESUN, which puts it on Ethernet.
Modelled for the timing, emulated for the parts that are Ethernet. There is no open UALink implementation, so nothing here speaks that protocol; the astra-sim node models the domain instead — give it a collective and a rack and it tells you how long the collective takes and how much of it left the rack. Its 8-GPU HGX H100 model is validated against real hardware. ESUN is different: its lower layers are Ethernet with PFC, credit-based flow control and link-level retry, it pulls in UEC 1.0 by reference, and the three things its validation talks name — lossless transport, tail latency, interop between vendors — are what the pfc-classes and uet-pair pods exercise on real veths. The 4-byte ESUN header itself has no open implementation yet.
Scale-out
real traffic, real wireHigh-radix, lossless connectivity between racks, carrying the collectives that synchronise training.
Rail-optimised and fat-tree generation, PFC and ECN on a chosen priority, real RDMA verbs over soft-RoCE, and Ultra Ethernet frames on the wire via the UEC's own reference provider.
Scale-across
real traffic, real wireConnectivity between data-centre clusters, so one workload can span sites.
Routed and VXLAN interconnect between pods. A 931-node fat tree was verified at 182 of 182 ordered pod pairs over a BGP EVPN overlay.
Scale-above
real traffic, real wireRegions that are a public transit apart, not one operator's DCI. Not an industry term; the name is ours for the shape above scale-across.
The scale-above pattern: N regions, each peering with every one of T transit routers over eBGP. Each transit is a step further than the last and every return leg a step longer than the outbound, so the path between two regions is BGP's choice and the two directions need not match. Stop a transit and it re-routes rather than partitions. Ships as a pod: three regions, two transits, configs seeded, converges on boot.
Scale-edge
real traffic, real wireMany small sites on bad links — inference at the edge, reached over a last mile rather than a fabric. Also our name, not the industry's.
The scale-edge pattern: a core with an inference-api behind an access switch, and N edge sites, each a router and its hosts. Each edge link carries a different impairment preset — satellite (550 ms, 2 Mbit), 3G (150 ms, 1% loss), Wi-Fi (jitter), lossy WAN (5% loss) — so the same request arrives four different ways. Real tc on real veths; the pod ships with four edges.
Front-end
real traffic, real wireThe ordinary north-south network: users, storage, the internet, inference requests arriving.
NAT, a cloud NIC onto a physical segment, uplink bridges, and the management network. Ordinary networking, and Labtris has always done it.
Why scale-up is modelled and not emulated. Ultra Ethernet works here because it rides on Ethernet, so a veth is a legitimate link, and because the Ultra Ethernet Consortium published a reference implementation under MIT that you can actually run. OCP’s ESUN has the first of those and not yet the second: its 1.0 specification (March 2026) keeps the lower layers Ethernet — PFC, credit-based flow control, link-level retry — and includes UEC 1.0 by reference, so the lossless behaviour a validation session at OCP asks about is testable on a veth today, while the compact 4-byte header that replaces IP and UDP has no open implementation to package. UALink has neither. Its PHY is an Ethernet one \u2014 the 200G IEEE P802.3dj layer \u2014 so the wire is not the obstacle. What rides on it is: memory semantics, load and store and atomics, carried by protocols like Infinity Fabric and CXL rather than by IP. A veth will carry Ethernet frames all day and still has nothing that resembles a cache-coherent transaction. And although the 1.0 specification is public and silicon is being built, there is no open implementation to package, so a node claiming to speak it would be a guess with no reference to check against. There is not one. What this page previously said was worth building instead — a behavioural model of a scale-up domain, and above all the question of when traffic leaves the rack and lands on the scale-out fabric — turned out to already exist. The astra-sim node packages it: a hierarchy whose first dimension is the scale-up network and whose outer ones are the fabric above. Its validated 8-GPU HGX H100 model answers in cycles, not packets. That is the same bargain the uet-htsim node makes, and it is stated for the same reason. The distinction is the one this whole page is built on: is the protocol correct and is the algorithm good are different questions, and a lab is only useful if it is clear which one it is answering.
Generate the topology a GPU cluster actually uses
Rail-optimised is how accelerator clusters are wired, and it is not a Clos: each NIC on a host goes to its own rail, and the rails do not talk sideways. Fat-tree is k-ary Al-Fares — (k/2)² core, k²/2 aggregation, k²/2 edge, k³/4 hosts. Both arrive with an FRR BGP EVPN underlay already written into every node.
labtris lab generate $LAB --pattern rail-optimised \
--rails 4 --hosts-per-rail 8 --with-bgp-evpn- Why a rail fabric has no east-west path between rails, and what that costs you when one rail congests.
- What an unnumbered eBGP underlay with
advertise-all-vnilooks like in real FRR config, across sixty nodes, without typing it. - The per-rail iBGP islands rail-optimised needs and spine-leaf does not.
The generator defaults spines to bmv2, which carries no FRR to configure — pass --spine-kind frr if you want routing on them.
Make one priority lossless and watch the other one suffer
This is the lesson RoCE is built on, and the one you cannot demonstrate without per-priority queues. Turn on PFC for priority 3, flood priority 0 with TCP, and the RDMA traffic keeps going. Turn it off and it does not.
{ "pfc": true, "pfc_priorities": [3, 4], "rate_kbit": 100000 }- Head-of-line blocking as an experiment rather than a diagram: eight strict-priority bands, and what happens to class 3 when class 0 fills.
- Why ECN marking is the signal and a drop is the failure — RED marks CE while the queue average sits inside a band, which is exactly what DCTCP and DCQCN are reading.
- How
tbf → prio → redcomposes, since rate limits and netem hang above the priority tree rather than beside it.
Labtris ships the queueing half of 802.1Qbb, not the PAUSE-frame half. Classes are isolated and ECN-marked; no MAC-control PAUSE is emitted, so a sender is never explicitly told to stop. That needs XDP and is not built. It is the half that makes RoCEv2 usable, not the half that makes “lossless” literally true.
Change what the switch does, under a running lab
bmv2 nodes take a P4 program each. Four ship: a plain L2/L3 switch, ECMP over a five-tuple hash, ECN marking above a queue threshold, and UEC-style packet trimming. Swap one in and the dataplane changes beneath a lab that is already up.
labtris node p4 set <id> --builtin ecmp- What packet trimming is for: UEC drops the payload and forwards the header, so the sender hears about congestion from the receiver instead of from a timeout.
- How a five-tuple ECMP hash distributes — and why per-flow hashing is the thing UET’s per-packet spraying exists to escape.
- Reading and editing real P4. The programs live in
packaging/p4-programs/and are short enough to follow in one sitting.
No P4Runtime and no simple_switch_CLI yet, so tables are populated from the program’s own apply block rather than installed at runtime.
Run real RDMA without an RDMA NIC
The rdma-host node is Ubuntu with rdma-core, ibverbs-utils and perftest baked in. Soft-RoCE puts a real ibverbs device on an ordinary veth, so ib_send_bw — and your own verbs code — runs against a real RoCEv2 flow between two lab nodes, on a machine with no RDMA hardware in it. Measured at 713.85 MB/s average over a Labtris bridge — 64 KB messages, 1000 iterations, mainline 7.2.
rdma link add rxe0 type rxe netdev eth0 # host needs: modprobe rdma_rxe
rdma link show # state ACTIVE means verbs are live
ib_send_bw -d rxe0 # on a
ib_send_bw -d rxe0 <a-ip> # on b- The actual ibverbs API — queue pairs, completion queues, memory registration — against a device that behaves like one, because it is one.
- What RoCEv2 looks like on the wire. It is UDP port 4791; capture the link and read it.
- Why RoCE wants a lossless class — combine this rung with rung 02 and watch what a lossy class does to it.
Needs Linux 7.1 or newer on the host. Before 7.1, rdma_rxe created its UDP 4791 socket only in the host namespace, so a device made inside a container was bound to a socket that did not exist there — the transfer connected, exchanged GIDs and then moved no packets. Check with ss -lun | grep 4791: a per-namespace listener means you are fine, one global 0.0.0.0:4791 means too old. Ubuntu 24.04 and 26.04 both ship kernels that are too old; a mainline 7.1+ works. Separately, soft-RoCE is a software device: the figure above is 64 KB messages on one host, and throughput moves a long way with message size and CPU, so treat any single number as that run rather than a spec. It is nowhere near a ConnectX either way — this teaches and tests RDMA code rather than benchmarking hardware.
Put Ultra Ethernet frames on a real wire
uet-ref builds the Ultra Ethernet Consortium’s own reference provider — SES with tagged and untagged messages and RMA, PDS, TSS with AES-GCM, all four delivery modes, and a partial implementation of UET congestion control. It runs over a raw Ethernet socket or AF_XDP on an ordinary veth, so a pair of containers on one segment is the whole requirement. UET is its own transport on IP protocol 253, not UDP.
UET_IFNAME=eth0 UET_PDS=pds uet server rma 10.77.0.2 # node a
UET_IFNAME=eth0 UET_PDS=pds uet client rma 10.77.0.1 # node b
scripts/uet_test.sh client eth0 10.77.0.1 rma pds rawsock- Thirteen operations on one pair — RMA, tagged and untagged messaging, unexpected messages, deferred sends, atomics and datagrams — across ten PDS configurations and two NIC shims.
- What each delivery mode actually costs: ROD restores order and pays in latency, RUDI drops the context entirely, UUD leaves sequence gaps that nothing repairs.
- Why spraying matters, by watching
uet.entropy.entropyhold one value for every packet — multi-path delivery is upstream’s stated gap, and the capture shows it in a single command.
Know what the numbers are worth. Packet-level facts — sequence gaps, retransmissions, NACK codes, whether trimming fires — are exact and identical on silicon. Absolute throughput measured on a veth is not: there is no serialisation delay, no ASIC and no real buffer. UET_PDS also defaults to sng, stop-and-go, so a throughput figure taken without UET_PDS=pds measures stop-and-wait rather than UET. Our older ue-stack node remains a proof of concept with no congestion control; prefer uet-ref.
“Claude and containerlab can already do this.”
For building a topology and poking at it, that is largely true, and worth saying out loud. An agent with a shell can write a .clab.yml, run containerlab deploy, docker exec into a node and read its logs. If that is the job, you do not need us for it.
The difference is not that our assistant is cleverer. It is that it can reach things that do not exist in the other stack — an agent cannot ask for 150 ms in one direction only, a P4 program swap, a RAM-inclusive snapshot, or a screenshot of a booting appliance if nothing underneath it implements those. Everything on this page is in that category.
- It reads structure, not scrapings.
link_statsandnode_logsreturn the lab as modelled. A shell agent reconstructs the topology by parsingip -d linkand guessing which veth is whose. - It can drive a screen.
vnc_screenshot,vnc_typeandvnc_readlet it operate a graphical appliance with no SSH. That follows from running real VMs, which containerlab does not. - It is bounded. Tools run as the signed-in user, and the destructive ones ask first. Handing an agent a shell on the lab host hands it the host.
So the honest version: come for the substrate, not the assistant. The assistant is how you drive it.
Three images you build yourself.
Rungs 01 to 03 work straight out of an install. Rungs 04 and 05 want two things first: the node images, which are not on a registry yet and build from Dockerfiles in the repo, and — for RDMA only — a host kernel of Linux 7.1 or newer.
There is a four-node pod in the repo — packaging/demo-pods/rdma-uet-demo.pod.tar.gz — with two soft-RoCE hosts and two Ultra Ethernet endpoints on one segment, so rungs 04 and 05 are a lab load rather than a build.
ss -lun | grep 4791 # per-namespace listener = 7.1+, good
# one global 0.0.0.0:4791 = too old
docker build -t labtris/rdma-host:latest packaging/dockerfiles/rdma-host/
sudo modprobe rdma_rxe
echo rdma_rxe | sudo tee /etc/modules-load.d/rdma_rxe.confPublishing these is on the list; until then it is an honest extra step rather than a hidden one. The contact address is the place to push it up the queue.