The claim was “thousands of nodes on one machine”. This is the test of it, including the half that did not work.
931 nodes, and it routes
A fourteen-pod fat tree: 49 core switches, 98 aggregation, 98 edge and 686 hosts — 931 nodes and 2,058 links, generated in 26 seconds. Every switch runs FRR with bgpd and a BGP EVPN underlay. Every host sits in one VXLAN segment.
It converged, and it forwards:
core (49) peers 14 RIB 225 routes 113
agg (98) peers 14 RIB 237 routes 119
edge (98) peers 7 RIB 477 routes 239 VTEPs 91
931/931 running zero failures
686/686 hosts reach a host in another pod
182/182 pod pairs 0% packet loss
18 GB of 100 GB ~16-20 MB a nodeThose switch numbers are not averages. Every one of the 245 switches reported the same figures, which is what a correct symmetric fabric looks like and what makes an outlier worth chasing when one appears. The audit takes 69 seconds for all 931.
Six bugs, each hiding the next
Getting there meant fixing six things, and the order matters because each one masked the one behind it.
The generated /etc/frr/daemons contained zebra_options= "-A 127.0.0.1 -s 90000000". In shell, VAR= "x" assigns VAR empty and then runs the quoted string as a command. FRR started perfectly well with both daemons running no options at all, and the only symptom was two lines of stderr nobody reads.
Then --with-bgp-evpn turned out never to apply the config it generates: that was a push and an exec, by hand, per node. On 3,267 nodes that is 6,534 operations, so in practice nobody did it and the fabric came up with the image’s stock bgpd=no.
Once applied, the configs established sessions and advertised nothing. All seven config builders wrote a loopback, BGP neighbours and an EVPN block, and not one had an address-family ipv4 unicast. Sessions came up anyway — frr defaults datacenter activates peers — so every switch showed 22 established peers, uptimes ticking, zero errors, and RIB entries 0. A fabric that peers perfectly and carries no routes looks, from every summary command, almost exactly like one that works.
And advertise-all-vni had no VNI to advertise. FRR learns VNIs from the kernel, so with no vxlan device in a bridge it advertises nothing — correctly. Meanwhile all 2,662 hosts had no address at all, so even a working overlay would have carried zero MAC routes.
Where 3,267 broke
The larger run is the more useful one. A k=22 fat tree — 3,267 nodes, 7,986 links — started in seven hours and never finished its configuration pass.
The start path is single-threaded. 7.77 seconds a node, with uvicorn at 88% of one core while netd idled at 5% and load sat at 3.4 across 24 CPUs. Sixteen-way concurrency just queued behind one thread. More cores do not help a saturated single thread.
Applying config restarted FRR on every switch. That is fine on a cold fabric and catastrophic on a converged one. One edge restart touches roughly 253 adjacencies; 242 edges at sixteen at a time put about 4,000 adjacency events in flight. All 24 cores pinned, guest load average 4,044, and sshd starved off the box so completely that the only way to stop the lab was SIGTERM to QEMU — on a host whose API was still answering in under a second.
That last detail is the one worth sitting with. The platform was healthy. What failed was our ability to see it.
The dedup that was not
Labtris has always enabled Kernel Samepage Merging and tuned it 25 times harder than stock, citing a measured 8.5:1. That figure came from fifteen QEMU VMs, and it only ever applied to VMs.
KSM merges only memory a process marks mergeable. QEMU does that for guest RAM. Nothing does it for a container. So on a container host, KSM was on, scanning hard, and merging exactly zero pages — while diagnostics reported “on”, because every knob genuinely was set. Opting the container runtime in changed that from 0 to 10.3 GB at 64.5:1 on the same kernel with the same knobs. The only difference is who asked.
What we are doing about it
Three of these are fixed and shipped: the config apply now reloads instead of restarting, container memory actually deduplicates, and the kernel limits a large lab needs ship with the installer rather than being tuned by hand.
The rest are not one-line fixes, and they share a cause. We could not see what the system was doing while it was doing it. Load average lied — a decaying average read as current contention when the box was 89% idle. Process CPU lied — ps reports an average since process start. Our own monitoring died while the thing it was monitoring stayed healthy, three times.
So the next body of work is observability: per-operation timing on the start and apply paths, a progress and health view that survives the load it is reporting on, and metrics a large lab can be run against rather than guessed at. That is what makes the remaining work — parallelising the start path, pacing by blast radius rather than node count, a VNI per pod instead of one flat segment — measurable instead of speculative.
It is also the piece we expect organisations running Labtris at scale to want supported, and we intend to support it. The lab stays free and open; running a few thousand nodes in production and being able to prove what they are doing is a different kind of need, and we would rather build it properly than pretend one box of adjectives covers both.
Everything above is reproducible: install it, generate a fat tree, and run the same audit.