PART I — ORIENTATION

01 · The datacenter is a different animal

The internet was built to survive routers you don't trust, on links you don't own, run by people you've never met. The datacenter is the opposite: you own every millimeter, you control every switch, and traffic patterns are wildly predictable — thousands of machines speaking to thousands of machines, all at once, within the same building. Almost every design decision in this course falls out of that one difference.

01.1 What actually changes at scale

Suppose you have four servers that all need to talk to each other at line rate. You run six cables and you're done. Eight servers: twenty-eight cables, seven NIC ports burned per machine. Sixteen servers: a hundred and twenty cables, and every server burns fifteen ports just on interconnect. The problem isn't theoretical — port count is money, and back-of-rack cable bulk is a physical constraint your datacenter facility people will describe to you in colorful language.

Now try it with a switch. Every server needs exactly one port, one cable, and the switch needs N. The total cost of interconnect went from O(N²) cables to O(N). This is the single most important economic fact in networking, and it generalizes: every switching layer you add collapses a quadratic problem into a linear one.

01.2 The east-west shift

The traffic inside a datacenter looks nothing like home internet traffic. When a web request arrives, it fans out across dozens of middle-tier servers, which talk to cache servers, which talk to storage, which replicates to other racks — hundreds of machine-to-machine conversations happening per user request, all staying inside the building. This is called east-west traffic (racks drawn side by side), as opposed to north-south (in and out of the building).

East-west traffic breaks an assumption embedded in enterprise network design: that you can size the connections between tiers for a fraction of the traffic each tier generates. In a DC, any server may need to send any other server line-rate traffic at any time — a Hadoop shuffle, a training job's gradient sync, a cache rebalance. Designing for the average is designing for a network that's congested whenever the workload is doing the thing it was bought to do.

01.3 The owner-operated advantage

Here's the piece most networking courses skip: you own this network. Every switch is the same few models. Every link is either a DAC you can replace in minutes or fiber your team pulled. Traffic matrices are known — you can instrument the workload itself. Failure is not an anomaly to route around blindly; it's a design input you budget for, like latency budget.

This ownership is why the datacenter world could abandon the Internet's "one box, one forwarding brain" model: when the operator owns both ends, protocols can be dramatically simplified, assumptions can be made uniform, and the network can be treated as one machine — a giant, warehouse-scale computer whose "bus" happens to be Ethernet. That framing (from the classic The Datacenter as a Computer) is the right mental model for everything that follows: you're not building a network between computers, you're building the computer.

01.4 What you should be able to do when this course is over

• Size a two-tier fabric for a given server count and bandwidth target, and say what it costs in ports, cables, and dollars.
• Argue L2 vs L3 designs from mechanism, and know exactly where each one breaks.
• Read a BGP table on a leaf switch and predict packet flow from it.
• Design failure domains so a single failure never takes a workload down, and prove it with numbers.
• Deploy VXLAN/EVPN tenancy without ever stretching Layer 2 — and explain why that discipline matters.
• Tune a lossless fabric for incast-heavy AI workloads without treating PFC like magic.
• Debug a real incident from switch counters alone.

Throughout, real-product numbers are used with the honesty they deserve: they're typical values, quoted so you can reason, not a purchasing guide. When the course says "on the order of", believe it.

PART I — ORIENTATION

02 · The switching problem

"Switching" sounds like a solved problem — hardware forwards Ethernet frames, fast. But at datacenter scale, the interesting question inverts: what is a switch, physically, such that a $10k box can forward a terabit per second, and why does it have buffers, queues, and a packet-drop policy at all? Every congestion behavior you'll ever debug in a fabric traces back to this one box's internal economics.

02.1 What the silicon actually does

Strip a modern switch down and there are three stages, and the reason they're separate is almost entirely about time:

• Parse. The PHY receives bits, the MAC aligns frames, and the parser carves the header into fields — in parallel across every port. This stage exists because serializing header processing would cap you at one port's speed.
• Lookup. Every packet must answer two questions — where does it go, and with what policy? Hardware does this with TCAM (exact fields) and LPM engines (longest-prefix match) in roughly a hundred nanoseconds, because the alternative — software — takes thousands of times longer.
• Queue and schedule. Multiple ingress ports can all want the same egress port at the same microsecond. The switch cannot forward them simultaneously, so it buffers and picks. This stage is the entire story of congestion.

That third stage deserves emphasis: a switch is not a wire with intelligence; it is a tiny contention-resolution computer. The moment two packets want one egress port, someone must wait. Everything about fabric design — non-blocking topologies, ECMP spread, lossless classes — is about making that contention rare, short, or harmless.

02.2 Buffers: why they exist, and why they're small

The naive assumption is that a big buffer makes a network safe. The truth is the opposite at DC scale. A buffer's job is to absorb microbursts — the sub-millisecond moments when arrival rate exceeds departure rate while the fabric's statistical multiplexing catches up. For that job you need milliseconds of queueing, and at 400 Gb/s a millisecond is 50 MB. Modern switch ASICs carry tens of MB shared across all ports — deep enough to absorb bursts, shallow enough to be affordable (buffer memory is expensive, power-hungry SRAM), and crucially shallow enough that queues drain before they add real latency.

The moment queues persist, the buffer becomes a liability: every byte sitting in a queue is latency added to every packet behind it, and a queue that fills to drop-tail converts congestion into loss, and loss into TCP timeouts. This is why the industry's answer to congestion moved from "buffer it" to "signal it" — ECN marking at a threshold, so senders slow down before the drop line.

02.3 Shared memory and the head-of-line problem

One more internal detail that will matter repeatedly: most modern switch ASICs use shared-memory buffering — one pool of SRAM that all ports draw from dynamically, rather than fixed per-port FIFOs. This is great for absorbing bursty traffic (any port can borrow headroom) and terrible in a specific way: one congested egress port can starve the whole switch by draining the shared pool, making unrelated ports drop traffic they had room for. The industry's fixes — virtual output queues (VOQs) at ingress, per-port/per-class reservation, and priority flow control — all exist to fence off this blast radius. Chapter 13 makes you watch it happen.

02.4 The forwarding table is the network's soul

Once lookup silicon exists, the interesting question becomes: who fills the table, and with what? A switch with an empty FIB forwards nothing. Everything called "control plane" — STP, routing protocols, controller pushes — is just different answers to "who programs the box, and how fast can it react when the answer changes?" Datacenter networking spent twenty years on this question, and the answer it converged on (few, big, simple tables, updated by BGP, with equal-cost paths used simultaneously) is what Part II is about.

Switch ASICs are surveyed in more depth in chapter 13, where buffer carving and scheduler mechanics get their own simulation. For now: parse → lookup → queue is enough scaffolding to hang the rest of the course on.

PART I — ORIENTATION

03 · L2 designs and where they break

Layer 2 — Ethernet bridging, MAC learning, one big broadcast domain — is the design your instincts reach for first, because it's simple: plug in cables, everything talks. It works for a rack. It works for a row. And then, somewhere between a few hundred and a few thousand ports, it becomes operationally radioactive. This chapter is about exactly where that cliff is, because the entire modern DC architecture is a direct response to it.

03.1 MAC learning works, until it can't

An Ethernet switch's default behavior is beautifully lazy: when a frame arrives, note (source MAC → ingress port) in a table, then forward the frame out every port except the one it came in on — flooding — unless the destination MAC is already learned. Unicast forwarding is free; broadcast traffic (ARP, DHCP) is a tax every host pays, and unknown-destination frames are a tax paid in duplicate.

At rack scale, this converges quickly and the tax is invisible. Scale it up and the taxes compound: MAC tables hold tens of thousands of entries; every host's ARP lands on every port of the domain; and — the killer — the domain must be a tree.

03.2 Loops: the fastest way to melt a datacenter

Redundancy in L2 means two paths between the same switches — which means a loop. And Ethernet bridging has no hop count. A broadcast frame that enters a loop is forwarded forever, and worse: it's duplicated at every crossing. Two copies become four, become eight — the frame count grows exponentially while switch CPUs parse every copy. This is a broadcast storm, and a full-mesh L2 with a loop is not "degraded", it is melting: control-plane CPUs spike, FIBs thrash, and the facility team starts getting very interested in your rack.

03.3 STP: the fix that became the problem

The Spanning Tree Protocol's answer to loops is to ensure there are none: elect a root bridge, compute lowest-cost paths to it, and block every other link. It's a brilliant solution to the wrong problem, and its costs are exactly why the industry abandoned it at scale:

• It disables your redundancy. You paid for two paths; STP uses one and blocks the other. Half your links carry zero traffic until something breaks — and then convergence is slow: classic STP reconverges in 30–50 seconds; Rapid STP still takes hundreds of milliseconds to seconds.
• It constrains topology. The tree's shape follows bridge priorities, not traffic. Hot rack pairs can't get an extra path; the blocked link sits idle next to a saturated one.
• It fails slowly and silently. A unidirectional fiber fault can make STP believe a blocked port's partner is dead, unblock, and create the loop it existed to prevent — the classic "STP failure" story is a storm at 3am, not a clean convergence event.

03.4 The 4,094-VLAN wall

VLANs segment a broadcast domain, but each VLAN you stretch across switches is a mini L2 domain with the same physics — and there are only 4,094 of them. That number sounds large until you count tenants: each tenant needing isolation across N racks consumes one VLAN per segment per zone, and the count grows with the product of tenants × segments. Hyperscalers hit the wall in the 2000s; any growing platform hits it eventually. There is no config line that raises it — it's the 12-bit VLAN ID in the frame, set in silicon by IEEE 802.1Q.

03.5 Multichassis LAG, a.k.a. buying back bandwidth with pain

The 2000s' answer to "STP blocks half my links" was MLAG: make two physical switches pretend to be one, so a LAG's member links can spread across both and no link is blocked. It works — most enterprise DCs still run it — but the price is a stateful distributed system: a peer-link that must carry sync traffic, careful handling of dual-active failures, per-pair configuration, and a scale ceiling measured in pairs. Chapter 12 dissects it properly. What matters here is the pattern: each patch to L2's problems added protocol state instead of removing assumptions.

Note on vocabulary: "leaf-spine" describes a physical topology; "L2 vs L3 fabric" describes what runs on it. They're orthogonal — and the course will keep them separate. Clos topology comes next (ch 04), then the routing that replaced STP inside it (ch 05).

PART II — THE FABRIC

04 · Clos fabrics from first principles

Ask a network in 2005 to connect 10,000 servers and it gave you a tree: access switches feeding aggregation switches feeding a core, with bandwidth thinning as you went up. That was fine when traffic was mostly north-south. East-west traffic made the tree's narrow waist fatal — and the fix, the Clos network, is the single most load-bearing idea in this whole course.

04.1 The tree's arithmetic problem

A tree's total capacity is capped by its root, and its links get fatter as you go up — 10G at the edge, 40G upstream, 100G at the core. That oversubscription is a feature for north-south traffic (you don't need every server talking to the internet at line rate simultaneously), and a bug for east-west (any two servers in different racks must traverse the waist, and there simply isn't enough waist to go around).

Worse, the tree has one root, so it has one failure that kills everything. Fixing that with redundant cores is fine, but the bandwidth math doesn't improve: two half-empty cores are still a narrow waist. The problem is structural, not a cable-count problem.

04.2 Clos's idea: width instead of depth

Charles Clos, a Bell Labs engineer, solved this in 1953 for telephone exchanges — where he had to connect thousands of lines without a single monster switch, and where a call blocked for lack of a middle-stage port was a business problem. His answer: build the big switch out of many small, identical ones. Replace one fat, unique core with many thin, interchangeable middle-stage switches, and connect them so any input can reach any output through any middle box.

Applied to a datacenter: servers connect to leaves (top of rack); every leaf connects to every spine; every spine connects to every leaf. There is no "core switch" — there are k spines, each one a complete, independent path between any two leaves.

• Path count: any leaf pair has exactly k distinct 2-hop paths (one per spine) — your redundancy and your capacity are the same object.
• Uniform latency: every inter-rack flow is exactly 2 hops, regardless of where the racks sit. No "far side of the core".
• No oversubscription needed: provision leaf uplink capacity ≥ downlink capacity and the fabric is non-blocking — any server can push line rate at any other server, simultaneously, forever.
• Scale by repetition, not by redesign: need more servers? Add leaves. Need more bandwidth per leaf? Add spines. The design doesn't change, only counts.

04.3 The 3-tier version, and when you actually need it

A single 2-tier Clos is bounded by switch radix — a 64×400G leaf gives you 32 down + 32 up at 1:1, and 64 ports is 64 ports. When you exceed what one leaf-spine pod can hold, you don't buy bigger switches; you repeat the trick one level up: leaves-of-leaves. Super-spines connect pods the same way spines connect leaves. The recursion is the point: same design, three tiers, scale to hundreds of thousands of ports with a fixed, small set of switch models — and identical failure properties at each level.

Practical guidance: 2-tier is the default for pods up to ~2,000–4,000 400G endpoints depending on radix; 3-tier above that. Don't build 3-tier for a 500-server cluster — you'd spend a tier buying latency and troubleshooting surface you don't need.

04.4 Cost of non-blocking, and the oversubscription decision

Full non-blocking costs: every leaf port goes two directions (down and up), so a 1:1 fabric has exactly half its ports facing servers. Doubling server bandwidth means doubling spines, cables, and optics — the bill is linear and unhidden. This is why oversubscription is a deliberate decision, not a sin: a 3:1 leaf uplink ratio means each spine count buys you three times the server ports, and if your workload's simultaneous cross-rack demand is under a third of line rate, you'll never notice.

The failure isn't choosing oversubscription — it's choosing it without knowing the workload's concurrency. A storage-replication fabric at 3:1 is an outage-in-waiting; a mostly-frontend farm at 3:1 is free money. Chapter 15 gives you the full math and a calculator.

04.5 Why identical boxes: the operational argument

Clos's uniformity is also an operations strategy. One switch model means one NOS image to qualify, one spare pool, one monitoring dashboard, one training path for on-call. In a 10,000-switch facility the marginal cost of a second switch model isn't the hardware — it's the second failure-mode library the on-call engineer must hold in their head. Hyperscalers push this to spares-as-consumables: identical boxes, racked, cabled, powered, running, waiting only for a config push.

Clos's original paper is "A Study of Nonblocking Switching Networks" (Bell System Technical Journal, 1953) — five pages, worth reading. Its datacenter renaissance came with Facebook's "Fabric Aggregator" and later the widely-copied leaf-spine pod designs of the 2010s. The recursive version is usually called fat-tree in the literature; in practice people build Clos.

PART II — THE FABRIC

05 · Routing the fabric: eBGP, ECMP, and the RIB

Chapter 03 killed the broadcast domain, chapter 04 gave you a topology with k parallel paths between any two racks. The remaining question is who programs the boxes: how does a leaf know a server subnet lives on leaf 7, and how does it use all four spines at once instead of one? The datacenter's answer — plain eBGP between every adjacent pair — sounds boring and is one of the most elegant load-bearing decisions in modern infrastructure.

05.1 Why not OSPF, or a controller, or statics?

Interior routing protocols (OSPF/IS-IS) are designed to keep one administrative domain's link-state in sync, with every node holding a full map of the topology. That's the wrong shape for a fabric: a 1,000-switch fabric is ~2,000 adjacencies and a link-state database that refreshes on every link flap anywhere. A centralized controller writing FIBs directly is fast but reintroduces the single-brain failure mode the topology just eliminated, and adds a scaling ceiling for churning state.

eBGP — the same protocol that runs the Internet — inverts both: each session is a point-to-point conversation between neighbors, nobody holds the full map (only next-hops), and the protocol's entire job is "tell your neighbors what you can reach, through you". In a Clos, the routes a leaf needs are: my own server subnets (connected), and everything else is one or two hops away. The protocol carries almost no state for the amount of forwarding it buys.

05.2 eBGP on a leaf-spine fabric, concretely

The standard design (used at hyperscale since ~2011) is almost trivial to describe:

• Every switch gets an ASN from a small per-tier block (e.g. leaves 65001–65099, spines 65101–65108).
• Every physical link is a /31 with eBGP running directly on it. No shared subnets, no reflection, no confederations in the common case.
• Every switch also has a loopback /32 — its stable identity, used for route next-hops and for management/telemetry.
• Leaves announce their server subnets (network 10.k.0.0/24) plus the loopback /32. Spines relay: a spine's job is to accept a leaf's prefix and re-advertise it to every other leaf, prepending nothing — the Clos makes every path equal-cost, so everything is ECMP.

The result is a fabric where the "routing table" on leaf 3 for a server subnet on leaf 8 is: via spine 1... or spine 2... or spine 3... or spine 4. Not a primary and backups — four equal answers, all installed in the FIB, all carrying traffic right now.

05.3 ECMP: how "four equal answers" becomes traffic

Equal-cost multi-path is implemented as flow hashing: the switch takes a tuple from the packet — classically src-IP, dst-IP, src-port, dst-port, protocol (the "5-tuple" plus L4 ports; a "flow") — and hashes it into an index over the member links. All packets of one flow take the same member, so TCP never sees reordering; different flows spread out.

This is where the classic ECMP pathology lives: hashing is statistical, not clairvoyant. With few, fat flows (VM backups, storage replication, AI collectives), collisions are near-certain — three 100G flows can hash two-deep on one member while a third member sits empty. Remedies exist at every layer: more hash entropy (L4 ports in the tuple), more members, per-packet spray with receiver-side reorder (rare, needs support), flowlet switching (re-hash at gaps), and the datacenter-AI world's adaptive routing (chapter 17). But the base mechanism stays: hash, don't think.

05.4 BGP tuning for the fabric

Vanilla eBGP is tuned for inter-domain policy — path selection by AS-path length, 30-second timers, hold-time-based liveness. A fabric wants different defaults, and the modern NOSes (SONiC, FBOSS, EOS, and friends) ship exactly these:

• BFD on every session (50ms×3 typical): link or neighbor death detected in ~150ms, not hold-time seconds.
• Short keepalive/hold (e.g. 3s/9s or less) as a backstop under BFD.
• Maximum-prefix and add-path where multi-path matters (keeping all equal paths advertised, not just the best one).
• Graceful-restart off by default in the data plane — in a fabric, blackholing while a restarting speaker holds state is worse than re-converging.

One design note that trips newcomers: eBGP between leaves and spines works because the fabric's ASNs are arranged so no prefix is learned from two different AS-paths of different length — every path through the Clos is exactly two AS hops (leaf → spine → leaf), so best-path selection collapses to "all equal, install all". The day you bolt on a route-reflector topology or dual-plane fabric you must preserve that property or your ECMP quietly degrades to one path.

The canonical reference for routed fabrics is the RFC 7938 ("Use of BGP for Routing in Large-Scale Data Centers", Google, 2016). Read it after this chapter — it will read like a summary of what you just built, which is the point.

PART II — THE FABRIC

06 · Failure domains, redundancy, and convergence

At small scale, redundancy means "two of everything". At datacenter scale that's not just expensive — it's wrong, because failures correlate: the same firmware bug hits all 128 spines at once, the same optic vendor's part dies in the same week, the same power feed takes out a whole row. Failure-domain design is the discipline of making sure the set of things that can die together is small, known, and survivable — and then proving it with convergence numbers, not adjectives.

06.1 Blast radius is a design output

Define it precisely: for a failure F, the blast radius is the set of traffic that stops or degrades until repair. In a Clos with k spines and ECMP hashing, the numbers are almost mechanical:

• One dead spine = 1/k of paths gone = 1/k capacity loss if the fabric was non-blocking, and zero if it was oversubscribed enough to absorb it (spines die silently on a 4:1 fabric — which is itself a monitoring trap).
• One dead leaf = that rack's servers gone. Nothing else. The leaf IS the rack's failure domain.
• One dead server NIC = 1/NIC of one server — invisible, if the server is dual-homed (chapter 12).
• One dead spine-pair sharing a PoE... does not exist. Correlated failures come from shared fate: same rack, same power feed, same optic batch, same firmware. You manage them by distribution, not by count.

The discipline: enumerate the things that share fate, then make each shared-fate group small and make its death non-catastrophic. This is why pods exist (chapter 04's 3-tier), why MLAG pairs are exactly 2 (chapter 12), and why hyperscalers spread a cluster's racks across rows and power domains — the physics of failure correlation beats the arithmetic of part counts.

06.2 The two clocks: data-plane repair vs. control-plane convergence

When something dies, traffic recovers on one of two very different clocks:

• ECMP local repair — when a member link of an ECMP group dies, the hashing silicon drops it from the group in microseconds to milliseconds. No protocol involved. If the failure is "one of several equal paths", the data plane self-heals before any control plane even notices. This is the fastest repair in networking, and Clos+ECMP gives it to you for free on every spine/link failure that leaves a path standing.
• Routing convergence — when the last path to a prefix dies, the FIB has no entry until the protocol withdraws/re-advertises: physical/BFD detection (~50–150ms), update propagation (per-hop ~ms), FIB re-program (~ms to tens of ms on modern ASICs). Total: a few hundred ms typical, worst-case seconds.

Design rule that falls out: never let a single failure require routing convergence when local repair can hold it. This is why k≥2 spines (so link/spine loss stays in ECMP-land) and why dual-homed servers (chapter 12) exist. Convergence is the backstop for true last-path failures, not the primary mechanism.

06.3 Redundancy's arithmetic: M+N, not N+N

Textbook redundancy is N+N (two of everything, one idle). At scale you instead buy M+N: M units carrying load plus N units of headroom shared across the failure set. Four spines at 60% utilization carry 3-spine load on 3 spines at 80% — survivable with margin. The design variables are utilization and N: run hotter (more M-loaded) and you need bigger N; run cooler and N shrinks. Hyperscalers run fabrics at 50%–ish precisely to keep N cheap while preserving the "any single failure is invisible" property.

06.4 Convergence budgets, in numbers

Put real targets on your design before you build it. A reasonable modern budget:

Two implications. First, your application timeout floor should sit above the worst budget in this table — request timeouts under 1 second will "randomly" fail on every leaf-class event, and you'll spend weeks blaming the network. Second, the numbers are only achievable with the tuning from chapter 05 (BFD everywhere, short timers, modern FIB programming); a default-configured fabric converges an order of magnitude slower.

Chapter 21's incident simulator runs you through exactly these failure classes with nothing but counters and logs. The failure-domain figures here are simplified models — real facilities add power, cooling, and fiber-path correlation, which is why "same rack" is never just "same rack".

PART II — THE FABRIC

07 · Addressing and the anycast gateway

A routed fabric needs an addressing plan that scales to thousands of links and never requires a human to hand-assign an IP again. The scheme the industry converged on — /31s on every fabric link, /32s for switch identities, one subnet per rack — is boring on purpose. The one genuinely clever piece is at the bottom: the same default gateway IP living on every leaf in the building.

07.1 The plan: three address classes, zero thought required

Pick a private /16 (or a slice of one) per pod and divide it by role:

• Loopbacks: /32 per switch. A switch's identity — BGP next-hop, management, telemetry. Never changes when a link dies, never collides, one line in a template.
• Fabric links: /31 per link. Two addresses, no broadcast, no network/broadcast addresses to waste. A 128-leaf pod with 8 spines consumes 1,024 /31s — trivially templateable: 10.0.a.b/31 where the octets encode (spine, leaf) mathematically. A human can decode any address to a location in seconds, which is worth more than it sounds at 3 a.m.
• Server subnets: /24 per leaf (or /2x for partial racks). Announced by that leaf in BGP, so the fabric's routing is the inventory: the routing table literally answers "which rack is 10.40.7.0/24 in?".

Two properties fall out that are worth stating explicitly. First, summarization is for exits, not the fabric — leaves advertise /24s and spines pass them; nobody aggregates inside the pod, because aggregation hides reachability and breaks ECMP's per-path granularity. Second, the plan must be algorithmic (octet math), because at thousands of links any plan needing per-link human decisions will have drift, and drift in addressing surfaces as unexplainable routing behavior.

07.2 The anycast gateway: one IP, every leaf

Here's the problem the scheme has to solve: servers want a default gateway. If each leaf has its own gateway IP, then moving a server between racks means re-addressing it — and in a bare-metal fleet, machines move (rack migration, replacement, re-boning). The routing fabric made movement cheap; unique gateway IPs would make it expensive again.

The fix: every leaf in the pod carries the same gateway IP for its local subnet — say 10.k.0.1 on rack k's leaf — with an anycast-style shared MAC (modern implementations use a virtual MAC shared across all leaves, or per-pod virtual MACs; the principle is identical). A server ARPs for its gateway and its own rack's leaf answers. Packets to off-rack destinations get routed from the local leaf; the fabric carries them the rest of the way.

• Moving a server is now routing, not re-addressing. Rack k's leaf withdraws 10.k.0.0/24... no wait — the subnet stays; what moves is the host /32. In the modern pattern (used with EVPN in chapter 11) the host's /32 route is advertised by whichever leaf it's physically on. Same IP, new leaf, table update, done — no ARP-cache poisoning to clean up beyond the neighbors' caches, which age out in seconds.
• No VRRP election, no failover gap. The gateway isn't a device that can fail over; it's a role every leaf plays simultaneously for its own rack. There's nothing to elect.
• ECMP-friendly. Traffic to the shared gateway from outside hits any leaf that announces it — anycast works precisely because the fabric is ECMP-native.

07.3 Why /31 and not /30, /24-per-link

Because fabric links are never shared subnets. A /31 (RFC 3021) gives exactly two addresses — one per end — with no network/broadcast waste, halving address consumption versus /30 and, more importantly, matching the semantic: a point-to-point routed link between two routers is exactly two identities. The alternative patterns — unnumbered interfaces (borrowing the loopback's /32) — are equally valid and even leaner; hyperscalers commonly run BGP-unnumbered (IPv6 link-local sessions, RFC 5549-style), getting address-plan-free adjacency. The principle either way: the fabric link is not a network; it's a wire between two loopbacks.

07.4 The whole scheme in one block

# pod 4 (10.4.0.0/16)
loopbacks:   10.4.0.0/24  → 10.4.0.s/32      (s = switch index, spine or leaf)
fabric:      10.4.16.0/20 → 10.4.16.a.b/31   (a = spine#, b = leaf pair#)
racks:       10.4.64.0/18 → 10.4.64+k*8.0/24 per leaf k... (or simpler:
             leaf k servers: 10.4.k.0/24, gateway 10.4.k.1 — anycast on ALL leaves)

Nothing here needs a database: octet math encodes role, location, and identity; BGP carries the inventory; the gateway is a constant. Boring by design — the interesting engineering moved up a layer, into overlays (Part IV), where the address plan's job is to make tenant networks invisible to the fabric underneath.

Anycast gateways in EVPN fabrics are usually called "distributed anycast gateway" — same idea, standardized MAC handling (chapter 11). If you're wondering "what stops two leaves from both routing a given host's traffic?" — nothing, and that's fine: with a shared virtual MAC and per-leaf symmetric hashing, both directions of a flow hash consistently. The subtle cases (asymmetric routing with stateful firewalls) are a chapter 18 problem.

PART III — THE PHYSICAL PLANT

08 · Racks, cables, and optics

The most elegant fabric design on a whiteboard is a cardboard box of undeployable geometry if you get the physical layer wrong. This chapter is about the parts that make ops people sigh: why in-rack links are copper and leaf-spine links are glass, why transceivers cost as much as they do, and why the cheapest cable in the building can be the most expensive one to choose wrong.

08.1 The media ladder

Each link class picks its media by one variable: distance. The economics are monotone — copper is cheap and short, multimode glass is middling and short, single-mode glass is more expensive per port and goes kilometers:

Read the pattern: cost and power jump together, and the jump is buying distance, not bandwidth. Two rules fall out. First, minimize the number of glass ports — a leaf-spine fabric where every uplink needs DR4 has a per-port optic bill rivaling the switch itself. Second, co-locate what talks — every 100 meters you shave off inter-rack distances converts SMF ports into DAC, and DAC into saved dollars and watts. Datacenter layout is a network cost decision.

08.2 Transceivers: the market that doesn't care about your topology

A transceiver is a small, hot-pluggable device combining a laser (or LED/driver for DAC), a photodetector, a SerDes interface, and — for long-reach parts — a DSP doing PAM4 modulation and forward error correction. At 400G/800G the DSP is the expensive, power-hungry part: it's why an 800G FR8 costs more and burns more than an 800G DR8, and why "LPO" (linear pluggable optics — stripping the DSP out, relying on the switch's own SerDes) is the industry's current lever for short-reach power reduction.

The 800G generation runs 8 lanes × 100G PAM4; form factors are OSFP and QSFP-DD800 (your switch's cage dictates which). Variants you'll actually quote: SR8/VR8 (multimode, 50–100 m), DR8 (parallel single-mode, 500 m, MPO-16), 2×FR4 (two 400G CWDM duplex-LC pairs, 2 km — the "break one 800G port into two 400G endpoints" trick), and LR4-class for 10 km. The "2×" family matters in AI fabrics: one 800G spine port feeding two 400G NICs is a standard density move.

08.3 Cabling as a system: MPO, fiber counts, and the back-of-rack reality

Every leaf-spine link in your fabric is two transceivers plus a fiber run through trays you'll never want to re-pull. This drives three practices:

• Structured cabling with patch panels at row ends. Spine-side ports land on a central patch area; moves are patch cords, not new pulls. The fabric's logical topology lives in the patch panel, not the tray.
• MPO trunk fiber (12/16/24-fiber ribbons) per path, breaking out at the patch panel. One 16-fiber MPO carries one DR8 (8 pairs) — plan fiber count as fabric capacity, with slack.
• Port-profile discipline: identical leaf configs mean identical cable trees per rack; a new rack is a cookie-cutter operation. The first custom-cabled rack is where your fleet becomes a museum.

And the honest operational note: cabling quality is where outages hide. Bent fibers, dirty connectors (yes, people still skip the inspection scope), over-bent DAC jackets in door channels — these produce the intermittent, distance-from-median failures that page you at 3 a.m. The best fabric teams treat fiber cleanliness and bend radius as design rules, not advice.

08.4 The rack as a unit of design

A standard compute rack is ~40–48 servers feeding 1–2 ToRs at the top, with server↔ToR DAC. The interesting constraint is power, not ports: a rack drawing 10–20 kW with modern servers means leaf-spine port density aligns with power density — you can't fill a rack with 400G-endpoint servers and a 10 kW feed. This is why high-density AI racks (chapter 17) changed facility design: when a rack draws 60–130 kW, you get maybe 8–12 nodes, each with 8 NICs, and the "rack" stops being a convenient unit of network scale.

Costs in the table are rounded 2026-typical spot numbers for context, not quotes — optics pricing moves fast and volume discounts are the norm. The reach classes (3m/30m/100m/500m/2km/10km) are the stable part; memorize those.

PART III — THE PHYSICAL PLANT

09 · NICs and the host edge

The fabric ends at a NIC, and the NIC is not a wire: it's a small computer with its own queues, DMA engines, hash functions, and — increasingly — its own CPU. Host-side choices (MTU, offloads, RSS, interrupt routing) silently set the ceiling on your fabric's real throughput and tail latency, and they're the first thing to blame wrongly when a network feels slow.

09.1 The receive path, and why pps costs more than Gbps

A packet entering a modern host goes: PHY → MAC → descriptor ring (a queue of buffer pointers, in host RAM) → DMA write into those buffers → interrupt → driver → kernel stack → socket queue → application. Two resources are consumed per packet: DMA/buffer work (roughly proportional to bytes) and CPU/interrupt work (strictly per-packet).

That asymmetry is the chapter's core fact. At 1500-byte MTU, 25 Gb/s is ~2 Mpps — cheap. At 64-byte packets, 25 Gb/s is ~37 Mpps, and no host stack on earth does 37 Mpps through the kernel: you cap out at a few million pps per core, so "line rate" small-packet traffic means the NIC drops, the kernel drops, or the CPU melts. This is why DC fabrics quote both bandwidth and pps, why kernel-bypass (DPDK, io_uring, RDMA) exists, and why "the network is slow" investigations should start with a packet-size histogram.

09.2 Offloads: what the NIC does so the CPU doesn't

• RSS (receive-side scaling): the NIC hashes each flow's 5-tuple into one of N hardware queues, each pinned to a core. Without it, one core takes all interrupts — the default single-queue death of a fast receiver.
• Checksum offload: NIC computes/validates L3/L4 checksums. Small win per packet, but it's per-packet — it scales with pps.
• TSO/LRO/GRO: segmentation and coalescing — the stack handles few, large super-frames while the NIC chops/merges on the wire. This is the offload that makes 100G+ hosts possible; it converts per-packet work into per-byte work in both directions.
• RDMA: the endgame — the NIC itself terminates transport, DMA-ing payloads directly to/from application memory with zero kernel involvement. Chapter 14 makes it the protagonist; here it's enough that RDMA bypasses everything this chapter said was expensive.

09.3 PCIe: the bus that quietly bottlenecks

A modern 400G NIC on PCIe Gen4 x8 has ~16 GB/s of host bandwidth — right at the edge of what 400G line-rate bidirectional traffic needs (~50 GB/s... no: 400 Gb/s ≈ 50 GB/s unidirectional, and a x8 Gen4 link carries ~16 GB/s. This is why 400G NICs are Gen5 x16, and why "8 lanes is enough" folklore dies at 400G). The practical rules:

• Match NIC generation to slot generation: a 2×200G card in a Gen4 x8 slot can be bus-starved under bidirectional load even when link counters look idle.
• Watch for PCIe replay/NAK storms from marginal risers or signal integrity — they look like NIC pauses under load and don't show on any network counter.
• NUMA placement matters: a NIC on the wrong socket adds cross-socket latency and halves effective DMA bandwidth on many boards. cat /sys/class/net/*/device/numa_node before you benchmark anything.

09.4 Choosing NICs for bare-metal fleets

For a general-purpose fleet: dual-port 25/50/100G cards are the boring default; two ports give bond redundancy (chapter 12's MLAG pattern) and headroom. For storage/RDMA tiers: single or dual 100/200G with confirmed RDMA support (RoCEv2 at minimum, chapter 14), on-board crypto if the tier needs it. For AI nodes: the NIC decision is not standalone — it's part of the fabric design (8×400G per node in rail-optimized topologies, chapter 17), and SR-IOV/smartNIC offload capabilities matter as much as line rate.

One fleet-level rule worth more than any spec sheet: minimize NIC model diversity. Every driver/firmware generation is its own failure-mode library — same argument as chapter 04's switch uniformity, applied one layer down.

Numbers used here: Gen4 x8 ≈ 16 GB/s usable (after encoding overhead), 25G @64B ≈ 37 Mpps, kernel stack throughput ceiling ~1–5 Mpps/core depending on tuning — all order-of-magnitude correct for 2026-era hardware; verify against your actual parts before quoting them in a design doc.

PART IV — OVERLAYS

10 · Why overlays exist

Part II built a fabric that routes /24s between racks, fast and clean. Then reality arrives: tenants want their own IP space (10.0.0.0/8 again, for the fourth customer this week), their own L2 segments spanning racks, and the ability to move machines without re-addressing. The fabric can't carry any of that natively — and the answer, overlays, is the conceptually simplest idea in this course wearing intimidating acronyms.

10.1 The multi-tenant problem, precisely

Strip it to the conflict: the fabric is a routed IP network with one address plan (chapter 07). Tenants bring:

• Overlapping addresses. Two tenants both using 10.0.0.0/16 is normal; the fabric can't route two identical prefixes.
• L2 adjacency expectations. Legacy apps, clustering software, and DHCP-server models assume "same subnet = same broadcast domain", across racks if needed.
• Lifecycle independence. Tenant networks get created, resized, and deleted daily. Nobody wants to re-cable or reconfigure the fabric's routers for any of it.

Chapter 03's VLAN answer fails on count (4,094) and on scope (VLANs are one building's L2, with all the loop/STP baggage). The overlay insight is to stop trying to make the fabric be the tenant network, and instead carry tenant frames as payload inside fabric packets.

10.2 The overlay idea in one sentence

Build the tenant's network out of virtual switches (VTEPs) at the edge, and make the fabric a dumb IP transit between them.

A VTEP (VXLAN tunnel endpoint — in practice, a Termination point on the ToR leaf, or the host's NIC) takes a tenant's Ethernet frame, wraps it in a fabric-routable envelope (outer IP/UDP header + a tenant ID), and sends it across the plain routed fabric to the VTEP nearest the destination, which unwraps and delivers it. To the tenant, two racks share an L2. To the fabric, it's just IP traffic between two leafs' loopbacks. Both worlds stay true to themselves.

• Address overlap: solved. The outer header carries fabric addresses; inner addresses are payload, invisible to fabric routing. Two tenants' 10.0.0.0/16s never touch the RIB.
• Scale: solved. The tenant ID (VXLAN's VNI) is 24 bits — 16M tenants versus VLAN's 4,094, and none of them touch the fabric's FIB.
• Stretching L2 across racks: solved without L2. The tenant's broadcast domain is a tunnel between VTEPs, not a physical domain — no STP, no loops, no storm physics. The fabric under it is exactly the chapter 05 routed Clos.
• Lifecycle: solved. A new tenant network is config on VTEPs + control-plane state (chapter 11), not cabling and per-switch VLANs.

10.3 The price list

Nothing is free, and an honest overlay budget includes:

• Overhead. VXLAN adds 50 bytes per packet (chapter 11 has the anatomy). On 1500-byte inner MTU that's ~3.4% plus whatever the fabric's own MTU policy costs you (jumbo inner frames need jumbo-capable underlay paths or fragmentation — always prefer a fabric MTU ≥ 1600).
• New failure modes. The tunnel adds a state machine: VTEP reachability, table sync (chapter 11), MTU mismatches (classic: a 9000-byte VM frame silently dies in a 1500 underlay), and outer-ECMP hashing quirks (inner-packet entropy — see VXLAN's UDP source-port trick, chapter 11).
• Debugging indirection. A packet now has two headers and two address planes. The discipline that survives: always identify which plane you're debugging — underlay path (fabric routing) or overlay path (VTEP tables) — before touching anything. Chapter 21 drills this.
• A control plane you must run. VXLAN itself is just encapsulation; making VTEPs learn remote MACs without flooding is chapter 11's EVPN, and it's the hard part.

10.4 What overlays are not

Overlays are not a substitute for a good underlay. Every overlay property — bandwidth, convergence, redundancy — is bounded by the fabric carrying it; VXLAN cannot fix a 4:1 oversubscribed spine any more than a VLAN could. The right mental model is a strict dependency: underlay first (chapters 04–07), overlay second (11–12). Teams that deploy EVPN onto a fabric with unknown oversubscription ratios are borrowing surprises.

And for completeness: not every DC needs overlays. A single-tenant bare-metal fleet with one address plan and no cross-rack L2 needs is better off without them — plain routed fabric, fewer moving parts, less to debug. Overlays exist to buy tenant freedom; if you're not selling that freedom, don't pay for it.

The general idea is older than VXLAN — IP tunnels, GRE, and MPLS VPNs all carry-network-as-payload. VXLAN won in DCs because it rides UDP, which the fabric already ECMP-hashes well, and because it needed zero changes to switches' forwarding silicon. Chapter 11 shows exactly why that mattered.

PART IV — OVERLAYS

11 · VXLAN and EVPN

VXLAN is 8 bytes of header; EVPN is the control plane that makes those 8 bytes scale to a million endpoints. Separately they're two acronyms — together they're the modern answer to "how do I run thousands of tenant networks over one fabric", and by the end of this chapter you'll be able to read a real VTEP's table and know exactly what each line is doing.

11.1 VXLAN: the envelope, byte by byte

When a VTEP encapsulates a tenant frame, what goes on the wire is an ordinary Ethernet/IP/UDP packet whose payload is the entire original frame plus an 8-byte shim:

[outer eth 14][outer IP 20][UDP 8][VXLAN 8]→ [inner eth 14][inner IP 20][payload]
                                     ↑
              flags(1) reserved(3) VNI(3) reserved(1)

The VNI (24 bits) is the tenant's bridge domain — "VLAN 100 of tenant B" becomes VNI 100100, and its 16.7M-space is why chapter 10's scale wall vanishes. Everything else has exactly one job worth remembering: the UDP header is a hashing handle. The fabric ECMP-hashes on the 5-tuple, and an inner-flow's entropy would be invisible under a fixed outer header — so the VTEP writes a hash of the inner flow into the outer UDP source port, restoring per-flow spreading across the underlay's ECMP groups. Small trick, enormous consequence.

11.2 The problem EVPN exists to solve: learning

VXLAN alone leaves VTEPs with an old question: which remote VTEP has the MAC I need? Data-plane learning (flood the first frame everywhere, learn from what comes back) is exactly chapter 03's MAC learning at fabric scale — BUM flooding across every VTEP in the VNI, with all its waste and failure modes. EVPN's move is to replace data-plane learning with control-plane advertisement: a VTEP that learns a MAC locally announces it to the others via BGP, before any tenant packet needs it.

EVPN runs as an address family inside BGP (typically iBGP with route reflectors — leaves peer with a couple of reflector spines rather than full-mesh, at exactly the scale-chapter 05 worried about). The workhorse route is the type-2 (MAC/IP advertisement): "VNI 100100, MAC aa:1:2:3:4:5, IP 10.20.0.5, next-hop me". Type-3s carry multicast/underlay-per-VNI state; type-5s advertise subnets for routed tenants. The RD keeps overlapping tenant prefixes from colliding; RTs say which VTEPs should import which tenancy — the policy knob operators actually turn.

11.3 What EVPN's control plane buys, concretely

• No flooding in the common case. Unknown destination → check the EVPN table → unicast VXLAN straight to the right VTEP. BUM traffic shrinks to genuinely-unknown cases (usually misconfig or a host that hasn't spoken yet).
• Host moves that converge like routing. A VM's /32 follows its port with a BGP update — the same withdraw/advertise mechanics as chapter 07's anycast-gateway moves, now for MACs and IPs, at the same speed.
• ARP suppression. Because the leaf already knows MAC↔IP from type-2s, it can answer ARP locally — chapter 07's stale-ARP failure mode gets engineered away at the edge.
• Multitenant ECMP that works. Symmetric IRB + anycast gateway (below) means both fabric directions hash cleanly; asymmetric routing across state is the historical footgun, and EVPN's symmetric mode was designed precisely so you don't keep it.

11.4 IRB: routing between VNIs without leaving the box

Tenants need to talk to each other (and to the internet) through controlled points. EVPN does this with integrated routing and bridging: each VNI gets an anycast-gateway SVI (chapter 07's shared-IP pattern) on every VTEP, and a packet from VNI A to VNI B is routed on the ingress leaf — decap tenant-A, route, encap tenant-B, one box, one lookup pipeline. Symmetric IRB additionally uses a per-tenant transit VNI so both directions traverse symmetric fabric paths (hashing stays consistent); asymmetric IRB is simpler but routes on ingress only and needs every VNI present everywhere — fine for small campuses, a FIB bomb at scale. Modern practice: symmetric, with route reflectors keeping leaves' tables to "what this leaf actually serves".

11.5 Operational shape of a real deployment

What you actually configure, in order: (1) underlay — chapter 05/07 exactly as before; (2) VTEPs on leaves: VNI↔VLAN↔VRF mapping per tenant, anycast MAC+IP on every SVI; (3) iBGP EVPN sessions to reflector spines, per-tenant RT policy; (4) MTU ≥ 9000 on the underlay (inner 9000 needs outer headroom — the failure mode from chapter 10, again); (5) BUM policy: head-end replication for small VNI counts, underlay multicast where the scale justifies it. None of these steps is exotic; all of them are things chapter 21 will ask you to debug at 3 a.m., so the design studio (chapter 20) makes you configure the whole stack once, end to end.

References if you want the primary sources: RFC 7348 (VXLAN), RFC 8365 (network virtualization overlay with EVPN), and the EVPN spec family starting at RFC 7432. VXLAN's UDP source-port hashing trick is in RFC 7348 §3.1 — three sentences that justify the entire design.

PART IV — OVERLAYS

12 · MLAG and the hybrid designs

EVPN is the modern answer, but the industry ran a decade of datacenters on a different trick for the same problem: make two physical switches into one logical switch, so hosts can dual-home with a plain LACP bond. MLAG — multichassis link aggregation, vendor-flavored as vPC, MC-LAG, stack, or "virtual chassis" — is still everywhere, and understanding exactly how it works and how it fails is required equipment for operating real infrastructure, whether or not you ever choose it.

12.1 The problem MLAG solves for the host

A server with one uplink has a failure domain of one cable. The boring fix is a two-member LACP bond — but a bond's members must land on one LACP partner (one logical switch). Two physical ToRs can't be one partner... unless they lie. MLAG is that lie, carefully engineered: both leaves present the same LACP system-ID and the same virtual MAC to the host; the host believes it's bonded to a single box; the pair internally decides, per-host, which member carries traffic.

What this buys: host-side redundancy with zero host configuration — no routing protocol, no EVPN smarts, no awareness at all. The host's bond hashes and fails over transparently. For general-purpose fleets this simplicity is exactly why MLAG survived so long, and why it's still a reasonable choice at modest scale.

12.2 The peer-link: where the lie lives

To act as one switch, the pair must share state: MAC tables, ARP entries, the "who forwards which bond" decision. They exchange this over a dedicated peer-link (and usually a peer-keepalive for liveness when the peer-link itself dies). The peer-link also carries the odd-destination traffic: a frame arriving on leaf A for a MAC whose bond member lives on leaf B must cross the peer-link to exit — the reason peer-links are sized generously (2–4× a host link) in production.

12.3 The failure modes, honestly

• Dual-active / split-brain. Peer-link dies but both leaves live: both still think they're the forwarder for every dual-homed host. Duplicate frames, flip-flopping MAC tables, and — with a stateful device in the middle — session chaos. Every MLAG implementation has a defense (disable secondary's members, isolate one side, poison its own MACs), and every operator has a story about the window before it fires. The sim's "fast failover fix" is that defense.
• Consensus dependency. MLAG is a distributed system with a two-node quorum — the most fragile quorum in computing. Maintenance that reboots the wrong one first, or a keepalive that dies with the peer-link, produces exactly the window above. The runbook answer is boring and strict: peer-link first up, last down; never assume "they'll figure it out".
• Scale ceiling. State sync, config coupling, and upgrade choreography (one at a time, verify, repeat) make MLAG pairs fine at tens-of-pairs and miserable at hundreds. Hyperscalers abandoned it not because it breaks but because operating thousands of tightly-coupled pairs is slow.
• Failure correlation. The pair is, by construction, a shared fate for every dual-homed host — a deliberate two-node blast radius (chapter 06's language). Put both ToRs on one power feed and you've rebuilt the N+N trap.

12.4 Choosing: a decision rule

The unifying principle across all four rows: host redundancy is table stakes; the question is only where the state lives — in a peer-link's consensus (MLAG), in a BGP control plane (EVPN), or nowhere at all (single-homed + fast convergence). Pick the cheapest state location your operations can carry.

Naming reality check: vPC (Cisco), MC-LAG (Arista/Juniper generic), IRF/stacking (HPE and others), virtual-chassis (Juniper) — same architecture, different marketing and failure windows. When a vendor says "seamless", ask what happens when the peer-link dies with both members alive; that's the whole evaluation.

PART V — CAPACITY & CONGESTION

13 · Inside the switch: buffers and queues

When a receiver is slower than its senders, every architecture must decide which packet waits. A FIFO answer is easy to implement and wrong under mixed traffic; VOQ is the answer operators buy when a slow destination must not poison unrelated traffic.

13.1 Head-of-line blocking is a queueing mistake

An ingress FIFO contains packets for many egress ports. Put a packet for a blocked egress at its head and packets behind it for perfectly healthy egresses cannot pass. Nothing is wrong with their destination, the fabric, or the sender; the queue encoded the wrong dependency. Virtual output queues split ingress state by destination, so congestion at E2 fills only queues headed to E2. The crossbar scheduler then services E1/E3 normally.

PART V — CAPACITY & CONGESTION

14 · Congestion, incast, and lossless fabrics

TCP is excellent at sharing a bottleneck when flows arrive gradually. Incast is the adversarial case: many synchronized workers each send a modest reply to one receiver, and the aggregate arrives faster than its one port can drain. At 100–800G this happens in microseconds — faster than normal end-host feedback.

14.1 Why a small fan-in can be catastrophic

Eight 100G senders can inject 800G into one 100G receiver. A 16MB buffer buys roughly 1.4ms at the excess rate; then drop-tail discards packets. TCP's retransmission timers and synchronized multiplicative decrease turn a brief burst into a sawtooth: queue full, loss, idle, all senders retry, queue full. RDMA is even less forgiving: loss can stall a work queue and leave expensive GPUs idle.

PART V — CAPACITY & CONGESTION

15 · Bandwidth math and oversubscription

A topology drawing does not tell you how much bandwidth you built. Port counts do. The calculation has no magic: sum server-facing bandwidth on a leaf; sum uplink bandwidth; divide. The only difficult part is deciding which ratio your workload can survive.

15.1 The leaf is where the ratio lives

For 32×25G servers and 4×100G uplinks: 800G down / 400G up = 2:1. That is the maximum aggregate cross-rack demand the leaf can satisfy; adding more spines changes path diversity, not the 400G bottleneck. A 1:1 fabric has uplink sum ≥ downlink sum. Bisection bandwidth is the sum of capacity crossing a cut; an any-to-any fabric needs enough of it for the traffic matrix you promise, not the average counters you observed last Tuesday.

PART VI — AI FABRICS

16 · The AI job as a network workload

Traditional services have independent flows and tolerate uneven timing. Synchronous training has a barrier every step: all workers compute, then exchange gradients, then the slowest worker decides when everybody computes again. The network is on the critical path of every training iteration.

16.1 Collectives make topology visible

All-reduce combines a tensor from N ranks and distributes the result. Ring all-reduce keeps every link busy and moves roughly 2(N−1)/N × S bytes per rank — bandwidth optimal for a large tensor, but it takes 2(N−1) serialized phases. A binary tree uses 2log₂N phases — fewer latency steps, but fewer links are active at high levels and the root-adjacent links become hot. Libraries choose based on message size and hardware; fabric engineers must know the resulting traffic matrix.

PART VI — AI FABRICS

17 · Rail-optimized fabrics and scale-up

A GPU server may expose eight high-speed NICs, one per GPU/rank. A normal ToR design groups them by server. Rail optimization groups them by rank: NIC 0 from every server enters rail 0's leaf, NIC 1 enters rail 1's leaf, and so on. It is a cabling discipline designed around the collective, not a way to remove required capacity.

Current context: UEC 1.0 was released in 2025 and defines an Ethernet AI/HPC stack. Its 2026 1.0.2 update retains optional link/network enhancements; treat it as a developing standard, not a replacement for engineering the fabric you deploy today.

PART VII — EDGE & LIFECYCLE

18 · Edge connectivity and DCI

The fabric's clean east-west world ends at the border. North-south traffic needs policy, public addressing, DDoS capacity, and stateful appliances; inter-DC traffic needs a failure model measured in buildings and providers. Keep both concerns at the leaf edge — never let them contaminate the universal spine.

PART VII — EDGE & LIFECYCLE

19 · Operations, telemetry, automation

At ten switches, SSH plus intuition works. At a thousand, it manufactures outages. The operating model must be declarative config, continuous state validation, and telemetry that arrives before the incident has moved on.

19.1 What to collect

Per port: admin/oper state, speed/FEC, pre- and post-FEC errors, discards, ECN marks, PFC pause duration, queue occupancy, ECMP member utilization. Per control plane: BGP session state, prefix count, route changes, BFD events. Per host: NIC ring drops, PCIe errors, NUMA placement. Export at the smallest useful interval; aggregate after collection, not before.

PART VIII — CAPSTONE

20 · Design studio

A design is a list of promises: how many endpoints, what bandwidth, which failures are invisible, how tenants work, and how operators observe it. Start with those promises. Diagrams come afterward.

20.1 A real design-review checklist

1. Traffic matrix: peak east-west, north-south, storage, backup, collective; not average.
2. Leaf math: downlink sum, uplink sum, ratio, headroom after one spine failure.
3. Failure matrix: cable, optic, NIC, leaf, spine, power feed, row, software rollout.
4. Addressing: algorithmic /31 or unnumbered links, loopbacks, per-rack subnets, gateway model.
5. Control plane: BGP topology, timer/BFD budgets, ECMP width, route policy and limits.
6. Overlay only if needed: VNI/VRF/RT policy, MTU budget, BUM behavior, host-move semantics.
7. Operations: golden config, preflight/rollback, FEC/queue/BGP telemetry, spare plan.

PART VIII — CAPSTONE

21 · Incident simulator

The fastest way to make networking knowledge operational is to order evidence correctly. Don't start with a fix. First decide whether a packet is missing because the path is wrong, the path is congested, the receiver is overwhelmed, or the overlay state is stale.

21.1 The four-question runbook

1. Is the destination route present in RIB and FIB?  (control plane)
2. Is the selected next-hop/link healthy and error-free? (physical)
3. Is a queue or host RX ring dropping/marking? (congestion / host)
4. Does the overlay table resolve inner MAC/IP to the expected VTEP? (overlay)

PART VIII — REFERENCE

22 · Reference and glossary

Use this as a compact index, not a substitute for the failure models in the chapters. If a term here feels memorized rather than derived, go back to the simulator it belongs to.

Primary references: RFC 7938 (BGP DC fabrics), RFC 7348 (VXLAN), RFC 7432/8365 (EVPN), IEEE 802.1Q (VLAN), IEEE 802.3bb (PFC), UEC Specification 1.0.2 (January 2026). Validate vendor behavior — ASIC/NOS details vary.