The network is one of the easiest places to cut corners, because it's invisible right up until the moment it isn't. Everything looks fine until a busy evening or a DDoS lands, the uplink saturates, and suddenly every server behind it is lagging for reasons nobody can point at. So here is how ours is built, and why.
It's the same rule as the rest of our stack: we don't oversubscribe. Not the CPU, not the RAM, and not the wire.
Every server is dual-homed
Every server gets two 10G links, one into each of a redundant pair of Arista top-of-rack switches, bonded with LACP (802.3ad) using layer3+4 hashing.
The headline benefit is resilience. A switch can fail, a cable can be pulled, a NIC port can die, and your server stays online on the surviving link with no manual intervention. There is no single point of failure between your server and the fabric.
The second benefit is headroom. Traffic is spread across both links by source and destination (IP and port), so a busy node isn't funnelling everything down one pipe.
One honest note, because the numbers here are easy to oversell: LACP doesn't give a single connection 20G. Any one flow still rides one 10G link, because that's how the hashing works. It doesn't matter for game hosting, where the traffic is thousands of small flows rather than one giant stream. The point of the second link is redundancy and spread, not a bigger single-stream number.
The kit, and what actually matters about it
We run Arista throughout, in three tiers, each redundant:
- Switching at the server edge. Fast top-of-rack switches your servers dual-home into, in MLAG pairs so losing one doesn't take a rack with it.
- Routing and filtering at the edge. This tier does the heavy lifting: DDoS filtering, QoS, the 100G uplinks, and the BGP that lets us move traffic between sites. We pick for buffer depth and forwarding latency here, because this is where traffic from the whole site concentrates.
- A separate out-of-band management network. Management lives on its own physical network, so a problem on the data plane never locks us out of fixing it, and the management network is never reachable from the game side.
On the vendor choice, since people ask: Juniper's MX series is genuinely good at the edge and we rate it. We went with Arista for the fabric because we get low forwarding latency alongside the buffer depth we want in the same box, one operating system (EOS) runs every tier from the server port to the edge so the whole fabric is consistent and automated, and the classification engine does the DDoS and QoS work described below in hardware. The MX is an excellent router. Arista was simply the better fit for a low-latency, single-OS fabric we can automate end to end.
Two things about the switches genuinely affect your server, and one thing doesn't, so here are all three:
- Buffer depth matters where traffic concentrates. When lots of servers burst at once (everyone loading chunks, a big login wave), a switch with shallow buffers drops packets, and dropped packets are what players feel as lag. The fix is depth at the aggregation point, where every rack's bursts arrive at the same instant, so that is what we size for.
- Non-blocking capacity and 100G uplinks matter. The link from the server-edge switches up to the edge is where a network becomes oversubscribed. If that's a thin shared pipe, it's the choke point the moment a rack gets busy. Sizing the uplinks so the fabric is never the bottleneck is the network version of not overselling.
- Switch latency does not matter for your ping, and we won't pretend it does. Port-to-port switch latency is measured in nanoseconds. Your ping to the server is measured in milliseconds of internet distance, which is around a hundred thousand times larger. The switches earn their place on buffers and capacity, not on nanoseconds.
It's redundant the whole way up. Two links from your server into two switches, switches in redundant pairs, and at least two uplinks out of every site. There's no single cable, card, switch or upstream connection whose failure quietly takes you offline.
DDoS mitigation, in layers
The busier and more popular your server gets, the more likely it is to be attacked, and that's exactly when being knocked offline hurts most. The cheap way to stop an attack is a null-route, dropping every packet to the target, which stops the attack by taking your server offline with it. We would rather keep you up, so we scrub in layers instead of flipping one switch.
- On-prem scrubbing. We clean traffic on our own hardware first, so we are not waiting on an upstream provider to decide when to act.
- In-fabric filtering. Our edge classifies traffic in hardware at line rate and drops attack traffic at switching speed, in the fabric rather than in a bottleneck box downstream. The same engine handles QoS, so game traffic is prioritised over everything else on the wire.
- Per-node eBPF. Every node runs eBPF filtering to catch whatever slips past the layers above.
- Off-prem scrubbing on tap. For the largest attacks we can toggle on off-premises scrubbing, real scrubbing that cleans your traffic and returns it, not a null-route that takes you offline.
One honest detail most hosts skip: no mitigation engages the instant an attack starts. In the first moments some traffic seeps through before the upstream layers lock on. Because every node has 20Gbps of link (the dual 10G), it has the headroom to absorb that early seepage without falling over, and the per-node eBPF drops it at the host while the fabric catches up.
We don't publish our scrubbing headroom. Capacity numbers just become a target. The point is the posture: at every layer, including the last resort, we scrub and keep your server on the internet rather than null-routing it off.
Why we tell you this
Same reason as the rest of the stack. None of this is secret knowledge. Dual-homed LACP, MLAG, deep buffers where they count and fat uplinks are standard enterprise networking, and plenty of people know how to build it. The reason to actually build it this way is that it only works if you don't cram too much behind it, and that is a decision about how many customers you put on a network. It's the same decision the rest of our stack rests on.
If a network is oversubscribed, no amount of switch specs makes up for it. If it isn't, the boring, well-built version just works, quietly, on the night it matters.