Protecting the tick: core isolation, kernel latency and the numbers

This is the long version. If you want the overview, our stack covers the whole machine in a few minutes and this paper covers one layer of it in detail: how a game server is given cores, what we changed in the kernel those cores run, and what we measured afterwards.

We are publishing it because the claim underneath it is easy to make and hard to check. Every host says their servers are fast. Almost nobody shows you the mechanism, the configuration, the test, or the numbers that came out of it, including the numbers that are awkward. So here is all of it, with enough detail to reproduce on your own hardware.

The short version of the finding: on an idle node, the stock Debian kernel had a better average wake-up latency than ours, one microsecond against six. On the same run its worst wake-up was 93 milliseconds, which is most of two Minecraft ticks lost to a single scheduling decision. Our current build's worst wake-up across five minutes was 20 microseconds.

What we measure, and why it is not TPS

Minecraft runs a fixed loop. Twenty times a second the server moves every entity, ticks every block that asked to be ticked, runs redstone, handles chunk loading and saves. That gives each tick a budget of 50 milliseconds. Finish inside it and the server holds 20 TPS. Miss it and the tick runs long, the next one starts late, and the server is behind.

Three different problems get called lag, and they have nothing to do with each other.

Ping is the time your packets spend crossing the internet. It is set by distance and routing, and no amount of CPU tuning changes it. We make the same point about switch latency in how our network is built, where the honest answer is that nanoseconds of port-to-port latency do not matter against milliseconds of internet distance.

TPS is the rate the loop achieves. It is capped at 20, so it is a one-sided measure: it tells you when things are already bad and nothing at all while things are fine.

MSPT, milliseconds per tick, is the real number. It says how much of the 50 millisecond budget each tick actually used. A server at 20 TPS with 45ms ticks is one mob farm away from trouble. A server at 20 TPS with 8ms ticks has room. What causes Minecraft server lag goes through this distinction properly.

The work in this paper sits one layer below MSPT. Before the tick loop can use its 50 milliseconds, the operating system has to actually run the thread when it is ready. If the kernel hands the core back 200 microseconds late, that is 0.4% of the budget gone before the first entity moves. If it hands it back 93 milliseconds late, the tick is not late, it is skipped, along with the one after it.

So what we measure here is scheduler wake-up latency: ask for a thread to run at a precise moment, see when it actually runs, and record the distribution. That is a kernel measurement. It is not a TPS measurement, and the two do not convert into one another. More on that limitation further down, because it matters.

The number to watch in any latency distribution is the tail, not the average. A server owner never feels an average. They feel the single worst tick during the evening when forty people were online, and so does everyone standing in that chunk.

The problem with buying CPU as a percentage

Game panels express a CPU limit as a percentage. Set 200% and you have asked for two cores' worth of processing time. Underneath, that becomes a quota in the Linux control group interface: a period, normally 100 milliseconds, and an allowance to spend inside it.

The mechanism is simple and its failure mode is specific. Your container runs on whatever cores the scheduler picks, accumulating time against the allowance. When the allowance runs out, every thread in the container stops until the period rolls over.

Two consequences follow, and both land on the tick.

The first is the throttle itself. A tick that needed 12 milliseconds of work can be stopped halfway through because the container reached its quota, and then it waits for the remainder of the period before finishing. The machine can be almost idle while this happens. The stall is an accounting artefact, not a shortage of hardware, which is what makes it so hard to diagnose from inside the game: nothing in the server log explains it, and every graph on the panel looks healthy.

The second is placement. A quota says how much, never where. The scheduler is free to move the tick thread between cores, and it does, because from its point of view all cores are equivalent and balancing work across them is its job.

On the processors a modern game fleet runs, they are not equivalent. A Ryzen with more than eight cores is built from multiple core complexes, and each complex has its own L3 cache. A thread that moves from one complex to the other arrives with none of its working set in the local cache and has to fetch it again across the die. For a workload whose main loop spends much of its time chasing pointers through a heap, that is the most expensive thing that can happen to it. We go into why cache dominates this workload in choosing a CPU for Minecraft.

It is worth being clear that this is the default nearly everywhere, and it was our configuration too until this quarter. The panel field that assigns cores sat empty across our fleet, so every container received a quota and no core assignment. We are not describing a mistake we watched somebody else make. We are describing one we found in our own estate, which is why the rest of this paper exists.

The design

Eight decisions. Most of them remove a source of latency. Two of them are trades, and we say what we gave up. All of them are reversible.

Fingerprinting the machine first

Nothing is hardcoded, because a hand-written core list is wrong the moment it meets a different chip. On first run a fingerprint reads the machine's real topology from sysfs: which cores share an L3 cache, which are hyperthread siblings of each other, how large each cache domain is.

From that it reserves housekeeping cores per cache domain rather than taking them from the front of the chip. On a part with two complexes that means a small number of cores reserved on each, not several on one and none on the other, so the operating system always has a core near the work it needs to do. On parts with six-core complexes it drops to a single housekeeping core per domain, because reserving two would cost a quarter of the chip.

Parts with asymmetric cache get a specific rule. Where one complex carries stacked cache and the other does not, every housekeeping core is taken from the smaller-cache domain and the pool is laid out on the larger one first, so the threads that benefit most from cache keep the cache to themselves. That asymmetry only exists on our dedicated servers, where a large-cache part is something you can specify, so this rule applies there rather than fleet-wide.

Everything the fingerprint is not certain about, it refuses. If a domain has too few cores to reserve housekeeping and still leave a usable pool, the tool stops with an error instead of producing a plan that starves the box.

The fence, and the trap we avoided

With housekeeping cores chosen, something has to keep the operating system inside them. We do that with a control group partition: system services, user sessions, the container daemon and the monitoring agents are all confined to the housekeeping set, and the rest of the chip becomes a pool that only game servers occupy.

The obvious alternative is the kernel's own isolation boot parameter in its full form, and it backfires in a way worth documenting. Telling the kernel to treat the pool as isolated domains stops the scheduler balancing work inside the pool at all. Hand a Java process six of those cores and it will run every one of its threads on the first one, because nothing is moving them. You remove the interference and break the server in the process.

The control group fence keeps the operating system off the pool without taking load balancing away from the server. Inside its own cores, the server's threads spread out normally. Outside them, nothing of ours competes.

Cores, not percentages

Each server then gets named cores, and those cores sit inside one cache domain. Two things change for the tick. It cannot be throttled part way through, because there is no quota left to hit. It cannot lose its cache to a migration across the die, because the cores it can reach all share the same L3.

Interrupts

A core that is not running your server can still be interrupted on its behalf. Network cards, timers and storage queues all raise interrupts, and the kernel will service them wherever it likes unless told otherwise.

Every interrupt that can be moved is moved onto the housekeeping cores, spread round-robin so a multi-queue network card still has somewhere to spread its queues. Storage queues are a special case: the kernel manages those itself and refuses to have them reassigned, so a boot parameter keeps them off pool cores instead. Both paths are re-applied on boot, and a device rule re-spreads interrupt vectors if a card is reloaded, because a driver reload silently resets affinity and a one-time fix would quietly stop being true.

Idle states

This is the piece that surprises people. Leaving a deep idle state on current Zen parts costs in the region of 350 microseconds, and that cost is paid on the first instruction after the core wakes. A tick thread that went quiet for a moment, which is exactly what a tick thread does twenty times a second, can pay it at the worst possible time.

On pool cores we keep only the shallow idle states, the ones whose exit latency is inside a 100 microsecond budget. Housekeeping cores keep every state they came with, which is where the power saving actually lives: on a loaded node those are the cores doing nothing.

Hyperthreading, and what sits on your sibling

Core isolation at the core level is not enough on a part with two threads per core, because two sibling threads share one physical core's execution resources. A neighbour scheduled onto your sibling takes a share of the core you were told was yours, and no cpuset drawn in terms of logical CPUs will show you that happening.

So slots are allocated in terms of physical cores. A density slot takes both threads of a physical core, so the pair is never split between two tenants. A determinism slot takes two physical cores in the same cache domain: the tick thread gets one core, both threads of a helper core carry the server's other work, and the tick core's own sibling is deliberately left out of every cpuset so that nothing at all shares a physical core with the tick.

That last part costs us a thread per determinism slot. We leave it idle anyway, because a tick thread sharing its core is the one kind of interference that stays invisible on every graph a customer can see.

Security mitigations, and what they cost

Processor vulnerability mitigations are not free. On current AMD parts the return-address and speculation mitigations cost real cycles on every kernel boundary crossing, and switching them off is one of the larger latency wins available on paper.

On shared nodes we leave them on. The whole justification for mitigations is that an untrusted neighbour might be sharing the hardware, which is exactly the situation a shared node is in, so turning them off to buy latency would be selling your tick a few microseconds at the cost of the isolation protecting your world data.

On a single-tenant dedicated server there is no untrusted neighbour on the box, so that trade changes, and those nodes can run with mitigations disabled. It is a genuine dedicated-only performance advantage, and it exists because of the tenancy rather than because of anything clever we did.

Reverting without a human

None of the above is worth much if a bad plan takes a node down and somebody has to drive to it. Changes that need a reboot are staged behind a boot gate. The node boots the new configuration once, and a service checks that the container runtime, the game server daemon and the fence are all healthy within two minutes. If they are, the configuration is committed. If they are not, the node reboots itself back to the last known good state.

Underneath that sits a counter in the bootloader's own environment. Each attempt at the new configuration is recorded before the kernel starts, and after two attempts without a successful commit the bootloader selects the previous entry by itself, with no working operating system required to make that decision. The kernel is also configured to panic rather than hang on an early failure, so a kernel that dies before userspace exists still counts as an attempt and still gets reverted.

The kernel

Debian's kernel is a good general-purpose kernel, and a game node is not a general-purpose machine. We build upstream stable with Debian's own configuration as the base, plus a fragment of our own, so the latency behaviour is compiled in rather than argued with at runtime.

The build itself is deliberately boring. The source is fetched from kernel.org, and the detached signature on the uncompressed tarball is verified against the stable maintainer's published release key before anything is unpacked. The fragment is merged onto Debian's configuration, and then every option that matters is checked again afterwards, because the configuration step silently drops settings that conflict with others. If one of them did not survive, the build stops rather than producing an image that looks right and is not.

Two honest limitations. We do not hold Debian's module signing key, so our images are unsigned and Secure Boot stays off on these nodes. And live patching covers individual functions, which means it covers an urgent one-line security fix but not a whole stable-series update, which is hundreds of patches. Those go through the staged reboot above. Hot-patch the one-liners, reboot for the rest, and say which is which.

Method

The test is the one that matches the failure. A thread asks to be woken every 200 microseconds on a pool core, and we record how late each wake-up actually is, over five minutes, twice. Once with the node otherwise idle. Once with every other core on the box driven to 100% by a load generator, which is the case the isolation exists to survive.

cyclictest -m -p99 -a <pool-core> -t1 -i 200 -d0 -D 300 -q -h 400

Locked memory, real-time priority, one thread pinned to the measured core, a 200 microsecond interval, no distance between threads, and a histogram that tops out at 400 microseconds. The load side is a matrix-product load generator restricted to every core except the one being measured.

Three details matter for honesty. The measured core is the same core in every run, because comparing two different cores on the same chip is not a comparison. The harness pins itself to the housekeeping cores, so the shell, the tracing and the parsing cannot perturb the thing being measured. And the histogram bound is the definition of the stall count: anything later than 400 microseconds does not fit in the histogram and is counted as an overflow, which is why that row is reported as a count rather than a percentile.

At a 200 microsecond interval over five minutes, each run is roughly 1.5 million wake-ups.

Alongside this we capture run-queue latency distributions, involuntary preemption counts, and hard and soft interrupt counts on the measured core, which should fall to approximately zero once the interrupt work has moved. Those confirm the mechanism did what it claims rather than the number improving for an unrelated reason.

Results

Three kernels on the same machine and the same core. The stock Debian 6.12 kernel, our 6.18 build, and our 7.2.13 build.

Read the worst-case column first, because the averages will mislead you.

On an idle node the stock kernel wins on average, one microsecond against our six. We have not chased that difference, and we would take the trade again, because of the number immediately to its right. Stock's worst wake-up in the same run was 93 milliseconds. Against a 50 millisecond tick budget that is 1.86 ticks, gone, on an idle machine, from one scheduling decision. That is the stall behind the "Can't keep up" line in a server console, which we cover in fixing "Can't keep up".

Under load the averages invert, and they stop being the interesting part anyway. Stock loses a tick's worth of time somewhere about every three seconds. Across five minutes that is in the order of a hundred stalls out of 1.5 million wake-ups, which is a rate of roughly one in fifteen thousand, and also an evening of unexplained stutter. The 7.2.13 build's worst moment across the same window was 20 microseconds, and nothing in the run crossed 400 at all.

The context switch figure is the cost of the kernel swapping one thread for another. Halving it does not show up as a headline number anywhere, and it is paid thousands of times a second by a server running a few hundred mods, each one with its own threads.

One row deserves an explanation rather than a quiet omission. On the 6.18 build the idle worst case, 822 microseconds, is worse than its own figure under load, 80 microseconds, which looks backwards. The mechanism is the idle state exit described earlier: on an idle node the core is asleep when the timer fires and pays to wake up, while on a loaded node it never gets the chance to doze. Capping the deep states on pool cores is the change that moves that number, and the 7.2.13 column is what it looks like afterwards.

Both of our builds carry an identical configuration fragment, so the difference between those two columns is upstream kernel work rather than anything we did. We are not going to attribute it to specific changes we have not traced.

How much of this a server actually needs

The honest answer is far less than we do.

A tick has 50 milliseconds to work with. A 20 microsecond wake-up is 0.04% of that, and the gap between six microseconds and one is not something a player feels. Had we stopped at "no stall longer than a few hundred microseconds" we would have captured nearly all of the benefit a server owner can actually perceive.

The part that genuinely matters is narrow. It is the multi-millisecond outliers. A 93 millisecond stall costs most of two ticks and a player sees that. A wake-up past 400 microseconds arriving every three seconds adds up to a visibly uneven evening. Those are real problems and fixing them is not optional. Nearly everything past that point is for us rather than for you.

So we should say why we keep going, because "more microseconds" is not a reason.

Headroom is cheaper than diagnosis. A node with a boring floor makes every other problem legible. When somebody reports a stutter we can rule the host out in a single step and go look at the plugin that is actually causing it, rather than spending an afternoon arguing about whether the machine is at fault. Most of the value of this work shows up in the arguments it ends rather than the microseconds it saves.

The margins also compound exactly where it is busiest. A ten player vanilla server on a quiet node would not notice any of this, and we are not going to pretend otherwise. A three hundred mod pack on a box with busy neighbours, on the evening everybody logs in at once, is where a sloppy floor turns into missed ticks. This configuration earns its keep in the worst hour of the week, which is the hour we are really selling.

We would rather tell you it is overkill than imply you need it. It is overkill, and we are not going to pretend otherwise. The tuning is a one-off cost on hardware we already own, and what it buys us is an afternoon we never spend arguing about whose fault the lag is.

What these results do not show

A paper that only lists its wins is marketing with footnotes, so here is the other column.

They are not TPS or MSPT figures. This test proves the kernel runs a thread when it said it would. A server's tick time depends on that, and on the plugins, the mods, the view distance, the entity count and the heap. Nobody should read a microsecond figure here as a promise about frame-to-frame smoothness in a game, and we are not offering one.

They were measured on an empty node under a synthetic load. That is the right way to isolate one variable, and it is not the same thing as a server full of players building a working redstone farm. Tick time under real player load is the next layer of our harness, using a bot swarm to generate genuine per-player cost, and it is not finished. When it is, we will publish it the same way, including the results that are unflattering.

The 7.2.13 column is a canary. Our fleet target is the long-term 6.18 series; 7.2.13 runs ahead of it on a single node so that we find out what breaks before the fleet does. Presenting it as what every node runs today would be false.

They are averages and worst cases rather than percentiles. A maximum is a single sample, and anyone serious about latency will want the distribution: the median, the 99th, the 99.9th and the shape of the tail between them. The test produces a full histogram and we have it; this table does not yet present it, and the next revision of this paper will.

One run per kernel. No repeat runs, so no run-to-run variance, which means the single-digit microsecond differences between our two builds should be read as indicative rather than settled. The 93 millisecond result does not need that caution. Nothing in a sound measurement produces a 93 millisecond outlier by accident.

We have not measured the firmware floor. System management interrupts happen below the operating system, cannot be scheduled around, and set the hard limit on how deterministic any of this can be. The tracer for it is compiled into our kernel and we have not yet run a campaign with it, so we do not know our own floor. It is the next thing we will measure after the application numbers.

The garbage collector is the dominant tail above all of this, and it would be dishonest to publish microsecond kernel figures without saying so. A collector pause target in the low hundreds of milliseconds makes every number in this paper a rounding error by comparison. That is why the kernel work is necessary rather than sufficient: it removes the stalls the operating system was adding, so that the ones left are the application's and can be attacked on their own terms. We A/B test JVM profiles, including a low-pause collector, against an identical synthetic player load for exactly that reason.

One box, one core, one workload generator. We have not run this across every processor model we own, and the numbers would not be identical if we did.

And the isolation work itself is live on the nodes we have converted and still rolling out across the rest. The honest consequence is that a server provisioned today may still be on the old quota configuration. That is an uncomfortable sentence to publish next to a set of favourable numbers, and the alternative is claiming a finished rollout that any customer could catch us on.

What we measure next

Three things, in order:

Tick time under real load. Worst MSPT and its high percentiles at a fixed player count, stock against tuned, on the same world seed and the same heap settings.

The density ceiling. Ramping concurrent servers on one host until the first one misses a tick, then comparing where that point falls on each configuration. The difference between the two is the honest definition of how much we could oversell and do not.

Cache evidence. Per-complex cache-miss and migration counters, to show that the cache isolation is doing what the design says rather than the latency improving for some other reason.

Reproduce it

Everything above uses standard tools: rt-tests for the measurement, stress-ng for the load, bpftrace and perf for the supporting counters. If you run your own hardware, the test costs you ten minutes:

# idle
cyclictest -m -p99 -a 2 -t1 -i 200 -d0 -D 300 -q -h 400

# then again with every other core loaded
taskset -c 0,1,3-15 stress-ng --cpu 0 --cpu-method matrixprod &
cyclictest -m -p99 -a 2 -t1 -i 200 -d0 -D 300 -q -h 400

Compare the two worst-case figures. If the loaded one is much larger than the idle one, your neighbours can reach your tick, and that is the problem all of the above exists to remove.

Why we publish the playbook

Every mechanism in this paper is public knowledge. Control group partitions, interrupt affinity, idle state limits and a custom kernel configuration are all documented, standard Linux, and any competent engineer could apply the lot this afternoon.

What cannot be copied is the thing underneath it. All of this assumes the cores are genuinely reserved, which means the box is not carrying three times the tenants it should. The moment capacity is sold twice, pinned cores are contended, reserved memory is not reserved, and a careful latency configuration measures a machine that is already losing. That is why overselling is the thing we argue about most, and why we would rather hand over the playbook than the margin.

If you want to check any of it, our dashboards are live rather than screenshots: Grafana and the status page. If you are running something unusual and want to know whether this applies to it, ask us in Discord. We like those questions.

Browse game hosting