Request Batching Simulator

The trick that tripled our throughput, as a toy you can break.

Transformers are throughput machines. A forward pass over 16 inputs costs nowhere near 16 times a single input, because the fixed overhead (kernel launches, memory movement) dominates small workloads. So production ML servers play a counterintuitive game: when a request arrives, they wait. A few milliseconds, hoping others arrive too, then run the whole group through the model at once. Each request pays a tiny waiting tax so that all of them get a massively cheaper ride.

This simulation runs that exact mechanism. Requests arrive randomly at your chosen rate, a batcher collects up to N of them for at most W milliseconds, and the "model" charges a realistic sub-linear cost per batch. Watch what happens at 800 RPS with batching off (batch size 1): the queue grows without bound and p99 goes vertical. That's a real Tuesday. Then give it a 10ms window and a batch of 16, and watch the same traffic become boring. Boring is the goal.

The model here charges 8ms + 2ms × batch^0.7 per forward pass, which is roughly how real inference scales: a batch of 16 costs about 3× a batch of 1, not 16×. That gap is the entire business case. The real story, including the load-testing cluster we built to find this: Terraform, GKE, and the 150-Millisecond Question →