Terraform, GKE, and the 150-Millisecond Question

March 2026 · 10 min read

In the previous post I wrote about training a multi-task intent classifier for a large Dutch e-commerce platform. This one is about everything around it: the Terraform that stands the system up, and the load tests that told us whether it could actually take a punch.

The question the whole serving side had to answer was blunt. P90 under 150 milliseconds at 1000 requests per second. Search-path latency budgets don't negotiate. Every millisecond we ate was a millisecond stolen from ranking, and nobody upstream was volunteering to give it back.

Three stacks, three lifecycles

The infrastructure repo split into three independent Terraform stacks, and that split turned out to be the most important design decision in it: training, serving, and stress-test. Not modules within one stack. Separate roots with separate state.

flowchart TB
    subgraph TRAIN["training stack (semi-permanent)"]
        C["Cloud Composer<br/>(Airflow)"] --> V["Vertex AI<br/>custom jobs (L4 GPU)"]
        V --> GCS["GCS bucket<br/>models + data"]
        V --> BQ["BigQuery<br/>experiment tracking"]
        AR["Artifact Registry"] --> V
    end
    subgraph SERVE["serving stack (lives with the model)"]
        LB["LoadBalancer"] --> POD["model server pods<br/>FastAPI + TorchScript"]
        INIT["init container<br/>gcloud storage cp model"] --> POD
        HPA["HPA @ 70% CPU"] -.scales.-> POD
        PROM["Prometheus + Grafana"] -.scrapes /metrics.-> POD
    end
    subgraph STRESS["stress-test stack (lives for an afternoon)"]
        LW["Locust workers<br/>(autoscaled)"] --> UI["Locust web UI :8089"]
    end
    GCS --> INIT
    LW ==>|1000 RPS| LB

The reason is lifecycle. Training infrastructure (Composer, the bucket, Artifact Registry, BigQuery) exists semi-permanently. Serving infrastructure lives as long as the model serves. And the stress-test stack exists for an afternoon at a time. Mixing resources with different lifespans in one state file means every terraform destroy becomes a nervous negotiation with the plan output. Separating them made destruction boring. And cheap. We tore down the stress-test stack after every session and the serving stack between test phases.

terraform destroy is a cost-control habit, not an emergency procedure.

The workflow around the stacks stayed deliberately plain. A terraform.tfvars.example in each root, copied and filled per environment. Plan reviewed by an actual human, apply, work, destroy. No wrapper scripts, no Terragrunt, no cleverness. With three small roots instead of one big one, plain Terraform stayed comfortable. Most IaC tooling complexity is compensation for state files that grew past what a person can read.

Learnings from the training stack

Cloud Composer, which is managed Airflow, was the piece with the most personality. The main thing nobody warns you about: provisioning takes 20 to 40 minutes. Not occasionally. Every time. You learn to structure your workday around Composer applies the way you'd plan around an oven. Start it, go do something else, come back when it beeps. Iterating on Composer config by re-applying is a day gone.

Two patterns from that stack I've reused since. First, passing configuration to DAGs via software_config.env_variables in Terraform. Bucket names, project IDs and regions all flow from the same variables that created the resources, so the DAG can't drift from the infrastructure. When the bucket is renamed in one .tf file, the DAG finds out at the next apply, not at 3am. Second, giving Composer its own service account with exactly the roles it needs (Vertex AI user, storage admin on one bucket, BigQuery data editor) instead of the default compute account. The role list gets long. Composer itself needs both a service-agent extension role and a worker role, which cost me an evening of IAM error archaeology. But the result is an orchestrator that can touch what it should and nothing else.

Even the workload sizing carried a lesson in restraint. The Airflow scheduler and web server run happily at 0.5 CPU and 2 GB each, with workers autoscaling between one and three. Our DAGs don't do work. They delegate work to Vertex AI and wait. An orchestrator sized like a compute cluster is a smell. Ours is sized like a receptionist, which is what it is.

Learnings from the serving stack

The GKE side is a cluster with an autoscaling node pool (preemptible where the workload tolerates it, at 60 to 70% cheaper), a deployment for the model server, an HPA keyed on CPU around 70%, and a LoadBalancer service in front.

The detail I like most is how the model gets onto the pod. The container image contains only code. The model artifact lives in GCS. An init container runs gcloud storage cp -r to pull the model and lexical data into a shared volume before the main container starts, and the main container mounts it read-only. Model updates don't require image rebuilds, the image stays small, and read-only mounts mean no pod can quietly corrupt its own weights. Workload Identity binds the Kubernetes service account to a GCP service account with storage.objectViewer. No key files anywhere in the cluster.

Probes deserve their own paragraph, because ML servers break the assumptions web-app defaults encode. A pod that's still loading a model into memory answers TCP but shouldn't get traffic. So the readiness probe waits for the actual /health endpoint, which flips only after the TorchScript artifact is loaded and a warm-up inference has run, and the initial delay is generous by web standards. An impatient readiness probe on an ML deployment is a machine for serving 503s during every rollout.

GPU nodes taught their own lesson. Setting gpu_driver_version = "DEFAULT" in the node pool config and letting Google manage driver installation removed a whole category of "why is the accelerator invisible" debugging that I did not miss.

A cluster whose only job is violence

Here's the thing about load testing at 1000 RPS: the load generator is a distributed system too. Run Locust from your laptop and you're measuring your WiFi. Run it from one VM and around a few hundred RPS the generator saturates before the target does, and your "results" are a portrait of the wrong bottleneck.

So the stress-test stack is a second GKE cluster whose only purpose is to attack the first one. Terraform deploys autoscaled Locust workers plus a web UI exposed on port 8089, sized by a single variable, so "give me a bigger army" is a one-line change. The Locustfile itself stays simple. Users posting realistic queries, with 0.1 to 0.5 seconds of think time between requests:

class GKEUser(HttpUser):
    wait_time = between(0.1, 0.5)

    @task(10)
    def predict_single_query(self):
        self.client.post("/v1/predict",
                         json={"query": random.choice(TEST_QUERIES)},
                         name="/v1/predict [single]")

The test queries deliberately span the intent spectrum. Exact product hunts, browsy inspiration phrases, how-to questions. A load test that only sends one query shape gets flattered by every cache in the path. A test-stop listener prints P50/P90/P95/P99 and flags any breach of the 150ms line, and every run ends with an HTML report you can attach to a decision instead of a vibe.

What the load tests actually found

The headline finding wasn't in the infrastructure at all. It was in the API process. The single biggest latency lever was request batching. The FastAPI service collects incoming requests in an async queue, and a worker drains up to 16 of them every 10 milliseconds into one forward pass, resolving each caller's future individually. Transformers are throughput machines. Feeding them one query at a time wastes almost everything they're good at. Batching tripled sustainable throughput on identical hardware, and the 10ms collection window costs less latency than it saves by an order of magnitude.

Try it

Request Batching Simulator — The batching trade-off above is interactive here. Tune arrival rate, batch size, and collection window, then watch p50/p99 latency and throughput respond in real time — the same dynamics that tripled throughput on this serving stack.

The counterintuitive runner-up: torch.set_num_threads(1). PyTorch's default eagerness to parallelize a single small inference across cores meant threads fighting each other under concurrent load. One thread per worker process, more worker replicas. Smoother latency, better P99, less CPU. We found both of these only because the load tests made the bottlenecks visible. Neither showed up under polite manual testing.

Observability closed the loop between test sessions. The API exports a Prometheus histogram, prediction_latency_seconds, with buckets chosen around the SLA: 10ms up through 1s, dense near 100 to 300ms where the truth lives. Grafana reads it. The point of matching bucket edges to the SLA is simple. A histogram with generic buckets can tell you the P90 was "somewhere between 100 and 500ms", which is a shrug wearing a chart. Ours could answer the 150ms question directly. Next to latency sat a drift monitor comparing live prediction distributions against training via KL divergence, because an API that's fast and confidently wrong passes every load test you'll ever run.

That ended up as my summary of the whole exercise. Terraform made the experiments cheap to run and cheap to clean up. Locust made the results honest. Prometheus made the improvements provable. Infrastructure-as-code isn't just reproducibility for production. It's what makes destructive experimentation affordable enough to actually do.

References