I reduce loss functions
for a living.

Search intent classification for a large Dutch e-commerce platform grew into three BERT models: buying intent, brand/creator recognition, and category prediction over roughly fifteen thousand labels. Three models meant three forward passes on every query, at search-traffic volumes.

All three were fine-tuned from the same domain-adapted encoder, and only the top layers diverged. So we composed them into a single system model: one pass through shared layers 0–9, branching at layer 10 into three task-specific heads. One tokenization, one encoder run, three answers.

The humbling lesson came from evaluation: the model looked great on synthetic held-out sets, then dropped eight F1 points on real labeled queries. Real data dominated evaluation from that day on.

The serving side of that classifier had one non-negotiable: P90 under 150 milliseconds at 1000 requests per second. Three Terraform stacks with separate lifecycles — training, serving, and a stress-test cluster whose only job was to attack the serving cluster with Locust, because at 1000 RPS the load generator is a distributed system too.

The headline finding wasn't in the infrastructure. It was in the API process: collecting requests in an async queue and draining up to 16 of them every 10 milliseconds into one forward pass tripled sustainable throughput on identical hardware. Runner-up: torch.set_num_threads(1), because PyTorch's eagerness to parallelize small inferences meant threads fighting each other under load.

My first iOS app: a personal AI second brain where the model runs entirely on the phone. No API, no server — the app's only network request is downloading the model itself, once. The context window is 8,192 tokens on a good device, and everything must fit.

The constraint became the architect. Search never touches the LLM: hybrid BM25 + vector retrieval answers in under half a millisecond. Token budgets are enforced by a cheap estimator validated against the real tokenizer in CI. And privacy is a compile-time property: a test scans every source file and fails the build if networking primitives appear outside the one sanctioned download function.