I Built a 2-Day Course on AI Agents. The Agents Were the Easy Part.

May 2026 · 10 min read

This spring I built a two-day training on production multi-agent systems. Eleven modules, about twelve teaching hours, built on Google's Agent Development Kit. I went in expecting the hard part to be explaining agent architectures. I came out having spent most of my design energy on something else entirely: convincing engineers that the interesting problems start after the agent works.

Everything happens to a coffee company

The first real design decision was narrative. Every module in the course happens to the same fictional company. Cosmic Bean, a coffee roaster with suppliers, an order backlog, and escalating operational problems. Students don't "learn the loop agent pattern". They fix Cosmic Bean's supplier-monitoring process, which happens to need a loop agent.

I did this because of a failure mode I kept seeing in technical trainings, including my own earlier ones: exercises as isolated puzzles. Students solve them, feel productive, and a week later can't say when they'd reach for the pattern. A continuous story forces every pattern to arrive with its why attached. The transition text between modules literally explains what went wrong at the company that makes the next technique necessary. It also compounds. By module eight, students know the product catalog and the supplier names, so nobody burns ten minutes re-reading a fresh scenario. The fixed cost of world-building pays out eleven times.

The shape of the two days

The arc runs from "an agent that answers questions" to "an agent fleet you'd trust in production", and the module list is really two courses with a hinge in the middle:

flowchart TB
    subgraph BUILD["day 1 — build (what your demo needs)"]
        s00["s00 ADK refresher<br/>agents, tools, state"] --> s01["s01 workflow patterns<br/>sequential · parallel · loop"]
        s01 --> s02["s02 custom orchestration<br/>(ships a failing test)"]
        s02 --> s03["s03 MCP<br/>consume + build servers"]
        s03 --> s04["s04 agent-to-agent<br/>services, not imports"]
    end
    s04 --> s05["s05 EVALUATION<br/>⟵ the pivot point"]
    subgraph SHIP["day 2 — ship (what your incident needs)"]
        s05 --> s06["s06 guardrails<br/>PII, rate limits, topics"]
        s06 --> s07["s07 eval mastery<br/>rubrics · A/B · hallucination"]
        s07 --> s08["s08 deploy + observe<br/>FastAPI → Cloud Run"]
        s08 --> s09["s09 testing pyramid<br/>prompt optimization"]
        s09 --> s10["s10 drift + feedback<br/>capstone"]
    end

Day one earns its keep with the less obvious modules. The MCP module has students both consume a Model Context Protocol server (a filesystem server exposing Cosmic Bean's policies and supplier documents) and understand what building one implies. And those business documents aren't props. They become the ground truth that faithfulness evaluation checks against, two modules later. The agent-to-agent module makes a point I wish more production teams had internalized: the quality-review agent is exposed as a service with its own server, not imported as a library. Two agents that talk over a protocol can be owned by two teams, deployed on two schedules, and replaced independently. It's microservices lessons replayed at agent scale, and teaching it as architecture rather than framework API is what makes it stick.

The pivot point

Module five is the hinge, and I told students so out loud. The first half of this course is the part your demo needs. The second half is the part your production incident needs.

Designing the evaluation modules taught me more than any other part. The framework distinguishes three kinds of graders, and the taxonomy holds up outside the classroom.

Code graders are deterministic checks. Did the agent call the right tool, with valid arguments, no more than N times, does the response contain the order ID. Cheap, fast, zero flakiness, and they catch a shocking share of real regressions. Model graders are LLM-as-judge with a rubric, scoring 1 to 5 with reasoning, for the qualities code can't check: tone, helpfulness, whether the refusal was polite. Faithfulness graders check whether a response is actually grounded in the documents the agent retrieved, which is the difference between an agent that consults the supplier contract and one that improvises something contract-shaped.

The uncomfortable part of teaching model graders is that the judge is also an LLM, and it's also flaky. Rather than hide that, I made it the lesson. The eval harness runs every model-graded case three times and takes a majority vote, and score differences under 0.3 are treated as ties, not improvements. Watching students discover that their "prompt improvement" was actually judge noise (same agent, different verdicts) did more for their production instincts than anything I could have said.

If your eval can't survive being run twice, you don't have an eval.

The course also hands out a scaling ladder for test suites, because "how many test cases do I need" is the first question every team asks, and the answer is a function of what's at stake. 10 to 20 cases while you're learning what good looks like. 50 to 100 before anything user-facing ships. 100 to 500 once evals gate your CI. And 1000+ when you're choosing between foundation models and the decision is worth real money. The eval suite grows with the consequences of being wrong. Most teams I meet are one rung lower than their blast radius warrants.

Test design gets the same treatment as grading. The base suite is 17 cases, and then students design their own edge cases against a checklist of the ways users actually behave. Misspelled product names, mixed Dutch-English messages, 500-word rambles with one actual question inside, and a prompt-injection attempt. Because someone in every cohort needs to watch an agent cheerfully obey "ignore your instructions" before guardrails feel urgent rather than theoretical.

Traces or it didn't happen

Under everything from module five onward sits observability. Arize Phoenix with OpenTelemetry instrumentation, capturing every LLM call, tool call and agent step as spans. The pedagogical payoff is immediate. When a student's agent does something baffling, "read the trace" replaces "add print statements", and multi-turn sessions group so you can watch a conversation degrade turn by turn. By module seven they're using Phoenix's datasets and experiments to version their test collections and compare runs, and the prompt playground to iterate on rubric wording with template variables. The tooling matters less than the reflex it installs: an agent you can't trace is an agent you're debugging by séance.

The intentional bug

Module two ships with a failing test. On purpose. It's not documented as such. Students hit it, assume they broke something, investigate, and discover the test was wrong about what the agent should do. The point lands harder than any slide: in agent systems, your test suite encodes opinions about correct behavior, and those opinions can be wrong. Several students told me afterward this was the moment the evaluation material clicked. One told me it was cruel. Both reviews accurate.

What nobody tells you about running LLM labs

The logistics of a classroom full of people hammering an LLM API deserve their own module, honestly. Costs first, because everyone asks. With Gemini 2.5 Flash, two full days of exercises runs $2 to $5 per student. Fifteen students, $30 to $75 total. Genuinely cheap. If you have quota.

Because the free tier's 15 requests per minute is not a rounding error away from enough. The classroom needs 2,000+ RPM. Fifteen students each running an A/B comparison (two prompt variants × 17 test cases × 3 judge retries) is a hundred-plus requests per student per exercise, all launched in the same five minutes, because classrooms synchronize. The eval-heavy modules are the spike. I now provision quota the way I'd provision for a load test, which after the modules five and seven experience is exactly what a classroom is.

The other logistics saver: GitHub Codespaces as the standard environment. 120 free core-hours a month means a 2-core machine covers the whole course without anyone entering a credit card, and "it works on my machine" stops being a sentence anyone says on day one.

What I'd tell past me

Watching engineers move through the material confirmed the bet behind the pivot-point structure. The build-phase modules went fast. Experienced developers pattern-match orchestration code quickly. The rooms slowed down, in the good way, at evaluation. Rubric design sparked actual arguments about what "helpful" means, and the A/B exercise produced the quietest, most focused half hour of both days.

Which is the real learning from building the thing. Agents are code, and engineers already know how to learn code. Evaluating agents is judgment under uncertainty, and that's the part worth twelve hours of everyone's time.

References