Teaching an AI Agent to Read Receipts (and Politely Decline Your Netflix Subscription)

June 2026 · 11 min read

Expense reports are the perfect agent problem, and I'll defend that. The input is a crumpled photo of a receipt. The output is structured data in an ERP system. In between sits a stack of company policy, a human who wants to spend zero minutes on this, and consequences if it goes wrong. Because the output is literally money.

This year I worked on an internal innovation project at Xebia: an expense assistant that lives in the company chat. You photograph a receipt, share it with the bot, have a short conversation, and the expense lands in the finance system. Along the way it reads the receipt with a vision model, checks what you're claiming against expense policy, and sometimes (this became my favorite part) has to tell a colleague, kindly, that their streaming subscription is not a business expense.

Two tiers, because chat and agents disagree about state

The architecture splits into two tiers, and the split exists because chat platforms and agent runtimes have opposite opinions about state.

Chat webhooks want a stateless receiver. Fast acknowledgments, retries, no memory. Agents are the opposite. A multi-turn conversation about correcting a receipt's date is state. So the ingress is a slim stateless proxy on Cloud Run that receives webhooks, and the agent itself, built on Google's Agent Development Kit, runs on Vertex AI Agent Engine, which owns sessions properly. One request through the system looks like this:

sequenceDiagram
    participant U as Employee
    participant Chat as Chat platform
    participant CR as Cloud Run proxy<br/>(stateless)
    participant FS as Firestore
    participant AE as Agent Engine<br/>(stateful)
    participant T as Tools
    U->>Chat: photo of receipt
    Chat->>CR: webhook event
    CR->>FS: thread_id → session_id?<br/>(per-thread lock, LRU cache)
    FS-->>CR: session found / created
    CR->>AE: run agent turn (session)
    AE->>T: add_pending_expense(receipt)
    T->>T: vision parse: merchant, date, total
    T->>FS: hash check — duplicate?
    AE-->>CR: "€23.50 at that ramen place, right?"
    CR-->>Chat: reply in thread
    Note over U,AE: ...corrections, policy checks...
    U->>Chat: "ok, submit"
    AE->>T: submit_pending_expenses()
    Note over T: before_tool_callback:<br/>explicit confirmation? only then run
    T->>T: POST to finance system

The glue between the tiers is the mapping from chat thread to agent session, kept in Firestore. This little component grew more engineering than I expected. Chat platforms happily deliver events concurrently, so two messages in a fresh thread can race to create "the" session. Solved with per-thread locks. And a Firestore read on every single message adds latency the user feels, so an LRU cache (500 entries, one-hour TTL) sits in front. A cache, locks, and eviction policy for what began as "just store the thread ID somewhere". Agent state is where the boring engineering hides.

Everything runs twice, of course. Separate dev and prod projects, same deployment shape, with CI running lint, type checks and tests before anything ships. And the finance-system client has a mock twin that mimics its API without moving money, which is what lets the whole conversation flow be tested end to end by machines and demoed to stakeholders without anyone accidentally expensing a test ramen dinner.

The agent's hands

The agent's capabilities are a small set of tools around one core object, the pending expense. Parse a receipt into one, correct a field, list them, merge several receipts into a single claim, discard one, and finally submit. The vision parsing is the least of it. Gemini reads merchants, dates and totals off photographed receipts (JPEGs, PNGs, PDFs) with very little coaxing. The conversation design around correction mattered far more, because the failure mode isn't "can't read the receipt". It's "read it slightly wrong and the human needs a frictionless way to say so".

The merge tool earns a special mention because it came straight from watching real behavior. A conference trip produces four receipts that belong on one claim, and the natural human instinct is to dump all four photos into the chat at once. An agent that forced four separate submissions would be correct and infuriating. combine_pending_expenses exists because the tool surface should match how people actually behave, not how the ERP's data model wishes they would.

Two mechanisms carry the safety load. First, duplicate detection. Every receipt image is hashed and checked against previously submitted hashes per employee. Because the most likely expense fraud isn't fraud, it's a colleague accidentally sharing the same photo twice a week apart, and the agent should catch that without anyone getting embarrassed.

Second, the submit gate. Submission to the finance system is the one irreversible action, so it's wrapped in a before_tool_callback that refuses to run unless the user has explicitly confirmed in that conversation. The LLM cannot talk itself into submitting. The gate is code, not prompt.

The model proposes. Deterministic code disposes.

That's the pattern I'd generalize from this whole project. Anything that moves money, deletes data, or emails a human gets a hard gate outside the model's reach. Policy checking, by contrast, lives in the prompt. The expense rules are a markdown file loaded into the agent's instructions at boot, so finance can edit a document instead of filing a ticket against my code, and the agent explains rejections in natural language, citing the actual rule. Policy-in-prompt, gates-in-code. Each in the layer that suits it. Deterministic mappings stay in code too. The translation from expense category to the finance system's ledger accounts is a static table, because a language model should never be creative about which ledger your ramen lands in.

Around the agent sits a small plugin stack that does the unglamorous production work. Uploaded files are persisted as artifacts automatically. A context filter keeps conversation history from dragging irrelevant baggage into each turn. Structured logging captures every step. And a reflect-and-retry plugin gives the agent one shot at noticing its own tool-call errors and correcting course before a human sees the failure. None of these are features. All of them are the difference between a bot colleagues use twice and one they keep using.

How do you unit-test a conversation?

Then comes the question that defines whether an agent is a demo or a product. The receipt parser works. The tools work. But the behavior? Does it warn on a policy violation but still let a human decide? Does it refuse the clearly personal expense? Does it stay polite on the third correction? How do you regression-test that?

Our answer became an evalset. 25 scripted conversations, each a full multi-turn scenario with the receipt attachments, the user's messages, and the expected behavior at each turn. The names read like a policy FAQ. The over-limit dinner that should be flagged but not blocked. The personal subscription that should be declined outright. The receipt older than the claiming window. Each case is a small story with a correct ending, and together they're nine hundred lines of JSON that constitute the agent's behavioral contract.

Grading is rubric-based, done by a judge model that is deliberately not the model powering the agent, scoring relevance and helpfulness against a 0.8 threshold. Separate judge matters. Models grade their own family's phrasing generously, and you want the grader arguing with the agent, not agreeing with it.

The supporting trick that made the suite livable: VCR-style cassettes. LLM interactions are recorded once and replayed deterministically in CI, keyed by request hash. The hashing matters, because storing full prompts would bloat every cassette and leak receipt contents into test fixtures. Fast tests run without an API key and without flakiness. The genuinely-live tests are marked slow and run when it matters. Change a prompt and you re-record with --record-mode=rewrite. And a re-recording is itself a signal, a diff that says "behavior changed here, was that intended?".

Twenty-five conversations is not many. It's also more behavioral coverage than most agents in production have today, and the discipline of writing them changed the agent's design more than once. Scenarios that were hard to script turned out to be scenarios where the intended behavior was genuinely undecided. The evalset became the place where the team argued about what the agent should do, which is a much better venue for that argument than production.

What the corporation taught the agent

Building agents inside a company adds constraints no framework tutorial mentions, and they shaped real decisions. Receipts are personal data, so storage got deliberate retention tiers. Receipt images live 90 days and then vanish. Observability data lives a year in production, long enough to investigate a disputed expense. Traces live a month. Each number is a small negotiation between debuggability and data minimization, decided per data class rather than defaulting to "keep everything forever", which is what happens when nobody decides.

Everything the agent does is traced end-to-end with OpenTelemetry, because "the AI decided" is not an acceptable answer to a finance department asking why an expense was submitted. And the demo-to-trust pipeline runs through the eval suite. Showing stakeholders 25 named scenarios with pass marks moved more skepticism than any live demo did. A demo shows the happy path once. An evalset shows you went looking for the unhappy ones.

The expense assistant is in production now. Colleagues photograph receipts, argue briefly with a robot about dates, and get reimbursed. And somewhere in the eval suite, a scripted user tries to expense a personal subscription every single CI run, and the agent politely declines. Forever. There are worse legacies.

References