Back to writing

The Last Mile of AI

Most of the work in applied AI happens after the demo works. Notes from shipping agents into power plants, refineries, and financial institutions.

An AI demo only has to work once, in front of the right audience.

The distance between that demo and a system a power-plant operator trusts at 2am is the last mile, and it’s where most AI projects die. I’ve spent my career in that mile: threat detection scanning a million enterprise mailboxes, agents reasoning over inspection data from robots crawling boiler walls, and customer-facing voice and chat agents for financial institutions at Sierra. The same problems showed up in all three.

The demo is the easy 80%

A RAG pipeline that answers questions about your documents takes an afternoon. A RAG pipeline that answers questions about inspection reports from a 40-year-old boiler, where a wrong answer means someone defers maintenance on a component that fails takes months, and most of those months aren’t spent on the model.

They’re spent on the boring things: what happens when the source data is missing, or contradictory, or formatted the way a field technician formats things at the end of a twelve-hour shift. The model is rarely the bottleneck. The data it has to read usually is.

Evals are the product spec

In traditional software, tests verify behavior you designed. In AI systems, evaluations define behavior you can’t fully design. On the agents I’ve shipped, the evaluation suite came first and served as the specification. If you can’t write down what a good answer looks like for your hundred ugliest real-world cases, you don’t yet know what you’re building.

Simulated conversations, LLM-as-a-judge evaluations, and production call review sound like QA hygiene. In practice they are what let you change a prompt or a model on Friday without hearing about it from a customer on Monday.

Deployment is mostly a trust problem

Latency and context windows are solvable. The harder part of putting AI into a high-stakes environment is that a person who has done their job well for twenty years now has to decide whether to believe a machine. That decision happens over weeks, in the gap between what the system claims and what they later check with their own eyes.

You earn that trust the same way a new hire does: be right about small things repeatedly, be transparent about uncertainty, and never be confidently wrong about the same thing twice. Systems that show their work, with sources cited, confidence expressed, and escalation paths honored, get adopted. Systems that present answers as oracles get quietly ignored, no matter how good the benchmark numbers were.

Why I keep choosing the last mile

There’s a version of AI engineering that optimizes benchmarks, and a version that changes how a refinery schedules maintenance or how a bank serves a customer at midnight. The second one is slower, messier, and involves a lot more meetings in rooms without whiteboards. It is also where you find out whether any of it works.

My advice to a team starting a deployment: plan and staff for the last mile in the first week, not after the demo goes well.