Introduction
Most inference infrastructure is designed around someone waiting. A user staring at a cursor, a copilot that has to respond before the developer loses their train of thought, a search box that feels broken if it takes too long to respond. This assumption shaped everything downstream: leaving headroom so you won’t offer a laggy product.
That's the right call for interactive traffic. The problem is that a huge fraction of real AI work isn't interactive at all. It gets submitted to a queue and collected later, completion doesn’t have to be done immediately, or it runs in a loop for minutes or hours while a coding agent works through a task and a human reviews the result afterward. It can wait. And when you run that work on infrastructure tuned for the person-waiting case, you pay interactive prices for work that never needed them. Even worse, your workloads don’t finish or you get hit by rate limits.
Work that can wait is async inference, and it deserves its own inference.
What actually makes a workload "async"
This isn't only about "background agents," and it isn't about batch size. The real question is simpler: is there a latency SLA on any individual response that a human will feel?
Interactive work is judged on time-to-first-token and inter-token latency. A human perceives the delay, so the delay affects the product. Async work is judged on throughput and total cost to completion. One is optimized against response time. The other is optimized against dollars per task. It’s pretty obvious what async means for batch workloads, but it’s worthwhile examining the case of coding or background agents. A coding agent or a research agent does have a rough time budget. You'd like the pull request ready by morning, not next week, but it's a loose, aggregate budget measured in minutes or hours across the whole run, not a per-token one that a human feels tick by tick. No one is watching call number 47 of 300 return. What matters is that the whole job finishes, reliably, by the time someone needs it, and what it costs to get there. That is the async profile exactly.
There's one more thing that separates agents from classic batch jobs. Batch workloads are dependent on a common system prompt. Background or coding agents are not: an agent's calls are dependent. Each step decides the next, so you can't hand the whole workload over as a single file up front the way you can with a million documents. Agents need the low-friction, high-concurrency access of an online endpoint, but with the throughput-first economics of batch. That combination is precisely what interactive infrastructure doesn't give you and what file-based batch APIs can't express.
How to think about optimizing spend on Async AI
Cost per token is, roughly, the hourly cost of your hardware divided by the tokens it produces in that hour. As we've argued before, energy is a small slice of the cost of serving a token, so shaving your electricity price barely moves the number. But doubling effective throughput can cut cost per token by roughly half. Throughput is the dominant lever. And async workloads are precisely the regime where you can pull that lever all the way, determining when you need the workload done, and controlling costs that way.
So async AI isn't just a category of workload. It's the setting in which inference economics truly matter. Same silicon, dramatically more tokens per dollar, just because the work can wait.
Where async AI actually shows up
1. Coding agents
Autonomous coding agents are among the most demanding async workloads there are. A single task is not one call. It's a long loop: read the code, plan, edit, run the tests, read the failures, try again, dozens to hundreds of dependent model calls over minutes to hours. No human watches those intermediate steps; they review the finished pull request.The token volume is large, bursty, and completion-critical: what you care about is that every PR is ready by the time engineers sit down, and that the job doesn't die halfway through because it tripped a rate limit. Latency per step barely registers against a run that is measured in minutes. Throughput, concurrency, and reliable completion are everything, and because the calls are dependent, this work can't be shipped off as a static overnight batch. It needs an endpoint that behaves like an online one but is priced and scheduled like batch.
2. Background and multi-step agents
Long-running research agents, monitoring agents, and overnight multi-step workflows share the same shape as coding agents, pointed at different problems: gather sources and synthesize a report, watch a data feed and act on it, chain a dozen tool calls into a workflow that a human will read later. What matters is total throughput and getting the entire workflow to complete without a step silently failing or getting throttled. These jobs are bursty and deadline-flexible, and, just like coding agents, they generate their calls dynamically as they go, so they want throughput-first serving that still accepts requests one dependent step at a time.
3. Synthetic data and distillation
Generating training data, augmentations, preference pairs, and teacher outputs for distillation is an offline pipeline feeding your next training run. Volume and throughput are the entire point. Nothing about it is interactive. It's compute in service of a future model, run whenever capacity is cheapest.
The shape they share
Pull the thread through all of these and two common profile appear:
- The first is the static batch: a large, often known-in-advance body of work (documents, embeddings, eval rows) that you can hand over all at once.
- The second is the dynamic agent stream: work that generates itself as it runs, one dependent step at a time, and can't be submitted up front.
What unites them is what they don't have: a latency SLA on any individual response that a human perceives. Both are judged on throughput and total cost to completion rather than per-token speed. Both are tolerant of scheduling. Both are frequently bursty. That profile is the opposite of interactive traffic, and it wants the opposite serving strategy: deep batching where the work allows it, high utilization, and scheduling that fills the fleet instead of protecting headroom, all the while still accepting of the dynamic, dependent request pattern that agents depend on.
The mistake to avoid
The common failure is running async work on infrastructure tuned for interactive traffic and paying for all of it on work that could have been batched and run hot. Another mistake is letting async jobs share a fleet with latency-critical traffic, where a backfill quietly contaminates the tail latency of your live product. Different SLAs want different serving modes. Mixing them costs you on both ends: the async work runs expensively, and the interactive work runs slower.
Agents have their own version of this trap, in both directions. Point a fleet of coding or research agents at a standard interactive endpoint and you'll pay interactive prices for millions of tokens no human is waiting on. You will also hit rate limits that stall runs halfway through, which is often the harder problem of the two. But reach for a file-based batch API to fix the cost side and you'll find it doesn't fit either: agents can't submit their work as one upfront file, because every call depends on the last. Agents fall in the gap between the two default options, which is exactly why throughput-first serving that still accepts dependent, online-style requests matters for them.
Match the mode to the workload
A great deal of tokens, and a great deal of cost, live in async AI: the processing, the embeddings, the synthetic data, the evals, the backfills that nobody sees. Make your inference match async AI, not fight it.