The Token Factory: Why AI’s Next Era Will Optimize for Throughput, Not Speed
The biggest change in AI infrastructure may come from accepting slower answers, imperfect uptime, and whatever hardware and power the market has undervalued.
For the past few years, AI infrastructure has been optimized around a very specific moment: a person types a prompt and waits.
That experience makes latency king. The first token should appear immediately, the rest should stream smoothly, and the service should almost never go down. Achieving that responsiveness requires premium chips, fast interconnects, highly reliable data centers, and spare capacity ready for unpredictable traffic.
But what if most future AI work does not happen while a person is watching?
That was the idea that stayed with me after watching Patrick O’Shaughnessy’s conversation with Neil Movva, co-founder of Sail Research, on why AI could become dramatically cheaper. Movva’s argument is not simply that today’s chatbot can be served more efficiently. It is that the dominant AI workload may change—from real-time conversations to background agents that work for hours, days, or even weeks.
Once the workload changes, the entire infrastructure stack can be reconsidered.
From instant answers to long-running work
An interactive assistant has a strict latency budget because the user is waiting. A background agent has a completion goal.
Imagine asking an agent to:
- research thousands of sources and produce a continuously updated report;
- inspect a large codebase, run tests, locate defects, and attempt fixes;
- monitor production telemetry and investigate unusual behavior;
- evaluate many possible designs before recommending one;
- continuously test software from an attacker’s perspective.
Whether the next step takes 200 milliseconds or 20 seconds may not matter. What matters is whether the agent finishes useful work by tomorrow morning—and how much that completed work costs.
This changes the optimization target from tokens per second for one user to total tokens produced per dollar across the system. Latency still matters for chat, copilots, and other interactive products. But it matters far less for a job running unattended overnight.
Movva predicts that AI usage could move toward a roughly 90/10 split in favor of background work. Whether or not that exact ratio proves correct, the direction feels plausible: agents become more valuable when they are allowed to keep working after we close the window.
Throughput beats latency
GPUs are most economical when they stay busy. One way to keep them busy is batching—processing many requests together instead of immediately handling each request in isolation.
Larger batches can make an individual request wait longer, but they allow the hardware to produce more total tokens in the same period. That trade-off is painful in a live chat and often invisible in a background job. Modern inference systems already use continuous or in-flight batching to increase utilization; NVIDIA describes it as a way to execute different requests together and replace completed sequences with new ones while the remaining requests continue (NVIDIA).
The token-factory model pushes this idea further. Instead of asking, “How fast can this one response arrive?”, it asks, “How much useful intelligence can this facility produce today?”
That is a factory question, not a chatbot question.
There are no bad chips—only bad prices
Most of the AI industry competes for the same top-tier accelerators. That makes sense when maximum performance, mature software support, and low-latency communication are essential. It may make less sense when the goal is simply to generate the greatest number of acceptable tokens per dollar.
Movva summarizes the alternative with a memorable line: “There are no bad chips, only bad pricing.”
The practical implication is hardware arbitrage. Instead of insisting on one vendor or one perfect architecture, a token factory can search for underpriced compute: older GPUs, AMD accelerators, TPUs, specialized inference chips, or other capacity that does not fit the standard hyperscaler design.
The winning chip is not necessarily the fastest. It is the one whose purchase price, power consumption, memory, software effort, and output combine into the lowest cost per useful token.
This is harder operationally. A heterogeneous fleet demands portable software, strong scheduling, hardware-aware optimization, and a control plane capable of routing each workload to the right machine. But that complexity can become an advantage: the operator is no longer bidding for exactly the same scarce resource as every frontier AI lab.
The hidden cost of memory
Cheap compute alone is not enough. Long-running agents create long contexts, and long contexts create memory pressure.
During inference, models retain a representation of earlier tokens in a structure commonly called the KV cache. It prevents the model from recomputing the entire history for every new token, but it can consume enormous amounts of memory as conversations and agent trajectories grow. In some configurations, this working memory can become as important as the model weights themselves.
That makes memory compression and smarter cache management central to token economics. If infrastructure can store less information per token, reuse repeated context, or move memory efficiently between storage tiers, the same hardware can support more concurrent work.
For background agents, this may matter more than making the next visible token appear a fraction of a second sooner.
Smaller data centers, cheaper power, imperfect uptime
The same inversion applies to data centers.
Frontier-model training rewards enormous, tightly connected clusters because thousands of accelerators must participate in one coordinated run. Inference is more divisible. Independent agent workloads can be distributed across locations, paused, retried, checkpointed, or moved elsewhere.
Movva’s proposal is to aggregate many relatively small sites—around 1 MW each—instead of waiting for a single massive block of power. A distributed fleet could use locations and energy supplies that hyperscalers overlook, including intermittent solar and wind.
It could also accept something closer to 95% availability at an individual site. That would be unacceptable for an interactive service if it meant users regularly saw failures. For resilient background work, however, site availability does not have to equal job reliability. A strong control plane can checkpoint progress, reroute tasks, and retry work on another cluster.
The distinction is important: the goal is not unreliable AI. It is reliable outcomes built on cheaper, less reliable components.
Cheap tokens do not merely reduce the bill
The most important consequence of cheaper inference is not saving money on the prompts we already send. It is making entirely new behavior economical.
Today we ration AI. We decide whether a task is worth a prompt, whether an agent should get another attempt, and how much context to provide. At sufficiently low token prices, the default can reverse. Agents can explore several solutions, verify their own work, monitor systems continuously, and remain available in the background without every additional step feeling expensive.
Software engineering offers a simple example. A coding assistant that answers when asked is useful. An affordable background engineering agent could continuously map a codebase, identify risky dependencies, reproduce flaky tests, evaluate accessibility, benchmark performance, attempt refactors, and prepare evidence for human review. Most attempts may lead nowhere. That is acceptable if the cost of exploration is low and the final output is verifiable.
This resembles the economics of other general-purpose technologies: when the unit cost falls sharply, consumption does not remain fixed. People invent uses that were previously irrational. Lower token prices may therefore increase total compute demand rather than reduce it.
A different definition of performance
The AI infrastructure race is usually described as a competition to build larger clusters with faster chips. The token-factory thesis offers a different path: combine overlooked chips, stranded or intermittent power, aggressive batching, compressed memory, distributed facilities, and tolerance for delay.
None of these ingredients is individually magical. Their power comes from aligning the system with a different workload.
If AI remains primarily an interactive product, latency and premium infrastructure will continue to dominate. If AI becomes a population of persistent agents working beyond the user’s attention, then the winning systems may look less like supercomputers and more like industrial operations—distributed, opportunistic, fault-tolerant, and relentlessly optimized for output per dollar.
The next leap in AI may not be an answer that arrives faster.
It may be useful work that never would have been affordable to start.
This article reflects my interpretation of ideas discussed in “Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper”. The 1,000× cost goal, 90/10 workload prediction, and site-level infrastructure assumptions are forward-looking arguments from the conversation, not established industry outcomes.