Machines that think: from cloud to edge

There’s a comforting story about the edge, and it’s roughly the opposite of the one I took apart earlier. In this story, the data center bottleneck solves itself. Models get smaller, chips in phones and laptops get better, inference quietly decentralizes onto a few billion devices and the allocation politics of packaging capacity stop mattering quite so much. The pressure escapes through the edges.

I want to believe this story. It’s the optimistic one, and parts of it are true. But the numbers for this year say something more awkward: The edge is not draining the data center, and the reason it isn’t turns out to be the same reason the data center is constrained in the first place.

Start with what has actually arrived, because it isn’t nothing. The capability is real and it’s shipping. Counterpoint expects GenAI-capable smartphones to reach 45 percent of global shipments in 2026, up from 36 percent the year before, and AI-advanced PCs to pass 59 percent of global shipments, with neural processing units clearing Microsoft’s 40 TOPS bar as a matter of course. The models have come down to meet the hardware. Apple’s third-generation on-device foundation model is three billion parameters, with a larger sparse variant that activates only one to four billion at a time depending on the request. Could you have imagined this five years ago? It’s now the default tech stack on a consumer phone.

And yet. Deloitte’s 2026 predictions expect inference to account for roughly two-thirds of all AI compute this year, up from half in 2025. And expect almost all of it to still run in large data centers or enterprise on-premise systems. Edge inference remains a small fraction of total demand. Hundreds of millions of NPUs are shipping into devices, and they’re barely moving the aggregate needle.

So, we have a paradox: The edge is arriving everywhere and mattering very little. The usual explanation is that the on-device models simply aren’t good enough yet and that time will fix it. I think this explanation is wrong, or at least badly incomplete.

Here’s what I believe is actually going on. The constraint at the edge is not compute; it’s memory – again. A model has to fit in RAM to run. Not “fit on the device,” not “fit in flash,” but sit in memory alongside the operating system and whatever else you have open while working. That’s a hard ceiling, and it’s set by a bill of materials, not by a benchmark. Counterpoint’s analysts are explicit that memory “will remain a key factor determining how quickly GenAI expands beyond the high-end segment,” and that the extra DRAM needed to hold model weights is what keeps GenAI devices above roughly 400 dollar wholesale. You can put a 60 TOPS NPU in a mid-range phone; you can’t put an extra eight gigabytes of LPDDR in it and still hit the price point.

Which brings us to the part I find genuinely uncomfortable. That DRAM and the HBM stacked next to a data center accelerator are made in the same fabs, on the same wafers, by the same handful of vendors. IDC put it about as plainly as an analyst house ever does: Every wafer allocated to an HBM stack for an Nvidia GPU is a wafer denied to a consumer device. Memory is 15–20 percent of the bill of materials on a mid-range phone. IDC’s scenarios for 2026 run from a 2.9 percent smartphone market contraction with a 3–5 percent rise in average selling price to a 5.2 percent contraction with a 6–8 percent rise, and worse for PCs.

So, the edge isn’t an escape hatch from the substrate; it’s a competitor for it. The move from cloud to edge doesn’t route around the bottleneck I described in the previous part of this series; it queues up behind the same one. That’s the single most important thing to understand about this shift, and it’s almost entirely absent from how the edge gets discussed.

For a startup, this reframes a question that badly needed reframing. “Cloud or edge” isn’t an architectural preference to be settled by taste or by cost per token; it’s a question about which parts of your workload can live inside a memory budget you don’t control and can’t expand, on a device your customer already owns.

The trap is to treat the edge as the cheap version of the cloud. If your reason for going on-device is that inference costs too much, you’ve chosen a strategy that competes for scarce memory to escape scarce packaging and you’ve added a distribution problem on top. The far better reason is that some workloads aren’t merely expensive in the cloud but impossible there. Anything where the round trip is the product: low latency, operation without connectivity, data that must never leave the device for legal or commercial reasons. That’s a genuine edge workload. Those are defensible. “The same thing, but we pay less for it” isn’t.

The subtler consequence, and one I keep coming back to, is what running on the edge does to your ability to know whether the thing works. When inference happens in your data center, you see everything: every input distribution shift, every failure mode, every regression. When it happens on ten million devices, you own none of that by default. You’ve traded an allocation problem for an observability problem. The durable artifact in this world is the contract the model must honor plus the evidence that it still does, and the edge makes gathering that evidence structurally harder. Any team going on-device should be designing its evaluation telemetry at the same time as its quantization strategy, not two years later.

For large incumbents, the calculus, as usual, inverts. Cristiano Amon has been making the case for years and made it about as bluntly as possible at Davos this January: “Whoever has presence on the edge is going to win. The edge is where the humans are.” He sells the silicon that goes in the edge, so he would say that. But the strategic logic holds independently of who’s making the argument. If the agent becomes the primary interface, then whoever owns the device owns the default. Defaults, in consumer technology, have historically been worth more than features.

This is why the companies that control both the silicon and the operating system are in an enviable position, and why everyone else in the device business is in an increasingly awkward one. The awkwardness is again about memory. A vendor with the margin structure to swallow a rising memory bill will ship the AI features and eat the cost. A vendor without it has to choose between raising the price and cutting the RAM, and cutting the RAM means shipping a phone that has the NPU and not the memory to use it. Which is to say a phone that has the marketing and not the capability. The mid-tier squeeze I described in the previous post for compute buyers has an exact analogue one layer down the stack, and it will thin out the middle of the device market over the next two years.

For society, the on-device turn is the most encouraging development in this series so far, and it comes with a bill attached. The encouraging part is architectural rather than promissory. Data that never leaves the device can’t be breached at the provider, subpoenaed from the provider or quietly folded into the next training run. That’s a stronger privacy guarantee than any policy, terms-of-service clause or certification, because it’s a property of where the computation happens rather than of what someone promises to do with it afterward. For European enterprises in particular, a meaningful slice of the data sovereignty problem simply dissolves when the inference is local. This is the rare case where the commercially attractive option and the civically desirable one point the same way.

The bill is that AI capability is becoming a function of what hardware you can afford, at exactly the moment hardware is getting more expensive for reasons that have nothing to do with the buyer. The memory shortage is a direct consequence of data center demand, and it lands on the price of a phone in a market that has nothing to do with frontier models. We are, in effect, taxing the mid-range device to build the training cluster. If GenAI capability requires a device above a certain price, and that price is being pushed up by AI’s own appetite for memory, then the technology is drawing a line through the population and putting the people who would benefit most on the wrong side of it. That’s a distributional question, and better quantization won’t solve it.

One last point, and it’s the reason this post sits where it does in the series. Everything from here on, such as robots, factory cells, vehicles, instruments in a lab, is edge inference whether anyone calls it that or not. A robot arm can’t round-trip to a data center to decide whether to stop. A vehicle can’t make its control decisions while relying on a network. For the physical systems that fill the rest of this series, local inference isn’t an optimization; it’s a precondition. Everything I’ve described here as a constraint on phones and laptops becomes a design constraint on machines that move.

The models get the headlines. But intelligence only becomes infrastructure when it stops being somewhere you go and starts being something that’s simply present, at the point where the work happens. Mark Weiser saw the shape of this in Scientific American in 1991, long before there was anything to run: “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they’re indistinguishable from it.” That’s the destination. The memory bill is what we pay to get there.