Most on-device AI programs fail in the same place. A capable model is trained in the data center, handed to an optimization team, and squeezed through quantization, pruning and distillation until it fits on a phone. The result is reliably disappointing: a model that is too slow, too power-hungry, or too degraded in quality to ship.
I have spent years building generative models that run natively on mobile devices, and the pattern repeats. Compression applied after training can only trade away quality within decisions that were fixed months earlier. The depth and width of the network, the shape of the attention mechanism, the structure of the vision encoder: all of it was chosen with no reference to the silicon that must eventually execute it.
The fix is to stop separating model design from hardware design in the first place.
The limits of post-training compression
Post-training compression treats the accelerator as an afterthought, and accelerators punish that. A mobile NPU executes a specific set of operations extremely well and everything else poorly or not at all. When a server-designed architecture contains layers the accelerator cannot run natively, those layers fall back to the CPU, and a single fallback in the decode loop can erase every gain made elsewhere.
Quantization hits a similar wall. Pushing weights down to four bits or below after training routinely breaks model quality, because the network never learned to tolerate that precision. The teams that succeed at very low bit-widths bake quantization into training itself, which is an architecture and training decision, not a post-processing step.
There is a quality ceiling too. Every compression pass discards information from a design that was never shaped for the target. You can recover some of the loss with fine-tuning, but you cannot retrofit an architecture into being hardware-friendly.
The memory bandwidth problem
The deeper problem is that the bottleneck on a phone is rarely raw compute. Researchers at UC Berkeley have documented that peak hardware compute has scaled at roughly 3 times every 2 years, while DRAM bandwidth has grown only 1.6 times over the same period, which makes memory, not arithmetic, the dominant constraint for autoregressive decoding. A large language model generating text spends most of its time moving weights, not multiplying them.
Mobile silicon is racing ahead on the compute side regardless. Gartner expects NPUs exceeding 40 trillion operations per second to become standard in premium GenAI smartphones by 2027, with end-user spending on these devices already approaching 300 billion dollars in 2025. The hardware will have compute to spare while bandwidth stays scarce.
That asymmetry should drive architecture. A model designed to be compute-bound rather than memory-bound, through choices such as grouped-query attention, aggressive weight sharing or parallel decoding schemes, will outperform a nominally stronger model that drowns in memory traffic. None of those choices can be added after training.
What hardware-model co-design involves
Co-design means treating the accelerator’s instruction set, memory hierarchy and supported numeric formats as inputs to architecture search, on equal footing with benchmark accuracy. In practice it is an iterative discipline. You define a quality bar and a hard latency and power budget, generate candidate architectures across several parameter scales, and score each one against measured or carefully modelled on-device latency rather than FLOP counts, which lie.
The evidence that this matter is now public and reproducible. Meta’s MobileLLM work showed that at sub-billion scale, architecture choices such as deep-and-thin layouts and grouped-query attention beat parameter count as the driver of quality, a result that undermines the idea that architecture is settled and only scale matters. The decoding strategy belongs in the same loop: speculative decoding can deliver two to three times faster generation without changing model outputs, but only if the drafter model is designed alongside the main model so the two stay aligned.
Quantization moves upstream as well. If the target hardware rewards two-bit or four-bit weights, the model should be trained to live there from the start.
Lessons from a recent co-design program
A recent program I led makes the trade-offs concrete. We were building a multimodal language model for a flagship mobile platform, and rather than train one architecture and compress it, we evaluated more than twenty-five candidates between roughly 1.5 and 2.5 billion parameters, each scored against the accelerator’s operation support and bandwidth profile, with several trained to trillions of tokens before the final selection. Latency estimates from the silicon team fed every iteration, and any architecture that depended on unsupported operations was redesigned or cut.
The model that shipped was trained for very low-bit weights from the outset and paired with a vision encoder shaped around the same hardware. Prefill ran several times faster than our compress-after-training baseline, decode improved by a clear multiple, and quality held across every application that depended on it. The expensive part was discipline, not invention: dozens of candidates trained and discarded, and weekly arguments between model researchers and silicon engineers who had previously never shared a meeting.
Those arguments were the point. Each side surrendered something, the researchers gave up exotic layers, the hardware team exposed capabilities they had not documented, and the product was better for both.
Practical steps for engineering leaders
If you are deploying models to constrained devices, put your hardware and model teams inside one planning loop before pre-training begins, not after. Make measured on-device latency a formal gate at architecture selection. Budget for multiple candidate training runs, because the first architecture is rarely the right one, and treat quantization as a training decision rather than a packaging step.
The organizations shipping real on-device AI are not the ones with the cleverest compression pipeline. They are the ones whose models never needed rescuing.
