1. Executive Summary and Selection Criteria

By September 2026, local inference of foundation models has stopped being a hobbyist curiosity and become an operational requirement. Data sovereignty rules, per-token API economics, and the simple need to iterate without a network round-trip have pushed engineering teams toward portable workstations that can host real models on real silicon. The selection criteria, however, have shifted. Raw TFLOPS, the marketing number that dominated laptop spec sheets for a decade, is now a secondary metric. What governs the experience is a triad of physical constraints: memory bandwidth (GB/s), unified VRAM addressing capacity, and sustainable thermal dissipation (TDP under load). This guide breaks down the quantitative criteria needed to evaluate portable hardware capable of running frontier open-weight models such as Llama 4 Scout (with its 10M-token context window), Gemma 4 (12B) for edge deployment, and quantized variants of DeepSeek-V4.1-Flash. We examine the empirical advantages of unified memory architectures over discrete mobile GPUs, and how FP8 and INT4 quantization formats redefine what is actually feasible inside a chassis that weighs less than 2.5 kg.

2. The Inference Performance Equation: Bandwidth vs. Model Size

Autoregressive inference is a memory-bound process during the decoding phase. Every generated token requires transferring the entirety of the model weights from physical memory into the processing registers of the GPU or NPU. Compute units sit idle waiting for data; the bus, not the ALU, sets the ceiling. The theoretical maximum throughput follows a relationship so simple it is often ignored in marketing material:

Throughput (tokens/s) = Memory Bandwidth (GB/s) ÷ Model Size in Memory (GB)

Run a 31-billion-parameter model quantized to 4 bits, roughly 18 GB in memory including metadata, on a system with 136 GB/s of bandwidth, and the theoretical ceiling is 7.5 tokens per second. No amount of TFLOPS changes that arithmetic. The following table compares the memory architectures of reference mobile platforms as of September 2026:

3. Next-Generation Quantization: FP8 vs. INT4 in Development Environments

Quantization is no longer a last-resort compression trick. It is a native compilation standard, and the choice between formats has real consequences for what a model can still do.

FP8 (E4M3 and E5M2)

Native hardware support for 8-bit floating-point formats in NVIDIA Blackwell Mobile and AMD RDNA4 microarchitectures enables inference with semantic precision loss that is, in practice, imperceptible against an FP16 baseline. The E4M3 format, one sign bit, four exponent bits, three mantissa bits, is the preferred choice for weights and activations during inference, thanks to its finer resolution near zero. The E5M2 format is reserved for scenarios where dynamic range matters more than precision. The practical payoff: FP8 preserves the complex reasoning and coding behavior of models like DeepSeek-V4.1-Flash without requiring recalibration of attention layers.

INT4 (GGUF / AWQ)

4-bit integer quantization remains the pragmatic choice for running large models, Gemma 4 12B being the canonical example, on workstations with tight VRAM budgets. Activation-aware quantization (AWQ) and the flexibility of the GGUF format managed by llama.cpp shrink a model to roughly a quarter of its original weight footprint. The trade-off is measurable: perplexity degrades on extreme mathematical logic tasks. But on systems limited by the memory bus, execution throughput can triple, and that trade is often worth making.

4. Local Inference Ecosystems: Apple MLX vs. llama.cpp

The orchestration software determines the real efficiency of the silicon. Compiler-level optimization and unified memory management are the difference between a fluid workflow and system instability.

• Apple MLX: Built specifically for Apple Silicon, this open-source framework eliminates the need to duplicate data between CPU and GPU. Using unified memory addressing, a 70B-parameter model quantized in Q4 (approximately 38 GB) can reside entirely in the M4 Max's shared memory, letting GPU cores and the neural engine access the same buffers without PCIe copy penalties.
• llama.cpp: The quintessential cross-platform inference engine. Its ability to offload model layers to system RAM when they exceed the discrete GPU's VRAM is vital on Windows and Linux. The catch is that data crossing the PCIe Gen 5 x16 bus introduces a severe bottleneck: as soon as the model does not fit entirely in dedicated GPU memory, tokens per second collapse.

5. Local Inference Benchmarks and Real-World Performance

The following figures come from standardized tests in controlled development environments, measuring sustained token generation over an 8,192-token context window on representative models of the current ecosystem:

Technical note on results: The configuration with a discrete RTX 5090 Mobile GPU delivers the highest throughput on smaller models, such as Llama 4 Scout, that fit comfortably within its 16 GB of GDDR7 VRAM. When scaling to models like Gemma 4 (12B) or DeepSeek-V4.1-Flash, throughput plummets due to the data exchange bottleneck with system RAM through the host interface. This is precisely the scenario where unified memory solutions (Apple and AMD) maintain linear, predictable degradation instead of a cliff.

6. The Thermal Limit and Power Efficiency

The physics of a laptop chassis impose strict limits on sustained computing. A high-performance mobile GPU can draw up to 120 W instantaneously, but no standard chassis under 2.5 kg can dissipate that thermal load continuously without frequency degradation within minutes. During prolonged inference or LoRA fine-tuning workloads, the thermal behavior of systems diverges sharply:

• Systems with Discrete GPU (NVIDIA/Intel): Combined power draw of host processor and dedicated GPU typically exceeds 150 W under continuous inference load. Fans run at maximum acoustic capacity, often above 50 dBA, and after roughly 15 minutes of sustained execution, GPU core frequencies are reduced by 15% to 25% to keep temperatures below the safety junction (typically 85 °C).
• Unified Architecture Systems (Apple Silicon / AMD APU): By integrating compute engine and memory on the same silicon substrate, power consumption drops substantially. The M4 Max under maximum sustained inference load does not exceed 55 W of total system power. This allows maintaining peak token generation throughput indefinitely with minimal acoustic impact (below 35 dBA), a decisive advantage for prolonged deployments of local AI agents.

7. Architecture Decision Matrix and ROI

To optimize return on investment in AI workstations for engineering teams, purchasing guidelines should follow the development profile rather than the spec sheet.

Profile A: Model Engineers and Fine-Tuning

Aimed at professionals performing hyperparameter optimization, lightweight training, and execution of medium-sized models (up to 31B). The goal is to maximize addressable memory capacity and avoid tensor fragmentation.

• Recommendation: Systems with high-capacity unified memory (minimum 96 GB or 128 GB). Platforms based on Apple Silicon M4 Max or AMD Ryzen AI Max+ offer the best cost per gigabyte of usable VRAM, allowing the loading of massive models that would otherwise require desktop configurations with multiple dedicated GPUs.

Profile B: AI Application and Agent Developers

Aimed at software development that consumes hybrid APIs and deploys small local models (7B to 14B) for fast code autocompletion, syntactic analysis, and low-latency inference.

• Recommendation: Workstations with latest-generation NVIDIA RTX discrete GPUs (minimum 16 GB GDDR7 VRAM). The CUDA ecosystem remains the industry standard for integration libraries, and FP8 performance for small models on these cards surpasses any other mobile alternative on the market.