llama-modchip Enables CPU-Only Execution of 284B DeepSeek-V4 on 64 GB RAM via Cache-Aware SIMD Tiling
llama-modchip introduces a patched ggml/llama.cpp runtime that compresses the 284B DeepSeek-V4-Flash checkpoint to 56.56 GiB using 1.71 bpw quantization, allowing full CPU execution within 64 GB system memory. By combining multi-column AVX-512BW register tiling, non-temporal cache streaming, and Stockfish-PVS speculative drafting, the system achieves 44.4 tok/s prompt prefill without GPU hardware.
Custom Quantization Architecture and RAM Allocations
Standard UD-IQ1_S GGUF builds of DeepSeek-V4-Flash (284.33B total parameters, 13.17B active) occupy 76.87 GiB of disk space and exceed the ~62.5 GiB usable physical RAM boundary of 64 GB Linux hosts, resulting in page thrashing or out-of-memory kills. `llama-modchip` compresses the checkpoint to 56.56 GiB (1.71 effective bits per weight), capping peak VmRSS at 57.61 GiB under a 256K context window and running with zero NVMe swap usage.
The runtime applies targeted bit-width allocations across specific layer types. High-sensitivity tensors—including embeddings, dense layer 0, 43 shared experts, and MLA key/value projections—are preserved in Q4_K, Q5_K, or FP32. Attention output matrices (`wq_b` and `attn_output`) are transcoded into a custom `GGML_TYPE_Q2_0` (type ID 40) format termed Fractal-64, which packs 64-weight blocks with 8 sub-scales anchored by a BF16 scale. The 256 routed MoE experts primarily rely on IQ1_S (1.5625 bpw) with analytical SwiGLU-H2 second-order scale matching, while sensitive boundary layers step up to IQ2_XXS (2.06 bpw) and IQ1_M (1.75 bpw). By default, the system evaluates all Top-6 routed experts per layer with loss-free logit cosine similarity compared to unpatched execution.
AVX-512 Cache Tiling and Non-Temporal MoE Streaming
Standard CPU inference bottlenecks stem from cache thrashing and memory bus saturation during matrix-vector operations. `llama-modchip` replaces single-column quantization loops during prefill with a multi-column `{8,4,3,2}` tiling schedule partitioned across ZMM AVX-512BW registers, L1D (32 KB), L2 (1 MB), and L3 (64 MB) caches. On a 16-core AMD Zen 5 processor, this kernel optimization raises prompt prefill throughput from 25.0 tok/s to 44.36 tok/s.
To prevent cold routed expert parameters from evicting active KV cache lines from the CPU cache hierarchy during inference, the MoE routing path injects non-temporal memory hint instructions (`_MM_HINT_NTA`). Streaming expert weights directly through registers bypasses L3 cache allocation. Attention mechanisms utilize register-stationary MQA8 FlashAttention to limit memory bus traffic. On multi-channel platforms, such as an 8-channel Threadripper setup, register tiling saturates available DDR5 memory bandwidth to yield up to 111.6 tok/s decode throughput on 30B MoE models.
Speculative Execution and Fast Shared-Memory Context Restoration
For smaller MoE models such as Nemotron-3.5-30B, the framework integrates Stockfish-PVS v2, a 1.64 MB Zobrist n-gram self-speculative drafter operating inside L2/L3 cache memory. When speculative draft tokens fail validation, an $O(1)$ rollback mechanism handles state updates for recurrent and Multi-Head Latent Attention (MLA) structures.
Agentic workloads leverage direct POSIX shared memory (`/dev/shm`) for context slot snapshots. Preserving active KV state snapshots in shared memory reduces context restoration latency to 13.1–32.6 ms for complex prompt structures, removing re-prefill penalties during multi-turn execution.