Porting Gemma 4 MoE to Ivy Bridge: Resolving AVX2 Dependencies in llama.cpp
A technical breakdown of porting Google's Gemma 4 26B MoE model to a dual Intel Xeon E5-2690 v2 platform without AVX2/FMA3 support. By patching ik_llama.cpp to bypass missing vectorized fused kernels, the system achieves 5.2 tokens/second at Q8_0 precision.
The Hardware Constraints of Ivy Bridge
Running modern Mixture-of-Experts (MoE) architectures like Google’s Gemma 4 26B-A4B on legacy hardware requires navigating severe instruction set limitations. While recent optimization efforts target Broadwell-era silicon (which supports AVX2 and FMA3), older architectures like Intel's Ivy Bridge (v2 Xeons) are limited to AVX1. Running the optimized ik_llama.cpp fork on a dual Xeon E5-2690 v2 setup (2013 vintage, DDR3 memory, no GPU) fails at compile time and runtime due to hardcoded assumptions of AVX2 vectorized execution paths.
The Fused Kernel Dispatch Bug
Disabling GGML_USE_IQK_MULMAT at build time is intended to strip out AVX2-dependent matrix multiplication kernels and force a fallback to standard scalar or SSE implementations. However, doing so exposes a discrepancy between the model graph builder and the compute dispatcher.
The Gemma 4 MoE feed-forward network (FFN) emits two specialized operations: MOE_FUSED_UP_GATE (a per-expert gate and up-matrix multiplication fused with SwiGLU) and FUSED_UP_GATE (its dense counterpart). Within the compute dispatcher, both operations are conditionally gated under `#if GGML_USE_IQK_MULMAT`.
When GGML_USE_IQK_MULMAT is compiled out, the dispatcher's switch block lacks cases for these enums. Because of this, the execution falls through to default, leaving the destination tensors for every expert FFN uninitialized. With 30 layers and 8 active experts per token, roughly 240 tensors per forward pass go uncomputed, consuming arbitrary data from the residual stream memory buffer.
Debugging the Uninitialized Residual Stream
The runtime symptom of this dispatch failure is fluent-looking multilingual gibberish rather than standard segmentation faults or NaN outputs. Because the underlying memory buffer holds small positive floats, the uninitialized state offsets the residual stream systematically.
Instrumenting raw logits prior to sampling reveals a clean bias: a mean logit of +16 for the first predicted token and approximately 80% of the 262K vocabulary sitting at positive logits. The byte-identical outputs across single- and multi-threaded runs at temperature zero point directly to an uninitialized buffer rather than race conditions or numerical overflow.
Restructuring the Graph Builder Fallback
The fix, submitted under pull request ikawrakow/ik_llama.cpp#2138, routes around the missing AVX2 fused kernels by modifying the graph builder under `#if !GGML_USE_IQK_MULMAT`.
- Splits the tensor into two distinct ggml_view_3d slices.
- Executes two separate ggml_mul_mat_id calls (which utilize stock, non-IQK GGML code paths).
- Combines the intermediate outputs using ggml_fused_mul_unary with a SiLU activation.
If the gate and up weights are already distinct, the split phase is skipped, executing the same two matrix multiplications and fused unary operations directly. The dense ggml_fused_up_gate path receives a parallel modification. This approach replaces the fused AVX2 kernel with multiple discrete operations that already possess robust scalar and SSE fallbacks.
Resolving Compilation and CI Failures
Achieving a successful build on AVX1 hardware also requires cleaning up stray AVX2 references in the fallback paths. The scalar `#else` branches for quantize_row_q8_0_x4 and quantize_row_q8_1_x4_T in iqk_quantize.cpp incorrectly referenced AVX2-specific helpers like hsum_i32_8. These have been rewritten into portable scalar loops. Additionally, `#if GGML_USE_IQK_MULMAT` guards were wrapped around lingering IQK calls in ggml.c and ggml-quants.c.
Finally, the continuous integration stub files in the iqk source tree were synchronized with iqk_mul_mat.h. This resolved missing `
Runtime Configuration Notes
When running this configuration, the --run-time-repack flag must be omitted. This flag reorders quantized weights into an AVX2-only interleaved layout (Q8_0_R8) at startup, which garbles the model output on AVX1 targets.
Despite the overhead of running two separate matrix multiplications instead of a single fused kernel, execution is heavily bound by DDR3 memory bandwidth rather than compute. The patched fallback delivers 5.2 tokens/second during decode phases and 16 tokens/second during prompt evaluation on Q8_0 quantized weights.