Kimi K3 Architecture: Inside the First Open 2.8T Parameter Model Engineered for Compiler and Hardware Synthesis

Kimi K3 introduces a 2.8-trillion parameter architecture utilizing Delta Attention and a 16-of-896 Mixture of Experts routing configuration. The model demonstrates advanced autonomous capabilities in low-level systems engineering, including MLIR compiler design, custom GPU kernel optimization, and standard-cell ASIC synthesis.

Architectural Foundations: KDA and Sparse MoE Scaling

The newly introduced Kimi K3 is a 2.8-trillion parameter model featuring native vision capabilities and a 1-million-token context window. To optimize information flow across deep sequences, the architecture integrates Kimi Delta Attention (KDA) alongside Attention Residuals (AttnRes). Computation is managed via a scaled-up Mixture of Experts (MoE) configuration. Utilizing the Stable LatentMoE framework, the model activates 16 out of 896 total available experts. These structural enhancements, combined with refined training pipelines, deliver a 2.5× improvement in scaling efficiency over the previous Kimi K2 generation. Full weights for the model are scheduled for open release by July 27, 2026.

While its overall performance remains behind proprietary models like Claude Fable 5 and GPT 5.6 Sol, K3 operates at the frontier of open-source capabilities. It defaults to max thinking effort, with lower-effort processing modes planned for future rollouts.

Autonomous GPU Kernel Optimization and Compiler Generation

In systems evaluations, Kimi K3 was deployed in sandboxed environments to profile, rewrite, and optimize GPU kernels over 24-hour cycles. The workload included refining AttnRes, KDA, and 512-head-dimension MLA kernels across both NVIDIA H200 and alternative-vendor GPGPUs. Kimi K3 performed competitively with Claude Fable 5 and outperformed older proprietary engines, processing trajectories within acceptable numerical precision tolerances.

Beyond isolated kernels, the model synthesized MiniTriton, a functional GPU compiler built from scratch. MiniTriton features a tile-level Intermediate Representation (IR) layer situated over MLIR, automated optimization passes, and a proprietary PTX code-generation pipeline. In roofline benchmarks, the compiled outputs matched or exceeded standard Triton and torch.compile performance. MiniTriton also successfully compiled and ran end-to-end nanoGPT training, maintaining stable convergence loss curves that tracked reference baselines.

Standard-Cell EDA ASIC Synthesis

To test long-horizon agentic hardware engineering, Kimi K3 undertook an autonomous 48-hour chip design challenge. Utilizing open-source Electronic Design Automation (EDA) tools targeted at the Nangate 45nm cell library, the model designed a specialized accelerator optimized to run a nano model.

The resulting silicon layout occupies a 4 mm² footprint, closes timing at 100 MHz, and hosts 1.46 million standard cells. It incorporates 0.277 MB of SRAM alongside an INT4 Multiply-Accumulate (MAC) array equipped with fused dequantization logic. In simulation, the synthesized architecture sustains a decoding throughput exceeding 8,700 tokens per second.

Knowledge Work and Scientific Workflows

In scientific computing, Kimi K3 translated complex astrophysics literature into executable code. It reproduced the I-Love-Q universal relations by cross-validating over 20 academic papers, evaluating 300+ equations of state, identifying mathematical inconsistencies in published formulas, and writing a 3,000-line Python numerical pipeline accompanied by an interactive HTML visualization dashboard.

For enterprise knowledge tasks, the model supports recursive self-improvement loops. In a case study mapping 42 years of the ASIC industry, K3 conducted over 2,800 web searches, executed 1,100 terminal data pulls, and parsed 11,000 pages of quarterly reports and PDF documentation to generate comprehensive, data-driven visualizations and slides.