Kimi K3

Kimi K3: Open Frontier Intelligence

Kimi Team · arXiv 2026

Read paper · arXiv:2607.24653v2

Implement Kimi K3's core mechanisms for sequence mixing, depth aggregation, and bounded activations: KDA, Block AttnRes, and SiTU-GLU.

What you will implement

Independently implement bounded decay and delta-rule recurrence, attention over residual blocks, and SiTU gating; recreate the report's decay and activation curves.

• 33. Kimi K3: Bounded Decay and Delta Memory

  Implement channel-wise retention and delta-rule writes, then read the updated recurrent memory.

  Implement bounded_decay and recurrent_kda only; projections, convolutions, output gates and GPU kernels are outside this exercise.

• 34. Kimi K3: Block Attention Residuals

  Assemble depth sources, select earlier representations with a learned query, and aggregate original values.

  Implement assemble_sources and depth_attention; the full Transformer, block accumulation and layer projections are external.

• 35. SiTU-GLU Smooth Bounded Activation

  Smoothly cap both activation branches while retaining local SwiGLU behavior.

  Implement softcap and SiTU-GLU on projected tensors; MoE routing and the full FFN are outside this module.