Block Diffusion

Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models

Marianne Arriola et al. · ICLR 2025 Oral

Read paper · arXiv:2503.09573

Combine autoregression across blocks with diffusion within blocks.

What you will implement

Single-forward attention masks for concatenated noisy and clean tokens.

• 43. Block Diffusion Attention

  Process noisy blocks and clean context in one attention pass without leaking clean targets.

  Implement the specialized training mask and dense attention; discrete diffusion sampling, NELBO and KV caching are outside this module.