LeWorldModel

LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels

Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, Randall Balestriero · arXiv 2026

Read paper · arXiv:2603.19312

Jointly learn action-conditioned latent predictions and plan actions with latent rollouts.

What you will implement

Differentiable prediction and regularization, autoregressive rollout, terminal cost, and CEM elite refitting.

• 58. LeWM Joint Prediction and Latent Rollout

  Implement next-state teacher forcing with both encoder paths differentiable, then recursively roll out action-conditioned latent states.

  Encoder, causal predictor and SIGReg are supplied by the caller. Implement temporal alignment, objective composition and single-state-context rollout.

• 59. LeWM Latent CEM Planning

  Score action sequences by their terminal latent state and refit the CEM sampling distribution from elites.

  Implement terminal scoring and diagonal-Gaussian elite statistics; the sampling loop, frozen transition model and action bounds are supplied.