Dr. GRPO

Understanding R1-Zero-Like Training: A Critical Perspective

Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin · arXiv 2025

Read paper · arXiv:2503.20783

Analyze length and difficulty biases in GRPO and adjust advantage scaling and loss normalization.

What you will implement

Implement two corrections: remove group standard-deviation scaling and normalize by a fixed generation budget.

• 5. Dr. GRPO Length and Difficulty Bias Corrections

  Remove group standard-deviation scaling and use a fixed length denominator to correct two normalization biases.

  Implement the two normalization corrections; the token objective, masking helpers, and diagnostics are supplied.