DeepSeekMath · GRPO

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, Daya Guo · arXiv 2024

Read paper · arXiv:2402.03300

Construct relative advantages from a group of responses to the same prompt and update the policy with the GRPO objective.

What you will implement

Focus on the paper's GRPO module: group-relative advantages, clipped objectives, and response-level aggregation.

• 4. GRPO Group-Relative Policy Objective

  Implement within-group advantages and the clipped policy objective averaged over responses.

  Implement the advantage and loss functions; masking helpers, diagnostics, and sampling experiments are supplied.