DPO

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn · NeurIPS 2023

Read paper · arXiv:2305.18290

Turn the implicit reward margin of preference pairs into an objective that directly optimizes a language model.

What you will implement

Implement the preference loss and implicit rewards; token alignment and sequence log-probability calculation are supplied.

• 3. DPO Preference Objective and Implicit Rewards

  Compute the preference loss and implicit rewards from policy and reference sequence scores.

  Implement only dpo_loss; sequence log-prob aggregation and the toy training experiment are supplied.