PPO

Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov · arXiv 2017

Read paper · arXiv:1707.06347

Clip probability ratios in the policy objective to limit the incentive for excessively large updates.

What you will implement

Implement the clipped surrogate and its gradient behavior; value loss, normalization, and diagnostics are supplied.

• 2. PPO Clipped Policy Objective

  Implement the pessimistic clipped objective and verify gradients for both advantage signs.

  Implement only clipped_surrogate; value loss, entropy, normalization, and training are supplied.