Proximal Policy Optimization (PPO)

Unit 8, of the Deep Reinforcement Learning Class with Hugging Face 🤗

⚠️ A new updated version of this article is available here 👉 https://huggingface.co/deep-rl-course/unit1/introduction

This article is part of the Deep Reinforcement Learning Class. A free course from beginner to expert. Check the syllabus here.

⚠️ A new updated version of this article is available here 👉 https://huggingface.co/deep-rl-course/unit1/introduction

This article is part of the Deep Reinforcement Learning Class. A free course from beginner to expert. Check the syllabus here.

In the last Unit, we learned about Advantage Actor Critic (A2C), a hybrid architecture combining value-based and policy-based methods that help to stabilize the training by reducing the variance with:

• An Actor that controls how our agent behaves (policy-based method).
• A Critic that measures how good the action taken is (value-based method).

Today we'll learn about Proximal Policy Optimization (PPO), an architecture that improves our agent's training stability by avoiding too large policy updates. To do that, we use a ratio that will indicates the difference between our current and old policy and clip this ratio from a specific range [1−ϵ,1+ϵ] [1 - \epsilon, 1 + \epsilon] .

Doing this will ensure that our policy update will not be too large and that the training is more stable.

And then, after the theory, we'll code a PPO architecture from scratch using PyTorch and bulletproof our implementation with CartPole-v1 and LunarLander-v2.

Sounds exciting? Let's get started!

• The intuition behind PPO
• Introducing the Clipped Surrogate Objective
  • Recap: The Policy Objective Function
  • The Ratio Function
  • The unclipped part of the Clipped Surrogate Objective function
  • The clipped Part of the Clipped Surrogate Objective function
• Visualize the Clipped Surrogate Objective
  • Case 1 and 2: the ratio is between the range
  • Case 3 and 4: the ratio is below the range
  • Case 5 and 6: the ratio is above the range
• Let's code our PPO Agent

For two reasons: