Policy Gradient with PyTorch

Unit 5, of the Deep Reinforcement Learning Class with Hugging Face 🤗

⚠️ A new updated version of this article is available here 👉 https://huggingface.co/deep-rl-course/unit1/introduction

This article is part of the Deep Reinforcement Learning Class. A free course from beginner to expert. Check the syllabus here.

⚠️ A new updated version of this article is available here 👉 https://huggingface.co/deep-rl-course/unit1/introduction

This article is part of the Deep Reinforcement Learning Class. A free course from beginner to expert. Check the syllabus here.

In the last unit, we learned about Deep Q-Learning. In this value-based Deep Reinforcement Learning algorithm, we used a deep neural network to approximate the different Q-values for each possible action at a state.

Indeed, since the beginning of the course, we only studied value-based methods, where we estimate a value function as an intermediate step towards finding an optimal policy.

Because, in value-based, π exists only because of the action value estimates, since policy is just a function (for instance, greedy-policy) that will select the action with the highest value given a state.

But, with policy-based methods, we want to optimize the policy directly without having an intermediate step of learning a value function.

So today, we'll study our first Policy-Based method: Reinforce. And we'll implement it from scratch using PyTorch. Before testing its robustness using CartPole-v1, PixelCopter, and Pong.

Let's get started,

• What are Policy-Gradient Methods?
  • An Overview of Policy Gradients
  • The Advantages of Policy-Gradient Methods
  • The Disadvantages of Policy-Gradient Methods
• Reinforce (Monte Carlo Policy Gradient)

What are Policy-Gradient Methods?

An Overview of Policy Gradients

Why do we optimize the policy directly by estimating the weights of an optimal policy using Gradient Ascent in Policy Gradients Methods?

Let’s take a simple example:

The Policy Gradient algorithm (simplified) looks like this:

This has two consequences: