An Introduction to Q-Learning Part 2/2

Unit 2, part 2 of the Deep Reinforcement Learning Class with Hugging Face 🤗

⚠️ A new updated version of this article is available here 👉 https://huggingface.co/deep-rl-course/unit1/introduction

This article is part of the Deep Reinforcement Learning Class. A free course from beginner to expert. Check the syllabus here.

⚠️ A new updated version of this article is available here 👉 https://huggingface.co/deep-rl-course/unit1/introduction

This article is part of the Deep Reinforcement Learning Class. A free course from beginner to expert. Check the syllabus here.

In the first part of this unit, we learned about the value-based methods and the difference between Monte Carlo and Temporal Difference Learning.

So, in the second part, we’ll study Q-Learning, and implement our first RL agent from scratch, a Q-Learning agent, and will train it in two environments:

1. Frozen Lake v1 ❄️: where our agent will need to go from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoiding holes (H).
2. An autonomous taxi 🚕: where the agent will need to learn to navigate a city to transport its passengers from point A to point B.

This unit is fundamental if you want to be able to work on Deep Q-Learning (Unit 3).

So let’s get started! 🚀

• Introducing Q-Learning
  • What is Q-Learning?
  • The Q-Learning algorithm
  • Off-policy vs. On-policy
• A Q-Learning example

Introducing Q-Learning

What is Q-Learning?

Q-Learning is an off-policy value-based method that uses a TD approach to train its action-value function:

• Off-policy: we'll talk about that at the end of this chapter.
• Value-based method: finds the optimal policy indirectly by training a value or action-value function that will tell us the value of each state or each state-action pair.
• Uses a TD approach: updates its action-value function at each step instead of at the end of the episode.

The Q comes from "the Quality" of that action at that state.

If we take this maze example:

If we recap, Q-Learning is the RL algorithm that: