OpenAI Five

Given a learning algorithm capable of handling long horizons, we still need to explore the environment. Even with our restrictions⁠, there are hundreds of items, dozens of buildings, spells, and unit types, and a long tail of game mechanics to learn about—many of which yield powerful combinations. It’s not easy to explore this combinatorially-vast space efficiently.

OpenAI Five learns from self-play (starting from random weights), which provides a natural curriculum for exploring the environment. To avoid “strategy collapse”, the agent trains 80% of its games against itself and the other 20% against its past selves. In the first games, the heroes walk aimlessly around the map. After several hours of training, concepts such as laning⁠(opens in a new window), farming⁠(opens in a new window), or fighting over mid⁠(opens in a new window) emerge. After several days, they consistently adopt basic human strategies: attempt to steal Bounty⁠(opens in a new window) runes from their opponents, walk to their tier one⁠(opens in a new window) towers to farm, and rotate heroes around the map to gain lane advantage. And with further training, they become proficient at high-level strategies like 5-hero push⁠(opens in a new window).

In March 2017, our first agent⁠(opens in a new window) defeated bots but got confused against humans. To force exploration in strategy space, during training (and only during training) we randomized the properties (health, speed, start level, etc.) of the units, and it began beating humans. Later on, when a test player was consistently beating our 1v1 bot, we increased our training randomizations and the test player started to lose. (Our robotics team concurrently applied similar randomization techniques to physical⁠ robots⁠ to transfer from simulation to the real world.)