Requests for Research 2.0

OpenAI

We’re releasing a new batch of seven unsolved problems which have come up in the course of our research at OpenAI.

Warmups

If you’re not sure where to begin, here are some solved starter problems.

⭐ Train an LSTM to solve the XOR problem: that is, given a sequence of bits, determine its parity. The LSTM⁠(opens in a new window) should consume the sequence, one bit at a time, and then output the correct answer at the sequence’s end. Test the two approaches below:

• Generate a dataset of random 100,000 binary strings of length 50. Train the LSTM; what performance do you get?

• Generate a dataset of random 100,000 binary strings, where the length of each string is independently and randomly chosen between 1 and 50. Train the LSTM. Does it succeed? What explains the difference?

⭐ Implement a clone of the classic Snake⁠(opens in a new window) game as a Gym⁠(opens in a new window) environment, and solve it with a reinforcement learning⁠(opens in a new window) algorithm of your choice. Tweet⁠(opens in a new window) us videos of the agent playing. Were you able to train a policy that wins the game?

Requests for Research

⭐⭐ Slitherin’. Implement and solve a multiplayer clone of the classic Snake⁠(opens in a new window) game (see slither.io⁠(opens in a new window) for inspiration) as a Gym⁠(opens in a new window) environment.

⭐⭐⭐ Transfer Learning Between Different Games via Generative Models. Proceed as follows:

• Train 11 good policies for 11 Atari⁠(opens in a new window) games. Generate 10,000 trajectories of 1,000 steps each from the policy for each game.

• Fit a generative model (such as the Transformer⁠(opens in a new window)) to the trajectories produced by 10 of the games.

• Then fine-tune that model on the 11th game.

• Your goal is to quantify the benefit from pre-training on the 10 games. How large does the model need to be for the pre-training to be useful? How does the size of the effect change when the amount of data from the 11th game is reduced by 10x? By 100x?