WebGPT: Improving the factual accuracy of language models through web browsing

OpenAI

We’ve fine-tuned GPT‑3 to more accurately answer open-ended questions using a text-based web browser.

We’ve fine-tuned GPT‑3 to more accurately answer open-ended questions using a text-based web browser. Our prototype copies how humans research answers to questions online—it submits search queries, follows links, and scrolls up and down web pages. It is trained to cite its sources, which makes it easier to give feedback to improve factual accuracy. We’re excited about developing more truthful AI,¹ but challenges remain, such as coping with unfamiliar types of questions.

Language models like GPT‑3 are useful for many different tasks, but have a tendency to “hallucinate” information when performing tasks requiring obscure real-world knowledge.^{2, 3} To address this, we taught GPT‑3 to use a text-based web-browser. The model is provided with an open-ended question and a summary of the browser state, and must issue commands such as “Search ...”, “Find in page: ...” or “Quote: …”. In this way, the model collects passages from web pages, and then uses these to compose an answer.

The model is fine-tuned from GPT‑3 using the⁠ same⁠ general⁠ methods⁠ we’ve used previously. We begin by training the model to copy human demonstrations, which gives it the ability to use the text-based browser to answer questions. Then we improve the helpfulness and accuracy of the model’s answers, by training a reward model to predict human preferences, and optimizing against it using either reinforcement learning or rejection sampling.

ELI5 results

TruthfulQA results

Our models outperform GPT‑3 on TruthfulQA and exhibit more favourable scaling properties. However, our models lag behind human performance, partly because they sometimes quote from unreliable sources (as shown in the question about ghosts above⁠). We hope to reduce the frequency of these failures using techniques like adversarial training.

Evaluating factual accuracy

Risks of deployment and training

Conclusion

• GPT
• Language