Pre-Train BERT with Hugging Face Transformers and Habana Gaudi

In this Tutorial, you will learn how to pre-train BERT-base from scratch using a Habana Gaudi-based DL1 instance on AWS to take advantage of the cost-performance benefits of Gaudi. We will use the Hugging Face Transformers, Optimum Habana and Datasets libraries to pre-train a BERT-base model using masked-language modeling, one of the two original BERT pre-training tasks. Before we get started, we need to set up the deep learning environment.

View Code

You will learn how to:

1. Prepare the dataset
2. Train a Tokenizer
3. Preprocess the dataset
4. Pre-train BERT on Habana Gaudi

Note: Steps 1 to 3 can/should be run on a different instance size since those are CPU intensive tasks.

Requirements

Before we start, make sure you have met the following requirements

• AWS Account with quota for DL1 instance type
• AWS CLI installed
• AWS IAM user configured in CLI with permission to create and manage ec2 instances

Helpful Resources

• Setup Deep Learning environment for Hugging Face Transformers with Habana Gaudi on AWS
• Deep Learning setup made easy with EC2 Remote Runner and Habana Gaudi
• Optimum Habana Documentation
• Pre-training script
• Code: pre-training-bert.ipynb

What is BERT?

BERT, short for Bidirectional Encoder Representations from Transformers, is a Machine Learning (ML) model for natural language processing. It was developed in 2018 by researchers at Google AI Language and serves as a swiss army knife solution to 11+ of the most common language tasks, such as sentiment analysis and named entity recognition.

What is a Masked Language Modeling (MLM)?

MLM enables/enforces bidirectional learning from text by masking (hiding) a word in a sentence and forcing BERT to bidirectionally use the words on either side of the covered word to predict the masked word.

Masked Language Modeling Example:

“Dang! I’m out fishing and a huge trout just [MASK] my line!”

Let's get started. 🚀

Note: Steps 1 to 3 were run on a AWS c6i.12xlarge instance.