Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia

notebook: sagemaker/18_inferentia_inference

The adoption of BERT and Transformers continues to grow. Transformer-based models are now not only achieving state-of-the-art performance in Natural Language Processing but also for Computer Vision, Speech, and Time-Series. 💬 🖼 🎤 ⏳

Companies are now slowly moving from the experimentation and research phase to the production phase in order to use transformer models for large-scale workloads. But by default BERT and its friends are relatively slow, big, and complex models compared to the traditional Machine Learning algorithms. Accelerating Transformers and BERT is and will become an interesting challenge to solve in the future.

AWS's take to solve this challenge was to design a custom machine learning chip designed for optimized inference workload called AWS Inferentia. AWS says that AWS Inferentia “delivers up to 80% lower cost per inference and up to 2.3X higher throughput than comparable current generation GPU-based Amazon EC2 instances.”

The real value of AWS Inferentia instances compared to GPU comes through the multiple Neuron Cores available on each device. A Neuron Core is the custom accelerator inside AWS Inferentia. Each Inferentia chip comes with 4x Neuron Cores. This enables you to either load 1 model on each core (for high throughput) or 1 model across all cores (for lower latency).

Tutorial

In this end-to-end tutorial, you will learn how to speed up BERT inference for text classification with Hugging Face Transformers, Amazon SageMaker, and AWS Inferentia.

You can find the notebook here: sagemaker/18_inferentia_inference

You will learn how to:

• 1. Convert your Hugging Face Transformer to AWS Neuron
• 2. Create a custom inference.py script for text-classification
• 3. Create and upload the neuron model and inference script to Amazon S3
• 4. Deploy a Real-time Inference Endpoint on Amazon SageMaker
• 5. Run and evaluate Inference performance of BERT on Inferentia

Let's get started! 🚀

!mkdir code