Introducing the Hugging Face LLM Inference Container for Amazon SageMaker

This is an example on how to deploy the open-source LLMs, like BLOOM to Amazon SageMaker for inference using the new Hugging Face LLM Inference Container. We will deploy the 12B Pythia Open Assistant Model, an open-source Chat LLM trained with the Open Assistant dataset.

The example covers:

1. Setup development environment
2. Retrieve the new Hugging Face LLM DLC
3. Deploy Open Assistant 12B to Amazon SageMaker
4. Run inference and chat with our model
5. Create Gradio Chatbot backed by Amazon SageMaker

You can find the code for the example also in the notebooks repository.

What is Hugging Face LLM Inference DLC?

Hugging Face LLM DLC is a new purpose-built Inference Container to easily deploy LLMs in a secure and managed environment. The DLC is powered by Text Generation Inference (TGI), an open-source, purpose-built solution for deploying and serving Large Language Models (LLMs). TGI enables high-performance text generation using Tensor Parallelism and dynamic batching for the most popular open-source LLMs, including StarCoder, BLOOM, GPT-NeoX, Llama, and T5. Text Generation Inference is already used by customers such as IBM, Grammarly, and the Open-Assistant initiative implements optimization for all supported model architectures, including:

• Tensor Parallelism and custom cuda kernels
• Optimized transformers code for inference using flash-attention on the most popular architectures
• Quantization with bitsandbytes
• Continuous batching of incoming requests for increased total throughput
• Accelerated weight loading (start-up time) with safetensors
• Logits warpers (temperature scaling, topk, repetition penalty ...)
• Watermarking with A Watermark for Large Language Models
• Stop sequences, Log probabilities
• Token streaming using Server-Sent Events (SSE)

Officially supported model architectures are currently:

Let's get started!

1. Setup development environment

!pip install "sagemaker==2.175.0" --upgrade --quiet