Introducing the Hugging Face Embedding Container for Amazon SageMaker

We are excited to announce that the new Hugging Face Embedding Container for Amazon SageMaker is now generally available (GA). AWS customers can now efficiently deploy embedding models on SageMaker to build Generative AI applications, including Retrieval-Augmented Generation (RAG) applications.

In this Blog we will show you how to deploy open Embedding Models, like Snowflake/snowflake-arctic-embed-l, BAAI/bge-large-en-v1.5 or sentence-transformers/all-MiniLM-L6-v2 to Amazon SageMaker for inference using the new Hugging Face Embedding Container. We will deploy the Snowflake/snowflake-arctic-embed-m-v1.5 one of the best open Embedding Models for retrieval - you can check its rankings on the MTEB Leaderboard.

The example covers:

• 1. Setup development environment
• 2. Retrieve the new Hugging Face Embedding Container
• 3. Deploy Snowflake Arctic to Amazon SageMaker
• 4. Run and evaluate Inference performance
• 5. Delete model and endpoint

What is the Hugging Face Embedding Container?

The Hugging Face Embedding Container is a new purpose-built Inference Container to easily deploy Embedding Models in a secure and managed environment. The DLC is powered by Text Embedding Inference (TEI) a blazing fast and memory efficient solution for deploying and serving Embedding Models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5. TEI implements many features such as:

• No model graph compilation step
• Small docker images and fast boot times
• Token based dynamic batching
• Optimized transformers code for inference using Flash Attention, Candle and cuBLASLt
• Safetensors weight loading
• Production ready (distributed tracing with Open Telemetry, Prometheus metrics)

TEI supports the following model architectures

Lets get started!

1. Setup development environment

We are going to use the sagemaker python SDK to deploy Snowflake Arctic to Amazon SageMaker. We need to make sure to have an AWS account configured and the sagemaker python SDK installed.