Deploy models on AWS Inferentia2 from Hugging Face

AWS Inferentia2 is the latest AWS machine learning chip available through the Amazon EC2 Inf2 instances on Amazon Web Services. Designed from the ground up for AI workloads, Inf2 instances offer great performance and cost/performance for production workloads.

We have been working for over a year with the product and engineering teams at AWS to make the performance and cost-efficiency of AWS Trainium and Inferentia chips available to Hugging Face users. Our open-source library optimum-neuron makes it easy to train and deploy Hugging Face models on these accelerators. You can read more about our work accelerating transformers, large language models and text-generation-inference (TGI).

Today, we are making the power of Inferentia2 directly and widely available to Hugging Face Hub users.

Enabling over 100,000 models on AWS Inferentia2 with Amazon SageMaker

A few months ago, we introduced a new way to deploy Large Language Models (LLMs) on SageMaker, with a new Inferentia/Trainium option for supported models, like Meta Llama 3. You can deploy a Llama3 model on Inferentia2 instances on SageMaker to serve inference at scale and benefit from SageMaker’s complete set of fully managed features for building and fine-tuning models, MLOps, and governance.

Today, we are expanding support for this deployment experience to over 100,000 public models available on Hugging Face, including 14 new model architectures (albert,bert,camembert,convbert,deberta,deberta-v2,distilbert,electra,roberta,mobilebert,mpnet,vit,xlm,xlm-roberta), and 6 new machine learning tasks (text-classification,text-generation,token-classification,fill-mask,question-answering,feature-extraction).

Following these simple code snippets, AWS customers will be able to easily deploy the models on Inferentia2 instances in Amazon SageMaker.

Hugging Face Inference Endpoints introduces support for AWS Inferentia2

For supported models like Llama 3, you can select 2 flavors: