Why we’re switching to Hugging Face Inference Endpoints, and maybe you should too

Hugging Face recently launched Inference Endpoints; which as they put it: solves transformers in production. Inference Endpoints is a managed service that allows you to:

• Deploy (almost) any model on Hugging Face Hub
• To any cloud (AWS, and Azure, GCP on the way)
• On a range of instance types (including GPU)
• We’re switching some of our Machine Learning (ML) models that do inference on a CPU to this new service. This blog is about why, and why you might also want to consider it.

What were we doing?

The models that we have switched over to Inference Endpoints were previously managed internally and were running on AWS Elastic Container Service (ECS) backed by AWS Fargate. This gives you a serverless cluster which can run container based tasks. Our process was as follows:

• Train model on a GPU instance (provisioned by CML, trained with transformers)
• Upload to Hugging Face Hub
• Build API to serve model (FastAPI)
• Wrap API in container (Docker)
• Upload container to AWS Elastic Container Repository (ECR)
• Deploy model to ECS Cluster

Now, you can reasonably argue that ECS was not the best approach to serving ML models, but it served us up until now, and also allowed ML models to sit alongside other container based services, so it reduced cognitive load.

What do we do now?

With Inference Endpoints, our flow looks like this:

• Train model on a GPU instance (provisioned by CML, trained with transformers)
• Upload to Hugging Face Hub
• Deploy using Hugging Face Inference Endpoints.

So this is significantly easier. We could also use another managed service such as SageMaker, Seldon, or Bento ML, etc., but since we are already uploading our model to Hugging Face hub to act as a model registry, and we’re pretty invested in Hugging Face’s other tools (like transformers, and AutoTrain) using Inference Endpoints makes a lot of sense for us.

What about Latency and Stability?

Before switching to Inference Endpoints we tested different CPU endpoints types using ab.