Accelerate a World of LLMs on Hugging Face with NVIDIA NIM

AI builders want a choice of the latest large language models (LLM) architectures and specialized variants for use in AI agents and other apps, but handling all the diversity can slow testing and deployment pipelines. In particular, managing and optimizing different inference software frameworks to achieve best performance across varied LLMs and serving requirements is a time-consuming bottleneck to getting performant AI apps in the hands of end-users.

NVIDIA AI customers and ecosystem partners leverage NVIDIA NIM inference microservices to streamline deployment of the latest AI models on NVIDIA accelerated infrastructure, including LLMs, multi-modal and domain-specific models from NVIDIA, Meta, Mistral AI, Google and hundreds more innovative model builders. We’ve seen customers and partners deliver more innovation, faster, with a simplified, reliable approach to model deployment, and today we’re excited to unlock over 100,000 LLMs on Hugging Face for rapid, reliable deployment with NIM.

A Single NIM Microservice for Deploying a Broad Range of LLMs

NIM now provides a single docker container for deploying a broad range of LLMs supported by leading inference frameworks from NVIDIA and the community including NVIDIA TensorRT-LLM, vLLM and SGLang. When an LLM is provided to the NIM container, it performs several steps for deployment and performance optimization, without manual configuration:

Table 1. NVIDIA NIM LLM adaptation phases and functionality

The single NIM container supports common LLM weight formats, including:

Getting Started

To use NIM, ensure your environment has NVIDIA GPUs with appropriate drivers (CUDA 12.1+), Docker installed, an NVIDIA NGC Account and API Key for NIM Docker images, and a Hugging Face account and API token for models requiring authentication. Learn more about environment prerequisites in the NIM documentation.

For ease of use, let’s store some of the frequently used information in environment variables.