🚀 Accelerating LLM Inference with TGI on Intel Gaudi

We're excited to announce the native integration of Intel Gaudi hardware support directly into Text Generation Inference (TGI), our production-ready serving solution for Large Language Models (LLMs). This integration brings the power of Intel's specialized AI accelerators to our high-performance inference stack, enabling more deployment options for the open-source AI community 🎉

✨ What's New?

We've fully integrated Gaudi support into TGI's main codebase in PR #3091. Previously, we maintained a separate fork for Gaudi devices at tgi-gaudi. This was cumbersome for users and prevented us from supporting the latest TGI features at launch. Now using the new TGI multi-backend architecture, we support Gaudi directly on TGI – no more finicking on a custom repository 🙌

This integration supports Intel's full line of Gaudi hardware:

• Gaudi1 💻: Available on AWS EC2 DL1 instances
• Gaudi2 💻💻: Available on Intel Tiber AI Cloud and Denvr Dataworks
• Gaudi3 💻💻💻: Available on Intel Tiber AI Cloud, IBM Cloud and from OEM such as Dell, HP and Supermicro

You can also find more information on Gaudi hardware on Intel's Gaudi product page

🌟 Why This Matters

The Gaudi backend for TGI provides several key benefits:

• Hardware Diversity 🔄: More options for deploying LLMs in production beyond traditional GPUs
• Cost Efficiency 💰: Gaudi hardware often provides compelling price-performance for specific workloads
• Production-Ready ⚙️: All the robustness of TGI (dynamic batching, streamed responses, etc.) now available on Gaudi
• Model Support 🤖: Run popular models like Llama 3.1, Mixtral, Mistral, and more on Gaudi hardware
• Advanced Features 🔥: Support for multi-card inference (sharding), vision-language models, and FP8 precision

🚦 Getting Started with TGI on Gaudi

The easiest way to run TGI on Gaudi is to use our official Docker image. You need to run the image on a Gaudi hardware machine. Here's a basic example to get you started:

Once the server is running, you can send inference requests: