Optimize and deploy with Optimum-Intel and OpenVINO GenAI

Deploying Transformers models at the edge or client-side requires careful consideration of performance and compatibility. Python, though powerful, is not always ideal for such deployments, especially in environments dominated by C++. This blog will guide you through optimizing and deploying Hugging Face Transformers models using Optimum-Intel and OpenVINO™ GenAI, ensuring efficient AI inference with minimal dependencies.

Table of Contents

1. Why Use OpenVINO™ for Edge Deployment
2. Step 1: Setting Up the Environment
3. Step 2: Exporting Models to OpenVINO IR
4. Step 3: Model Optimization
5. Step 4: Deploying with OpenVINO GenAI API
6. Conclusion

Why Use OpenVINO™ for Edge Deployment

OpenVINO™ was originally developed as a C++ AI inference solution, making it ideal for edge and client deployment where minimizing dependencies is crucial. With the introduction of the GenAI API, integrating large language models (LLMs) into C++ or Python applications has become even more straightforward, with features designed to simplify deployment and enhance performance.

Step 1: Setting Up the Environment

Pre-requisites

To start, ensure your environment is properly configured with both Python and C++. Install the necessary Python packages:

pip install --upgrade --upgrade-strategy eager "optimum[openvino]"

Here are the specific packages used in this blog post:

transformers==4.44
openvino==24.3
openvino-tokenizers==24.3
optimum-intel==1.20
lm-eval==0.4.3

For GenAI C++ libraries installation follow the instruction here.

Step 2: Exporting Models to OpenVINO IR

Hugging Face and Intel's collaboration has led to the Optimum-Intel project. It is designed to optimize Transformers models for inference on Intel HW. Optimum-Intel supports OpenVINO as an inference backend and its API has wrappers for various model architectures built on top of OpenVINO inference API. All of these wrappers start from OV prefix, for example, OVModelForCausalLM. Otherwise, it is similar to the API of 🤗 Transformers library.