Running Privacy-Preserving Inferences on Hugging Face Endpoints

This is a guest blog post by the Zama team. Zama is an open source cryptography company building state-of-the-art FHE solutions for blockchain and AI.

Eighteen months ago, Zama started Concrete ML, a privacy-preserving ML framework with bindings to traditional ML frameworks such as scikit-learn, ONNX, PyTorch, and TensorFlow. To ensure privacy for users' data, Zama uses Fully Homomorphic Encryption (FHE), a cryptographic tool that allows to make direct computations over encrypted data, without ever knowing the private key.

From the start, we wanted to pre-compile some FHE-friendly networks and make them available somewhere on the internet, allowing users to use them trivially. We are ready today! And not in a random place on the internet, but directly on Hugging Face.

More precisely, we use Hugging Face Endpoints and custom inference handlers, to be able to store our Concrete ML models and let users deploy on HF machines in one click. At the end of this blog post, you will understand how to use pre-compiled models and how to prepare yours. This blog can also be considered as another tutorial for custom inference handlers.

Deploying a pre-compiled model

Let's start with deploying an FHE-friendly model (prepared by Zama or third parties - see Preparing your pre-compiled model section below for learning how to prepare yours).

First, look for the model you want to deploy: We have pre-compiled a bunch of models on Zama's HF page (or you can find them with tags). Let's suppose you have chosen concrete-ml-encrypted-decisiontree: As explained in the description, this pre-compiled model allows you to detect spam without looking at the message content in the clear.

Like with any other model available on the Hugging Face platform, select Deploy and then Inference Endpoint (dedicated):

Inference Endpoint (dedicated)

Create Endpoint

After a few seconds, the Endpoint is deployed, and your privacy-preserving model is ready to operate.

Endpoint is created

Using the Endpoint

Clone Repository