Faster TensorFlow models in Hugging Face Transformers

In the last few months, the Hugging Face team has been working hard on improving Transformers’ TensorFlow models to make them more robust and faster. The recent improvements are mainly focused on two aspects:

1. Computational performance: BERT, RoBERTa, ELECTRA and MPNet have been improved in order to have a much faster computation time. This gain of computational performance is noticeable for all the computational aspects: graph/eager mode, TF Serving and for CPU/GPU/TPU devices.
2. TensorFlow Serving: each of these TensorFlow model can be deployed with TensorFlow Serving to benefit of this gain of computational performance for inference.

Computational Performance

To demonstrate the computational performance improvements, we have done a thorough benchmark where we compare BERT's performance with TensorFlow Serving of v4.2.0 to the official implementation from Google. The benchmark has been run on a GPU V100 using a sequence length of 128 (times are in millisecond):

Batch size	Google implementation	v4.2.0 implementation	Relative difference Google/v4.2.0 implem
1	6.7	6.26	6.79%
2	9.4	8.68	7.96%
4	14.4	13.1	9.45%
8	24	21.5	10.99%
16	46.6	42.3	9.67%
32	83.9	80.4	4.26%
64	171.5	156	9.47%
128	338.5	309	9.11%

The current implementation of Bert in v4.2.0 is faster than the Google implementation by up to ~10%. Apart from that it is also twice as fast as the implementations in the 4.1.1 release.

TensorFlow Serving

The previous section demonstrates that the brand new Bert model got a dramatic increase in computational performance in the last version of Transformers. In this section, we will show you step-by-step how to deploy a Bert model with TensorFlow Serving to benefit from the increase in computational performance in a production environment.

What is TensorFlow Serving?

What is a SavedModel?

How to install TensorFlow Serving?

There are three ways to install and use TensorFlow Serving:

• through a Docker container,
• through an apt package,
• or using pip.