Introducing the AMD 5th Gen EPYC™ CPU

AMD has just unveiled its 5th generation of server-grade EPYC CPU based on Zen5 architecture - also known as Turin. It provides a significant boost in performance, especially with a higher number of core count reaching up to 192 and 384 threads.

From Large Language Models (LLMs) to RAG scenarios, Hugging Face users can leverage this new generation of servers to enhance their performance capabilities:

1. Reduce the target latency of their deployments.
2. Increase the maximum throughput.
3. Lower the operational costs.

During the last few weeks, we have been working with AMD to validate that the Hugging Face ecosystem is fully supported on this new CPU generation and delivers the expected performance across different tasks.

Also, we have been cooking some exciting new ways to leverage torch.compile for AMD CPU through the use of AMD ZenDNN PyTorch plugin (zentorch) to speed up even more the kind of workloads we will be discussing after.

While we were able to get early access to this work to test Hugging Face models and libraries and share with you performance, we expect AMD to make it soon available to the community - stay tuned!

AMD Turin vs AMD Genoa Performance - A 2X speedup

In this section, we present the results from our benchmarking of the two AMD EPYC CPUs: Turin (128 cores) and Genoa (96 cores). For these benchmarks, we utilized the ZenDNN plug-in for PyTorch (zentorch), which provides inference optimizations tailored for deep learning workloads on AMD EPYC CPUs. This plug-in integrates seamlessly with the torch.compile graph compilation flow, enabling multiple passes of graph-level optimizations on the torch.fx graph to achieve further performance acceleration.

We ran the benchmarks using two different batch sizes—16 and 32—across five distinct use cases:

Results for Llama 3.1 8B Instruct

Conclusion

You can find more information at AMD Zen Deep Neural Network (ZenDNN)