Llama 2 on Amazon SageMaker a Benchmark

Deploying large language models (LLMs) and other generative AI models can be challenging due to their computational requirements and latency needs. To provide useful recommendations to companies looking to deploy Llama 2 on Amazon SageMaker with the Hugging Face LLM Inference Container, we created a comprehensive benchmark analyzing over 60 different deployment configurations for Llama 2.

In this benchmark, we evaluated varying sizes of Llama 2 on a range of Amazon EC2 instance types with different load levels. Our goal was to measure latency (ms per token), and throughput (tokens per second) to find the optimal deployment strategies for three common use cases:

• Most Cost-Effective Deployment: For users looking for good performance at low cost
• Best Latency Deployment: Minimizing latency for real-time services
• Best Throughput Deployment: Maximizing tokens processed per second

To keep this benchmark fair, transparent, and reproducible, we share all of the assets, code, and data we used and collected:

• GitHub Repository
• Raw Data
• Spreadsheet with processed data

We hope to enable customers to use LLMs and Llama 2 efficiently and optimally for their use case. Before we get into the benchmark and data, let's look at the technologies and methods we used.

• Llama 2 on Amazon SageMaker a Benchmark
  • What is the Hugging Face LLM Inference Container?
  • What is Llama 2?
  • What is GPTQ?
  • Benchmark
  • Recommendations & Insights
    • Most Cost-Effective Deployment
    • Best Throughput Deployment
    • Best Latency Deployment
  • Conclusions

What is the Hugging Face LLM Inference Container?

What is Llama 2?

What is GPTQ?

Benchmark

To benchmark the real-world performance of Llama 2, we tested 3 model sizes (7B, 13B, 70B parameters) on four different instance types with four different load levels, resulting in 60 different configurations:

As metrics, we used Throughput and Latency defined as:

• Throughput (tokens/sec): Number of tokens being generated per second.
• Latency (ms/token): Time it takes to generate a single token