Bringing the Artificial Analysis LLM Performance Leaderboard to Hugging Face

Building applications with LLMs requires considering more than just quality: for many use-cases, speed and price are equally or more important.

For consumer applications and chat experiences, speed and responsiveness are critical to user engagement. Users expect near-instant responses, and delays can directly lead to reduced engagement. When building more complex applications involving tool use or agentic systems, speed and cost become even more important, and can become the limiting factor on overall system capability. The time taken by sequential requests to LLMs can quickly stack up for each user request adding to the cost.

This is why Artificial Analysis (@ArtificialAnlys) developed a leaderboard evaluating price, speed and quality across >100 serverless LLM API endpoints, now coming to Hugging Face.

Find the leaderboard here!

The LLM Performance Leaderboard

The LLM Performance Leaderboard aims to provide comprehensive metrics to help AI engineers make decisions on which LLMs (both open & proprietary) and API providers to use in AI-enabled applications.

When making decisions regarding which AI technologies to use, engineers need to consider quality, price and speed (latency & throughput). The LLM Performance Leaderboard brings all three together to enable decision making in one place across both proprietary & open models.

Source: LLM Performance Leaderboard

Metric coverage

The metrics reported are:

For further definitions, see our full methodology page.

Test Workloads

The leaderboard allows exploration of performance under several different workloads (6 combinations in total):

• varying the prompt length: ~100 tokens, ~1k tokens, ~10k tokens.
• running parallel queries: 1 query, 10 parallel queries.

Methodology

We test every API endpoint on the leaderboard 8 times per day, and leaderboard figures represent the median measurement of the last 14 days. We also have percentile breakdowns within the collapsed tabs.

For further definitions, see our full methodology page.