Introducing the Enterprise Scenarios Leaderboard: a Leaderboard for Real World Use Cases

Today, the Patronus team is excited to announce the new Enterprise Scenarios Leaderboard, built using the Hugging Face Leaderboard Template in collaboration with their teams.

The leaderboard aims to evaluate the performance of language models on real-world enterprise use cases. We currently support 6 diverse tasks - FinanceBench, Legal Confidentiality, Creative Writing, Customer Support Dialogue, Toxicity, and Enterprise PII.

We measure the performance of models on metrics like accuracy, engagingness, toxicity, relevance, and Enterprise PII.

Why do we need a leaderboard for real world use cases?

We felt there was a need for an LLM leaderboard focused on real world, enterprise use cases, such as answering financial questions or interacting with customer support. Most LLM benchmarks use academic tasks and datasets, which have proven to be useful for comparing the performance of models in constrained settings. However, enterprise use cases often look very different. We have selected a set of tasks and datasets based on conversations with companies using LLMs in diverse real-world scenarios. We hope the leaderboard can be a useful starting point for users trying to understand which model to use for their practical applications.

There have also been recent concerns about people gaming leaderboards by submitting models fine-tuned on the test sets. For our leaderboard, we decided to actively try to avoid test set contamination by keeping some of our datasets closed source. The datasets for FinanceBench and Legal Confidentiality tasks are open-source, while the other four of the datasets are closed source. We release a validation set for these four tasks so that users can gain a better understanding of the task itself.

Our Tasks

Example:

Evaluation Metrics: Correctness

Evaluation Metric: Accuracy

The magical creatures of the realm fear you. Not because you're a powerful wizard or a valiant knight but because you're the veterinarian of the realm.

Evaluation metrics: Toxicity Score