Introducing the Open FinLLM Leaderboard

Finding the best LLM models for finance use cases

The growing complexity of financial language models (LLMs) necessitates evaluations that go beyond general NLP benchmarks. While traditional leaderboards focus on broader NLP tasks like translation or summarization, they often fall short in addressing the specific needs of the finance industry. Financial tasks, such as predicting stock movements, assessing credit risks, and extracting information from financial reports, present unique challenges that require models with specialized skills. This is why we decided to create the Open FinLLM Leaderboard.

The leaderboard provides a specialized evaluation framework tailored specifically to the financial sector. We hope it fills this critical gap, by providing a transparent framework that assesses model readiness for real-world use with a one-stop solution. The leaderboard is designed to highlight a model's financial skill by focusing on tasks that matter most to finance professionals—such as information extraction from financial documents, market sentiment analysis, and forecasting financial trends.

Key Features of the Open Financial LLM Leaderboard

• Diverse Task Categories: The leaderboard covers tasks across seven categories: Information Extraction (IE), Textual Analysis (TA), Question Answering (QA), Text Generation (TG), Risk Management (RM), Forecasting (FO), and Decision-Making (DM).
• Evaluation Metrics: Models are assessed using a variety of metrics, including Accuracy, F1 Score, ROUGE Score, and Matthews Correlation Coefficient (MCC). These metrics provide a multidimensional view of model performance, helping users identify the strengths and weaknesses of each model.

Supported Tasks and Metric

Categories

Metrics

Individual Tasks

We use 40 tasks on this leaderboard, across these categories:

Click here for a short explanation of each task Click here for a detailed explanation of each task