🇵🇭 FilBench - Can LLMs Understand and Generate Filipino?

As large language models (LLMs) become increasingly integrated into our lives, it becomes crucial to assess whether they reflect the nuances and capabilities of specific language communities. For example, Filipinos are among the most active ChatGPT users globally, ranking fourth in ChatGPT traffic (behind the United States, India, and Brazil [1] [2]), but despite this strong usage, we lack a clear understanding of how LLMs perform for their languages, such as Tagalog and Cebuano. Most of the existing evidence is anecdotal, such as screenshots of ChatGPT responding in Filipino as proof that it is fluent. What we need instead is a systematic evaluation of LLM capabilities in Philippine languages.

That’s why we developed FilBench: a comprehensive evaluation suite to assess the capabilities of LLMs for Tagalog, Filipino (the standardized form of Tagalog), and Cebuano, on fluency, linguistic and translation abilities, as well as specific cultural knowledge.

We used it to evaluate 20+ state-of-the-art LLMs on FilBench, providing a comprehensive assessment of their performance in Philippine languages:

• 📄 Paper: https://arxiv.org/abs/2508.03523
• 🖥️ GitHub: https://github.com/filbench/filbench-eval

FilBench

The FilBench evaluation suite contains four major categories–Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation–divided into 12 tasks. For example, the Classical NLP category includes tasks such as sentiment analysis, whereas Generation tasks include different aspects of translation. In order to ensure that these categories reflect the priorities and trends in NLP research and usage, we curate them based on a historical survey of NLP research on Philippine languages from 2006 to early 2024. (Most of these categories exclusively contain non-translated content to ensure faithfulness to the natural use of Philippine languages.)

FilBench is now available as a set of community tasks in the official Lighteval repository!