SyncAI.news, a Varaisys broadcasting
Benchmarking Large Language Models for Linguistic Annotation
OT

Ond\v{r}ej Tich\'y (Faculty of Arts, Charles University), Anna Marklov\'a (Faculty of Arts, Charles University), Barbora Bulantov\'a (Faculty of Arts, Charles University), Magdalena Titlbachov\'a (Faculty of Arts, Charles University), Ji\v{r}\'i Mili\v{c}ka (Faculty of Arts, Charles University)

· 1 min read

ResearcharXiv cs.CL

Benchmarking Large Language Models for Linguistic Annotation

arXiv:2611.21291v1 Announce Type: new Abstract: Assessing large language models for linguistic annotation requires task-specific evaluation under documented conditions. We present dhAIBench, a Python benchmarking and annotation tool with a web interface for exploring and sharing results. It records prompts, configurations and predictions, enabling comparisons of annotation quality, confidence calibration, repeatability and estimated cost. We demonstrate the framework and the associated methodology through 293 runs of 48 models on nine linguistic annotation tasks spanning morphology, syntax, semantics, discourse-pragmatics, spelling normalization and learner-language correction. Models achieve high agreement with reference annotations on several tasks, including those on non-standardized, historical varieties as well as requiring contextual interpretation. The results support evaluating models against specific linguistic tasks, rather than relying on general rankings. To this end, we make the current and future benchmarks publicly available at dhaibench.silent3.ff.cuni.cz.

Original source

This story was published by arXiv cs.CL and written by Ond\v{r}ej Tich\'y (Faculty of Arts, Charles University), Anna Marklov\'a (Faculty of Arts, Charles University), Barbora Bulantov\'a (Faculty of Arts, Charles University), Magdalena Titlbachov\'a (Faculty of Arts, Charles University), Ji\v{r}\'i Mili\v{c}ka (Faculty of Arts, Charles University). SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News