SyncAI.news, a Varaisys broadcasting
QuanReview: Offline, Auditable Reconciliation of Human and LLM Span Annotations
MM

Matteo Musacchio, Juan Cruz Giner Pulero, Isabel Casta\~neda, Naomi Couriel, Yelena Mejova, Mariano G. Beir\'o, Kyriaki Kalimeri

· 1 min read

ResearcharXiv cs.CL

QuanReview: Offline, Auditable Reconciliation of Human and LLM Span Annotations

arXiv:2609.35685v1 Announce Type: new Abstract: Structured span annotations, such as quantities with their units, uncertainty modifiers, and event classes, are expensive to create and hard to keep trustworthy once language models enter the loop. We present QuanReview, an open-source system for auditing and correcting such annotation layers. QuanReview aligns two annotation streams over the same documents at character level, resolves unambiguous cases by an explicit and logged policy, and routes candidate conflicts to a browser-based adjudication interface where reviewers accept either side, build field-level hybrids, or flag items for re-annotation. A campaign manager assigns documents to multiple annotators with configurable redundancy, computes agreement at document and span level, auto-merges unanimous documents, and exports the corrected layer in the original file format, so that it can replace the original annotation files directly. Applied to a 4,457-record humanitarian benchmark and an LLM extraction stream, the system fully auto-merged 8% of documents, applied automatic policy decisions to a further 1,513 records, and concentrated human attention on 3,131 candidate conflicts, a mean of 5.4 per reviewed document.

Original source

This story was published by arXiv cs.CL and written by Matteo Musacchio, Juan Cruz Giner Pulero, Isabel Casta\~neda, Naomi Couriel, Yelena Mejova, Mariano G. Beir\'o, Kyriaki Kalimeri. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News