SyncAI.news, a Varaisys broadcasting
MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark
QZ

Qingyun Zou, Jiahao Cui, Nuo Chen, Bingsheng He, Weng-Fai Wong

· 1 min read

ResearcharXiv cs.AI

MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark

arXiv:2601.03708v3 Announce Type: replace-cross Abstract: Large language models (LLMs) have achieved strong performance on code completion in general-purpose programming languages, but existing repository-level benchmarks focus almost exclusively on software code and largely overlook hardware description languages (HDLs). We present \textbf{MHRC-Bench}, consisting of \textbf{MHRC-Bench-Train} and \textbf{MHRC-Bench-Eval}, the first benchmark for multilingual hardware code completion at the repository level with functional evaluation. MHRC-Bench covers three major hardware design coding styles and annotates each completion target with code-structure-level and hardware-oriented semantic labels derived from concrete syntax tree (CST) analysis. Post-training billion-scale code LLMs on MHRC-Bench-Train enables smaller open-weight models to outperform much larger general-purpose models under matched context windows. Beyond aggregate accuracy, our analysis surfaces four findings that distinguish hardware from software code completion: \emph{(i)}~performance varies sharply across HDLs, with high-level synthesis (HLS) easiest and VHDL hardest in most settings across functional pass rate, exact match (EM), and edit similarity (ES); \emph{(ii)}~hardware accuracy is non-monotonic in code-structure depth, in contrast with the monotonic depth--accuracy curve known for software; \emph{(iii)}~accuracy plateaus once the input context exceeds 2{,}048 tokens, while software completion continues to benefit from longer context; and \emph{(iv)}~certain repository-level retrievers tuned on software underperform a no-retrieval baseline on HDL repositories because their structural and dependency assumptions are violated by hardware code.

Original source

This story was published by arXiv cs.AI and written by Qingyun Zou, Jiahao Cui, Nuo Chen, Bingsheng He, Weng-Fai Wong. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News