
RL
Rami Luisto, Liisa Pet\"ainen, Tommi Gr\"onholm, Jan B\"ohm, Maarit Ahtiainen, Tomi Lilja, Ilkka P\"ol\"onen, Sami \"Ayr\"am\"o
· 1 min read
ResearcharXiv cs.CL
Continued Pretraining of FinBERT on Finnish Histopathological Reports: Train-Time Signals and Proxy Downstream Correlations
arXiv:2604.14815v2 Announce Type: replace
Abstract: In Natural Language Processing (NLP) classification tasks where a lack of labeled data is an issue, continued pretraining (CPT) of transformer models on unlabeled data is an established approach. In this paper, we have two aims.
(1) We describe our observations from continued pretraining of the Finnish BERT transformer model (FinBERT) on a Finnish histopathological dataset (below, \emph{the Histopathology data}).
(2) Since the Histopathology data has no classification labels, we gather public Finnish datasets as proxy data to analyze whether the signals observed in (1) are associated with downstream classification gains.
We observe that CPT train-time loss curves differ strongly by domain, and that, in an exploratory analysis, certain CPT-derived features correlate with proxy classification improvement. In particular, this report contributes to the limited literature on NLP for Finnish healthcare data.
Original source
This story was published by arXiv cs.CL and written by Rami Luisto, Liisa Pet\"ainen, Tommi Gr\"onholm, Jan B\"ohm, Maarit Ahtiainen, Tomi Lilja, Ilkka P\"ol\"onen, Sami \"Ayr\"am\"o. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


