SyncAI.news, a Varaisys broadcasting
Augmenting Pretrained Language Models with N-gram Embeddings through Incremental Training
SH

Shaohua Hu, Xusheng Gu, Haofei Yang, Minghao Shao, Nanhui Zhao, Caoyun Fan, Fangming Li

· 1 min read

ResearcharXiv cs.CL

Augmenting Pretrained Language Models with N-gram Embeddings through Incremental Training

arXiv:2611.20415v1 Announce Type: new Abstract: As a new scaling technique for large language models, the n-gram embedding improves the base ability of a model while adding almost no inference cost. Mainstream research realizes it through joint training of the n-gram embedding with the backbone, which severely restricts the settings in which an n-gram embedding can be used. We therefore propose training an n-gram embedding onto an existing model through incremental training, to extend where the technique can deliver value. Specifically, we explore several settings of the n-gram embedding under incremental training---the learning rate of the embedding, the vocabulary size, and the injection position---and train an n-gram embedding onto the existing model under the best configuration. Experiments show that the best configuration under incremental training does not fully coincide with that under pretraining; moreover, the incrementally trained knowledge is written predominantly into the n-gram embedding, while the general ability of the backbone is retained throughout this training process.

Original source

This story was published by arXiv cs.CL and written by Shaohua Hu, Xusheng Gu, Haofei Yang, Minghao Shao, Nanhui Zhao, Caoyun Fan, Fangming Li. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News