CDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising

arXiv:2607.28236v2 Announce Type: replace-cross Abstract: Pre-trained language models have significantly improved sentence representation learning, yet their embeddings remain sensitive to textual perturbations such as synonym substitution, masking, and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embeddings by jointly optimizing contrastive and reconstruction objectives to learn perturbation-invariant representations. We evaluate the proposed framework under multiple perturbation strategies and strengths, and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbation, with the gap widening as the perturbation strength increases, highlighting perturbation-invariant learning as a promising direction for improving the stability of sentence embeddings. The source code is publicly available at: https://github.com/ComputationIASBS/CDAE