论文标题
罗奇伯特：为中国的伯特微调努力
RoChBert: Towards Robust BERT Fine-tuning for Chinese
论文作者
论文摘要
尽管在各种任务上表现出色，但已证明预训练的语言模型（例如BERT）被证明受到对抗文本的影响。在本文中，我们提出了Rochbert，这是一个框架，该框架通过利用更全面的对抗图将中国语音和Glyph功能融合到微调过程中，以融合中国语音和Glyph特征，以构建更强大的BERT模型。受课程学习的启发，我们进一步建议通过与中间样品结合使用对抗性文本来增强培训数据集。广泛的实验表明，罗奇伯特以重要的方式优于先前的方法：（i）强大的 - 罗奇伯特大大提高了模型的鲁棒性，而无需牺牲良性文本的准确性。具体而言，国防部将无限和有限攻击的成功率降低了59.43％和39.33％，而准确度的准确性为93.30％； （ii）灵活 - 罗奇伯特（Rochbert）可以轻松地扩展到各种语言模型，以解决具有出色表现的不同下游任务； （iii）有效 - 无需从头开始训练的语言模型而无需直接将Rochbert应用于微调阶段，而建议的数据增强方法也很低。
Despite of the superb performance on a wide range of tasks, pre-trained language models (e.g., BERT) have been proved vulnerable to adversarial texts. In this paper, we present RoChBERT, a framework to build more Robust BERT-based models by utilizing a more comprehensive adversarial graph to fuse Chinese phonetic and glyph features into pre-trained representations during fine-tuning. Inspired by curriculum learning, we further propose to augment the training dataset with adversarial texts in combination with intermediate samples. Extensive experiments demonstrate that RoChBERT outperforms previous methods in significant ways: (i) robust -- RoChBERT greatly improves the model robustness without sacrificing accuracy on benign texts. Specifically, the defense lowers the success rates of unlimited and limited attacks by 59.43% and 39.33% respectively, while remaining accuracy of 93.30%; (ii) flexible -- RoChBERT can easily extend to various language models to solve different downstream tasks with excellent performance; and (iii) efficient -- RoChBERT can be directly applied to the fine-tuning stage without pre-training language model from scratch, and the proposed data augmentation method is also low-cost.
