论文标题
加权有限态传感器和文本归一化语言模型的浅融合
Shallow Fusion of Weighted Finite-State Transducer and Language Model for Text Normalization
论文作者
论文摘要
生产中的文本归一化（TN）系统在很大程度上使用加权有限态传感器（WFST）基于规则。但是，当归一化形式取决于上下文时，基于粮食计划署的系统与模棱两可的输入斗争。另一方面，神经文本归一化系统可以考虑上下文，但它们遭受了无法恢复的错误，并且需要标记的标准化数据集，这些数据集很难收集。我们提出了一种新的混合方法，结合了基于规则和神经系统的好处。首先，非确定性的WFST输出所有归一化候选者，然后神经语言模型选择了最好的候选者 - 类似于自动语音识别的浅融合。尽管WFST阻止了无法恢复的错误，但语言模型可以解决上下文歧义。该方法很容易扩展，我们表明它是有效的。与现有的最新TN模型相比，它取得了可比或更好的结果。
Text normalization (TN) systems in production are largely rule-based using weighted finite-state transducers (WFST). However, WFST-based systems struggle with ambiguous input when the normalized form is context-dependent. On the other hand, neural text normalization systems can take context into account but they suffer from unrecoverable errors and require labeled normalization datasets, which are hard to collect. We propose a new hybrid approach that combines the benefits of rule-based and neural systems. First, a non-deterministic WFST outputs all normalization candidates, and then a neural language model picks the best one -- similar to shallow fusion for automatic speech recognition. While the WFST prevents unrecoverable errors, the language model resolves contextual ambiguity. The approach is easy to extend and we show it is effective. It achieves comparable or better results than existing state-of-the-art TN models.
