论文标题
多通道手语的对手培训
Adversarial Training for Multi-Channel Sign Language Production
论文作者
论文摘要
符号语言是丰富的多通道语言，需要精确，复杂的方式表达手动（手）和非手动（面部和身体）特征。手语的生产（SLP）是从口语到标志语言的自动翻译，必须体现这种完整的符号形态，才能由聋人社区真正理解。先前的工作主要集中在手动特征生产上，其产出不足，其输出是由平均值的回归引起的。 在本文中，我们提出了对SLP的对抗性多通道方法。我们将基于变压器的发电机和条件歧视器之间的最小值签名为最小值游戏。我们的对手歧视者评估了以源文本为条件的标志生产的现实主义，将发生器推向了现实而清晰的输出。此外，我们将符号示意剂完全封装，其中包含非手动特征，产生面部特征和模式。 我们评估了具有挑战性的RWTH-PHOENIX-WEATER-2014T（Phoenix14t）数据集，并报告了手动生产的最先进的SLP反向翻译性能。我们为生产多通道标志的生产设定了新的基准，以支持将来的现实SLP研究。
Sign Languages are rich multi-channel languages, requiring articulation of both manual (hands) and non-manual (face and body) features in a precise, intricate manner. Sign Language Production (SLP), the automatic translation from spoken to sign languages, must embody this full sign morphology to be truly understandable by the Deaf community. Previous work has mainly focused on manual feature production, with an under-articulated output caused by regression to the mean. In this paper, we propose an Adversarial Multi-Channel approach to SLP. We frame sign production as a minimax game between a transformer-based Generator and a conditional Discriminator. Our adversarial discriminator evaluates the realism of sign production conditioned on the source text, pushing the generator towards a realistic and articulate output. Additionally, we fully encapsulate sign articulators with the inclusion of non-manual features, producing facial features and mouthing patterns. We evaluate on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset, and report state-of-the art SLP back-translation performance for manual production. We set new benchmarks for the production of multi-channel sign to underpin future research into realistic SLP.
