论文标题
罕见的事件故障测试案例生成在学习的控制器中
Rare event failure test case generation in Learning-Enabled-Controllers
论文作者
论文摘要
机器学习模型在许多实际问题中都有普遍的应用，这增加了这些训练有素模型的行为中正确性的重要性。找到一个可以揭示这些训练有素的系统潜在失败的好测试用例可以帮助重新训练这些模型以提高其正确性。对于训练有素的模型，失败的发生是罕见的。因此，由于较大的搜索空间，有限的计算资源和可用时间，通过评估输入搜索空间中的每个样本或随机搜索中的每个样本来搜索这些罕见情况。在本文中，我们试图解决与传统随机搜索更快地发现这些故障情况的挑战。我们方法的核心思想是将基于训练数据的观察结果，从现实世界统计数据中绘制的数据以及来自域专家的知识的高失败概率和低/最小故障概率区域的输入数据空间分开。使用这些信息，我们可以设计一个生成模型，从中我们可以生成具有很可能揭示潜在失败的场景。我们在两种不同的实验场景上评估了这种方法，并能够比传统的随机搜索快速发现此类故障一千倍。
Machine learning models have prevalent applications in many real-world problems, which increases the importance of correctness in the behaviour of these trained models. Finding a good test case that can reveal the potential failure in these trained systems can help to retrain these models to increase their correctness. For a well-trained model, the occurrence of a failure is rare. Consequently, searching these rare scenarios by evaluating each sample in input search space or randomized search would be costly and sometimes intractable due to large search space, limited computational resources, and available time. In this paper, we tried to address this challenge of finding these failure scenarios faster than traditional randomized search. The central idea of our approach is to separate the input data space in region of high failure probability and region of low/minimal failure probability based on the observation made by training data, data drawn from real-world statistics, and knowledge from a domain expert. Using these information, we can design a generative model from which we can generate scenarios that have a high likelihood to reveal the potential failure. We evaluated this approach on two different experimental scenarios and able to speed up the discovery of such failures a thousand-fold faster than the traditional randomized search.
