论文标题
一项有关测试片段如何影响开发人员的调查以及他们需要什么支持来解决它
A Survey on How Test Flakiness Affects Developers and What Support They Need To Address It
论文作者
论文摘要
非确定性通过和失败的测试用例，即所谓的片状测试，最近已成为软件工程研究的重点领域。尽管这项研究重点已经得到了行业的一些热情认可，但先前的工作主要通过采矿软件存储库以中心的方式研究了片状。但是，从软件存储库中提取哪些数据无法告诉我们，开发人员如何看待片状：开发人员日常工作中的测试片状如何普遍存在，它如何影响它们，最重要的是：他们希望我们的研究人员对此做些什么？为了回答这些问题，我们调查了335个专业软件开发人员和不同域中的测试人员。调查受访者确认片状测试是一个常见且严重的问题，因此加强了对片状测试检测的持续研究。开发人员不太担心重新运行测试引起的计算成本，以及在测试结果中失去信任的更多信息。因此，他们希望拥有IDE插件来检测片状代码以及更好的问题可视化，尤其是随着时间的推移显示测试结果的仪表板；他们还希望获得有关片状的更多培训和信息。这些重要方面将需要研究人员和工具开发人员的注意。
Non-deterministically passing and failing test cases, so-called flaky tests, have recently become a focus area of software engineering research. While this research focus has been met with some enthusiastic endorsement from industry, prior work nevertheless mostly studied flakiness using a code-centric approach by mining software repositories. What data extracted from software repositories cannot tell us, however, is how developers perceive flakiness: How prevalent is test flakiness in developers' daily routine, how does it affect them, and most importantly: What do they want us researchers to do about it? To answer these questions, we surveyed 335 professional software developers and testers in different domains. The survey respondents confirm that flaky tests are a common and serious problem, thus reinforcing ongoing research on flaky test detection. Developers are less worried about the computational costs caused by re-running tests and more about the loss of trust in the test outcomes. Therefore, they would like to have IDE plugins to detect flaky code as well as better visualizations of the problem, particularly dashboards showing test outcomes over time; they also wish for more training and information on flakiness. These important aspects will require the attention of researchers as well as tool developers.
