论文标题
部分可观测时空混沌系统的无模型预测
Predicting User Engagement Status for Online Evaluation of Intelligent Assistants
论文作者
论文摘要
Evaluation of intelligent assistants in large-scale and online settings remains an open challenge.基于用户行为的在线评估指标表明，在监视大规模的网络搜索和推荐系统方面具有很大的有效性。 Therefore, we consider predicting user engagement status as the very first and critical step to online evaluation for intelligent assistants.在这项工作中，我们首先提出了一个新颖的框架，将用户参与状态分为四类 - 实现，延续，重新制定和放弃。 We then demonstrated how to design simple but indicative metrics based on the framework to quantify user engagement levels. We also aim for automating user engagement prediction with machine learning methods. We compare various models and features for predicting engagement status using four real-world datasets. We conducted detailed analyses on features and failure cases to discuss the performance of current models as well as challenges.
Evaluation of intelligent assistants in large-scale and online settings remains an open challenge. User behavior-based online evaluation metrics have demonstrated great effectiveness for monitoring large-scale web search and recommender systems. Therefore, we consider predicting user engagement status as the very first and critical step to online evaluation for intelligent assistants. In this work, we first proposed a novel framework for classifying user engagement status into four categories -- fulfillment, continuation, reformulation and abandonment. We then demonstrated how to design simple but indicative metrics based on the framework to quantify user engagement levels. We also aim for automating user engagement prediction with machine learning methods. We compare various models and features for predicting engagement status using four real-world datasets. We conducted detailed analyses on features and failure cases to discuss the performance of current models as well as challenges.
