Sunday 3:00 PM–3:30 PM in C01

Beyond Clicks: Deep Sequential Models for Task Satisfaction Prediction with Conversational Agents

Rishabh Mehrotra

Audience level:
Intermediate

Description

Detecting and understanding implicit measures of user satisfaction are essential for meaningful experimentation aimed at predicting user satisfaction and enhancing user engagement. This talk presents insights from a deep multi-view sequential model implemented on a tera-scale user interaction data from a commercial conversational agent.

Abstract

Detecting and understanding implicit measures of user satisfaction are essential for meaningful experimentation aimed at predicting searcher satisfaction and enhancing web search quality. Search tasks have steadily emerged as accurate units to capture searcher's goals and seeking behavioral insights. However a major portion of existing work on modeling searcher satsifaction has investigated satisfaction prediction at the query level. Also most existing studies on satisfaction prediction rely on users' click activity and query reformulation behavior, however often such signals are not available for all search sessions and as a result, not useful in predicting satisfaction.

In this work, we consider the problem of prediction user satisfaction at the task level. We focus on considering holistic view of user interaction with a digital assistant (Microsoft Cortana) and construct detailed \textit{universal interaction sequences} of their activity. We go beyond query level analysis, and consider query sequences issued by the user in order to complete a task to make task level satisfaction predictions. We propose a novel deep sequential architecture to predict query level satisfaction. We enrich this model and propose a unified multi-view interaction model which combines the benefit of the query sequence model with auxiliary features which have been traditionally used for satisfaction prediction. Further, we go beyond query level satisfaction prediction and propose a number of functional composition techniques which take into account user's interactions for each query as well as the entire sequence of queries when predicting task level satisfaction. We conduct vigorous experimental analysis to demonstrate the usefulness of the proposed approach. Our findings have implications on metric development for gauging user satisfaction and on designing systems which help users with task completion.