How to guide task-oriented chatbot users, and when: A mixed-methods study of combinations of chatbot guidance types and timings

SF Yeh, MH Wu, TY Chen, YC Lin, XJ Chang… - Proceedings of the …, 2022 - dl.acm.org
SF Yeh, MH Wu, TY Chen, YC Lin, XJ Chang, YH Chiang, YJ Chang
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2022dl.acm.org
The popularity of task-oriented chatbots is constantly growing, but smooth conversational
progress with them remains profoundly challenging. In recent years, researchers have
argued that chatbot systems should include guidance for users on how to converse with
them. Nevertheless, empirical evidence about what to place in such guidance, and when to
deliver it, has been lacking. Using a mixed-methods approach that integrates results from a
between-subjects experiment and a reflection session, this paper compares the …
The popularity of task-oriented chatbots is constantly growing, but smooth conversational progress with them remains profoundly challenging. In recent years, researchers have argued that chatbot systems should include guidance for users on how to converse with them. Nevertheless, empirical evidence about what to place in such guidance, and when to deliver it, has been lacking. Using a mixed-methods approach that integrates results from a between-subjects experiment and a reflection session, this paper compares the effectiveness of eight combinations of two guidance types (example-based and rule-based) at four guidance timings (service-onboarding, task-intro, after-failure, and upon-request), as measured by users’ task performance, improvement on subsequent tasks, and subjective experience. It establishes that each guidance type and timing has particular strengths and weaknesses, thus that each type/timing combination has a unique impact on performance metrics, learning outcomes, and user experience. On that basis, it presents guidance-design recommendations for future task-oriented chatbots.
ACM Digital Library