research-article

Asymob: a platform for measuring and clustering chatbots

Authors:

Jose María López-Morales,

Pablo C. Cañizares,

Sara Pérez-Soler,

Juan de LaraAuthors Info & Claims

ICSE '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings

Pages 16 - 20

https://doi.org/10.1145/3510454.3516843

Published: 19 October 2022 Publication History

Abstract

Chatbots have become a popular way to access all sorts of services via natural language. Many platforms and tools have been proposed for their construction, like Google's Dialogflow, Amazon's Lex or Rasa. However, most of them still miss integrated quality assurance methods like metrics. Moreover, there is currently a lack of mechanisms to compare and classify chatbots possibly developed with heterogeneous technologies.

To tackle these issues, we present Asymob, a web platform that enables the measurement of chatbots using a suite of 20 metrics. The tool features a repository supporting chatbots built with different technologies, like Dialogflow and Rasa. Asymob's metrics help in detecting quality issues and serve to compare chatbots across and within technologies. The tool also helps in classifying chatbots along conversation topics or design features by means of two clustering methods: based on the chatbot metrics or on the phrases expected and produced by the chatbot. A video showcasing the tool is available at https://www.youtube.com/watch?v=8lpETkILpv8.

References

[1]

Ebtesam Hussain Almansor and Farookh Khadeer Hussain. 2021. Fuzzy prediction model to measure chatbot quality of service. In FUZZ-IEEE. IEEE, 1--4.

[2]

Francesco Basciani, Juri Di Rocco, Davide Di Ruscio, Ludovico Iovino, and Alfonso Pierantonio. 2016. Automated clustering of metamodel repositories. In CAiSE (LNCS), Vol. 9694. 342--358.

[3]

Botium. [n. d.]. https://www.botium.ai/. last access in 2021.

[4]

Josip Bozic and Franz Wotawa. 2019. Testing chatbots using metamorphic relations. In ICTSS (LNCS), Vol. 11812. Springer, 41--55.

[5]

Sergio Bravo-Santos, Esther Guerra, and Juan de Lara. 2020. Testing chatbots with Charm. In QUATIC (CCIS), Vol. 1266. Springer, 426--438.

[6]

Pablo C. Cañizares, Sara Pérez-Soler, Esther Guerra, and Juan de Lara. 2022. Automating the measurement of heterogeneous chatbot designs. In SAC. ACM, 1--8.

[7]

Márcio Braga dos Santos, Ana Paula Carvalho Cavalcanti Furtado, Sidney C. Nogueira, and Diogo Dantas Moreira. 2020. OggyBug: A test automation tool in chatbots. In SAST. ACM, 79--87.

[8]

Mingkun Gao, Xiaotong Liu, Anbang Xu, and Rama Akkiraju. 2021. Chatbot or Chat-Blocker: Predicting chatbot popularity before deployment. In DIS. ACM, 1458--1469.

[9]

Jiepu Jiang and Naman Ahuja. 2020. Response quality in human-chatbot collaborative systems. In SIGIR. ACM, 1545--1548.

[10]

Michael McTear, Zoraida Callejas, and David Griol. 2016. The Conversational Interface. Talking to Smart Devices. Springer.

[11]

Robert J. Moore and Raphael Arar. 2019. Conversational UX Design: A Practitioner's Guide to the Natural Conversation Framework. ACM, New York, NY, USA.

[12]

Sara Pérez-Soler, Esther Guerra, and Juan de Lara. 2020. Model-driven chatbot development. In ER (LNCS), Vol. 12400. Springer, 207--222.

[13]

Sara Pérez-Soler, Esther Guerra, and Juan de Lara. 2021. Creating and migrating chatbots with Conga. In ICSE Companion. IEEE, 37--40.

[14]

Sara Pérez-Soler, Sandra Juarez-Puerta, Esther Guerra, and Juan de Lara. 2021. Choosing a chatbot development tool. IEEE Softw. 38, 4 (2021), 94--103.

Digital Library

[15]

Ranci Ren, John W. Castro, Silvia Teresita Acuña, and Juan de Lara. 2019. Evaluation techniques for chatbot usability: A systematic mapping study. Int. J. Softw. Eng. Knowl. Eng. 29, 11&12 (2019), 1673--1702.

[16]

Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20 (1987), 53--65.

Digital Library

[17]

Elayne Ruane, Théo Faure, Ross Smith, Dan Bean, Julie Carson-Berndsen, and Anthony Ventresque. 2018. BoTest: A framework to test the quality of conversational agents using divergent input examples. In IUI Companion. ACM, 64:1--64:2.

[18]

Bayan Abu Shawar and Eric Atwell. 2007. Different measurements metrics to evaluate a chatbot system. In NAACL-HLT-Dialog. ACM, 89--96.

[19]

Amir Shevat. 2017. Designing bots: Creating conversational experiences. O'Reilly.

[20]

Marisa Vasconcelos, Heloisa Candello, Claudio S. Pinhanez, and Thiago dos Santos. 2017. Bottester: Testing conversational systems with simulated users. In IHC. ACM, 73:1--73:4.

Cited By

Cañizares PLópez-Morales JPérez-Soler SGuerra Ede Lara J(2024)Measuring and Clustering Heterogeneous Chatbot DesignsACM Transactions on Software Engineering and Methodology10.1145/363722833:4(1-43)Online publication date: 17-Apr-2024
https://dl.acm.org/doi/10.1145/3637228
Liang YMa SLin C(2024)Chatbotification for Web Information Systems: A Pattern-Based Approach2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00368(2290-2295)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00368

Index Terms

Asymob: a platform for measuring and clustering chatbots

Recommendations

Measuring and Clustering Heterogeneous Chatbot Designs
Conversational agents, or chatbots, have become popular to access all kind of software services. They provide an intuitive natural language interface for interaction, available from a wide range of channels including social networks, web pages, ...
Automating the measurement of heterogeneous chatbot designs
SAC '22: Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing

Chatbots are being increasingly used to provide a natural language interface to all kinds of software services. However, while there are many platforms and tools for chatbot development, they typically lack support to statically measure properties of ...
Chatbot or Chat-Blocker: Predicting Chatbot Popularity before Deployment
DIS '21: Proceedings of the 2021 ACM Designing Interactive Systems Conference

Chatbots are widely employed in various scenarios. However, given the high costs of chatbot development and chatbots’ tremendous social influence, chatbot failures may inevitably lead to a huge economic loss. Previous chatbot evaluation frameworks rely ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings

May 2022

394 pages

ISBN:9781450392235

DOI:10.1145/3510454

General Chair:
Matthew B Dwyer
University of Virginia

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

R&D programme of Madrid
Spanish Ministry of Science

Conference

ICSE '22

Sponsor:

SIGSOFT

ICSE '22: 44th International Conference on Software Engineering

May 21 - 29, 2022

Pennsylvania, Pittsburgh

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
70
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cañizares PLópez-Morales JPérez-Soler SGuerra Ede Lara J(2024)Measuring and Clustering Heterogeneous Chatbot DesignsACM Transactions on Software Engineering and Methodology10.1145/363722833:4(1-43)Online publication date: 17-Apr-2024
https://dl.acm.org/doi/10.1145/3637228
Liang YMa SLin C(2024)Chatbotification for Web Information Systems: A Pattern-Based Approach2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00368(2290-2295)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00368

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents