skip to main content
10.1145/3581641.3584064acmconferencesArticle/Chapter ViewAbstractPublication PagesiuiConference Proceedingsconference-collections
research-article
Open access

Addressing UX Practitioners’ Challenges in Designing ML Applications: an Interactive Machine Learning Approach

Published: 27 March 2023 Publication History
  • Get Citation Alerts
  • Abstract

    UX practitioners face novel challenges when designing user interfaces for machine learning (ML)-enabled applications. Interactive ML paradigms, like AutoML and interactive machine teaching, lower the barrier for non-expert end users to create, understand, and use ML models, but their application to UX practice is largely unstudied. We conducted a task-based design study with 27 UX practitioners where we asked them to propose a proof-of-concept design for a new ML-enabled application. During the task, our participants were given opportunities to create, test, and modify ML models as part of their workflows. Through a qualitative analysis of our post-task interview, we found that direct, interactive experimentation with ML allowed UX practitioners to tie ML capabilities and underlying data to user goals, compose affordances to enhance end-user interactions with ML, and identify ML-related ethical risks and challenges. We discuss our findings in the context of previously established human-AI guidelines. We also identify some limitations of interactive ML in UX processes and propose research-informed machine teaching as a supplement to future design tools alongside interactive ML.

    Supplementary Material

    PDF File (ux-tm-supplementary.pdf)
    Supplementary Materials

    References

    [1]
    Amazon Web Services. 2022. Amazon SageMaker Autopilot. https://aws.amazon.com/sagemaker/autopilot/.
    [2]
    Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 291–300.
    [3]
    Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. Ai Magazine 35, 4 (2014), 105–120.
    [4]
    Saleema Amershi, James Fogarty, and Daniel Weld. 2012. Regroup: Interactive Machine Learning for on-Demand Group Creation in Social Networks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for Computing Machinery, New York, NY, USA, 21–30. https://doi.org/10.1145/2207676.2207680
    [5]
    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI Interaction(CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300233
    [6]
    Apple. 2022. Machine Learning—Human Interface Guidelines. https://developer.apple.com/design/human-interface-guidelines/technologies/machine-learning/introduction.
    [7]
    Jonathan Ball. 2005. The Double Diamond: A universally accepted depiction of the design process. https://www.designcouncil.org.uk/news-opinion/double-diamond-universally-accepted-depiction-design-process.
    [8]
    Michel Beaudouin-Lafon and Wendy E Mackay. 2009. Prototyping tools and techniques. In Human-Computer Interaction. CRC Press, 137–160.
    [9]
    Jesse Josua Benjamin, Arne Berger, Nick Merrill, and James Pierce. 2021. Machine Learning Uncertainty as a Design Material: A Post-Phenomenological Inquiry. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 171, 14 pages. https://doi.org/10.1145/3411764.3445481
    [10]
    Hugh Beyer and Karen Holtzblatt. 1999. Contextual design. interactions 6, 1 (1999), 32–42.
    [11]
    Patricia Britten, Kristin Marcoe, Sedigheh Yamini, and Carole Davis. 2006. Development of food intake patterns for the MyPyramid Food Guidance System. Journal of nutrition education and behavior 38, 6 (2006), S78–S92.
    [12]
    Samuel Budd, Emma C Robinson, and Bernhard Kainz. 2021. A survey on active learning and human-in-the-loop deep learning for medical image analysis. Medical Image Analysis 71 (2021), 102062.
    [13]
    Raluca Budiu. 2017. You Are Not the User: The False-Consensus Effect. https://www.nngroup.com/articles/false-consensus/.
    [14]
    Michelle Carney, Barron Webster, Irene Alvarado, Kyle Phillips, Noura Howell, Jordan Griffith, Jonas Jongejan, Amit Pitaru, and Alexander Chen. 2020. Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3382839
    [15]
    Crystal Chao, Maya Cakmak, and Andrea L Thomaz. 2010. Transparent active learning for robots. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 317–324.
    [16]
    Mu-Huan Chung, Mark Chignell, Lu Wang, Alexandra Jovicic, and Abhay Raman. 2020. Interactive machine learning for data exfiltration detection: active learning with human expertise. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 280–287.
    [17]
    Anamaria Crisan and Brittany Fiore-Gartland. 2021. Fits and Starts: Enterprise Use of AutoML and the Role of Humans in the Loop. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 601, 15 pages. https://doi.org/10.1145/3411764.3445775
    [18]
    Databricks. 2022. Augment experts. Empower citizen data scientists.https://www.databricks.com/product/automl.
    [19]
    Dennis P Doordan. 2003. On materials. Design Issues 19, 4 (2003), 3–8.
    [20]
    Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 278–288. https://doi.org/10.1145/3025453.3025739
    [21]
    John J. Dudley and Per Ola Kristensson. 2018. A Review of User Interface Design for Interactive Machine Learning. ACM Trans. Interact. Intell. Syst. 8, 2, Article 8 (jun 2018), 37 pages. https://doi.org/10.1145/3185517
    [22]
    EPFL. 2019. Food Image Dataset. https://www.epfl.ch/labs/mmspg/downloads/food-image-datasets/.
    [23]
    Jerry Alan Fails and Dan R. Olsen. 2003. Interactive Machine Learning. In Proceedings of the 8th International Conference on Intelligent User Interfaces (Miami, Florida, USA) (IUI ’03). Association for Computing Machinery, New York, NY, USA, 39–45. https://doi.org/10.1145/604045.604056
    [24]
    James Fogarty, Desney Tan, Ashish Kapoor, and Simon Winder. 2008. CueFlik: Interactive Concept Learning in Image Search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). Association for Computing Machinery, New York, NY, USA, 29–38. https://doi.org/10.1145/1357054.1357061
    [25]
    Wikimedia Foundation. 2022. Wikimedia Commons. https://commons.wikimedia.org/wiki/Main_Page.
    [26]
    Nicolo Fusi, Rishit Sheth, and Melih Elibol. 2018. Probabilistic matrix factorization for automated machine learning. Advances in neural information processing systems 31 (2018).
    [27]
    Elisa Giaccardi and Elvin Karana. 2015. Foundations of Materials Experience: An Approach for HCI. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 2447–2456. https://doi.org/10.1145/2702123.2702337
    [28]
    Google. 2017. Teachable Machine. https://teachablemachine.withgoogle.com/.
    [29]
    Google. 2022. Cloud AutoML Custom Machine Learning Models. https://cloud.google.com/automl/.
    [30]
    Google. 2022. People + AI Guidebook. https://pair.withgoogle.com/guidebook/.
    [31]
    Google Research. 2021. Facets - Visualization for ML Datasets. https://pair-code.github.io/facets/.
    [32]
    H2O.ai. 2022. H2O AutoML. https://h2o.ai/platform/h2o-automl/.
    [33]
    Rex Hartson and Pardha S Pyla. 2012. The UX Book: Process and guidelines for ensuring a quality user experience. Elsevier.
    [34]
    Patrick Hebron. 2016. Machine learning for designers. " O’Reilly Media, Inc.".
    [35]
    Lars Erik Holmquist. 2017. Intelligence on tap: artificial intelligence as a new design material. interactions 24, 4 (2017), 28–33.
    [36]
    IBM. 2022. IBM Watson Studio - AutoML - IBM AutoAI. https://www.ibm.com/cloud/watson-studio/autoai.
    [37]
    Shubhra Kanti Karmaker (“Santu”), Md. Mahadi Hassan, Micah J. Smith, Lei Xu, Chengxiang Zhai, and Kalyan Veeramachaneni. 2021. AutoML to Date and Beyond: Challenges and Opportunities. ACM Comput. Surv. 54, 8, Article 175 (oct 2021), 36 pages. https://doi.org/10.1145/3470918
    [38]
    David Laredo, Shangjie Frank Ma, Ghazaale Leylaz, Oliver Schütze, and Jian-Qiao Sun. 2020. Automatic model selection for fully connected neural networks. International Journal of Dynamics and Control 8, 4 (2020), 1063–1079.
    [39]
    Liner.ai. 2022. Machine learning in a few clicks. https://www.liner.ai/.
    [40]
    Raymond Lister. 2011. Concrete and other neo-Piagetian forms of reasoning in the novice programmer. In Conferences in research and practice in information technology series.
    [41]
    Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, Xiuqiang He, Zhenguo Li, and Yong Yu. 2020. AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Virtual Event, CA, USA) (KDD ’20). Association for Computing Machinery, New York, NY, USA, 2636–2645. https://doi.org/10.1145/3394486.3403314
    [42]
    Lobe.ai. 2021. Machine Learning Made Easy. https://www.lobe.ai/.
    [43]
    Danwei Tran Luciani, Martin Lindvall, Jonas Löwgren, 2018. Machine learning as a design material: a curated collection of exemplars for visual interaction. DS 91: Proceedings of NordDesign 2018, Linköping, Sweden, 14th-17th August 2018 (2018).
    [44]
    Microsoft. 2022. Azure Automated Machine Learning - AutoML. https://azure.microsoft.com/en-us/products/machine-learning/automatedml/.
    [45]
    Microsoft. 2022. Collaborative tools to help you create more effective and responsible human-AI experiences.https://www.microsoft.com/en-us/haxtoolkit/.
    [46]
    Swati Mishra and Jeffrey M Rzeszotarski. 2021. Designing Interactive Transfer Learning Tools for ML Non-Experts. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 364, 15 pages. https://doi.org/10.1145/3411764.3445096
    [47]
    Robert Munro Monarch. 2021. Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster.
    [48]
    Robert J. Moore, Raphael Arar, Guang-Jie Ren, and Margaret H. Szymanski. 2017. Conversational UX Design. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI EA ’17). Association for Computing Machinery, New York, NY, USA, 492–497. https://doi.org/10.1145/3027063.3027077
    [49]
    Felicia Ng, Jina Suh, and Gonzalo Ramos. 2020. Understanding and Supporting Knowledge Decomposition for Machine Teaching. In Proceedings of the 2020 ACM Designing Interactive Systems Conference (Eindhoven, Netherlands) (DIS ’20). Association for Computing Machinery, New York, NY, USA, 1183–1194. https://doi.org/10.1145/3357236.3395454
    [50]
    Changhoon Oh, Seonghyeon Kim, Jinhan Choi, Jinsu Eun, Soomin Kim, Juho Kim, Joonhwan Lee, and Bongwon Suh. 2020. Understanding How People Reason about Aesthetic Evaluations of Artificial Intelligence. In Proceedings of the 2020 ACM Designing Interactive Systems Conference (Eindhoven, Netherlands) (DIS ’20). Association for Computing Machinery, New York, NY, USA, 1169–1181. https://doi.org/10.1145/3357236.3395430
    [51]
    Jean Piaget. 1976. Piaget’s theory. In Piaget and his school. Springer, 11–23.
    [52]
    Hema Raghavan, Omid Madani, and Rosie Jones. 2006. Active learning with feedback on features and instances. The Journal of Machine Learning Research 7 (2006), 1655–1686.
    [53]
    Gonzalo Ramos, Christopher Meek, Patrice Simard, Jina Suh, and Soroush Ghorashi. 2020. Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction 35, 5-6 (2020), 413–451.
    [54]
    Erica Robles and Mikael Wiberg. 2010. Texturing the "Material Turn" in Interaction Design. In Proceedings of the Fourth International Conference on Tangible, Embedded, and Embodied Interaction (Cambridge, Massachusetts, USA) (TEI ’10). Association for Computing Machinery, New York, NY, USA, 137–144. https://doi.org/10.1145/1709886.1709911
    [55]
    Téo Sanchez, Baptiste Caramiaux, Jules Françoise, Frédéric Bevilacqua, and Wendy E. Mackay. 2021. How Do People Train a Machine? Strategies and (Mis)Understandings. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, Article 162 (apr 2021), 26 pages. https://doi.org/10.1145/3449236
    [56]
    Téo Sanchez, Baptiste Caramiaux, Pierre Thiel, and Wendy E. Mackay. 2022. Deep Learning Uncertainty in Machine Teaching. In 27th International Conference on Intelligent User Interfaces (Helsinki, Finland) (IUI ’22). Association for Computing Machinery, New York, NY, USA, 173–190. https://doi.org/10.1145/3490099.3511117
    [57]
    Advait Sarkar. 2016. Constructivist Design for Interactive Machine Learning. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (San Jose, California, USA) (CHI EA ’16). Association for Computing Machinery, New York, NY, USA, 1467–1475. https://doi.org/10.1145/2851581.2892547
    [58]
    Burr Settles. 2011. From theories to queries: Active learning in practice. In Active learning and experimental design workshop in conjunction with AISTATS 2010. JMLR Workshop and Conference Proceedings, 1–18.
    [59]
    Burr Settles. 2012. Active learning. Synthesis lectures on artificial intelligence and machine learning 6, 1(2012), 1–114.
    [60]
    Aliaksei Severyn and Alessandro Moschitti. 2013. Automatic feature engineering for answer selection and extraction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 458–467.
    [61]
    Hariharan Subramonyam, Jane Im, Colleen Seifert, and Eytan Adar. 2022. Solving Separation-of-Concerns Problems in Collaborative Design of Human-AI Systems through Leaky Abstractions. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 481, 21 pages. https://doi.org/10.1145/3491102.3517537
    [62]
    Hariharan Subramonyam, Colleen Seifert, and Eytan Adar. 2021. ProtoAI: Model-Informed Prototyping for AI-Powered Interfaces. In 26th International Conference on Intelligent User Interfaces (College Station, TX, USA) (IUI ’21). Association for Computing Machinery, New York, NY, USA, 48–58. https://doi.org/10.1145/3397481.3450640
    [63]
    Hariharan Subramonyam, Colleen Seifert, and Eytan Adar. 2021. Towards A Process Model for Co-Creating AI Experiences. In Designing Interactive Systems Conference 2021 (Virtual Event, USA) (DIS ’21). Association for Computing Machinery, New York, NY, USA, 1529–1543. https://doi.org/10.1145/3461778.3462012
    [64]
    Nicole Sultanum, Soroush Ghorashi, Christopher Meek, and Gonzalo Ramos. 2020. A Teaching Language for Building Object Detection Models. In Proceedings of the 2020 ACM Designing Interactive Systems Conference (Eindhoven, Netherlands) (DIS ’20). Association for Computing Machinery, New York, NY, USA, 1223–1234. https://doi.org/10.1145/3357236.3395545
    [65]
    Keith S Taber. 2012. Constructivism as educational theory: Contingency in learning, and optimally guided instruction. In Educational theory. Nova, 39–61.
    [66]
    Anh Truong, Austin Walters, Jeremy Goodsitt, Keegan Hines, C Bayan Bruss, and Reza Farivar. 2019. Towards automated machine learning: Evaluation and comparison of AutoML approaches and tools. In 2019 IEEE 31st international conference on tools with artificial intelligence (ICTAI). IEEE, 1471–1479.
    [67]
    Daniel Weitekamp, Erik Harpstead, and Ken R. Koedinger. 2020. An Interaction Design for Machine Teaching to Develop AI Tutors. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3313831.3376226
    [68]
    Doris Xin, Eva Yiwei Wu, Doris Jung-Lin Lee, Niloufar Salehi, and Aditya Parameswaran. 2021. Whither AutoML? Understanding the Role of Automation in Machine Learning Workflows. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 83, 16 pages. https://doi.org/10.1145/3411764.3445306
    [69]
    Qian Yang. 2018. Machine learning as a UX design material: how can we imagine beyond automation, recommenders, and reminders?. In AAAI Spring Symposia.
    [70]
    Qian Yang, Alex Scuito, John Zimmerman, Jodi Forlizzi, and Aaron Steinfeld. 2018. Investigating How Experienced UX Designers Effectively Work with Machine Learning. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). Association for Computing Machinery, New York, NY, USA, 585–596. https://doi.org/10.1145/3196709.3196730
    [71]
    Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-Examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376301
    [72]
    Qian Yang, Jina Suh, Nan-Chen Chen, and Gonzalo Ramos. 2018. Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). Association for Computing Machinery, New York, NY, USA, 573–584. https://doi.org/10.1145/3196709.3196729
    [73]
    Zhongyi Zhou and Koji Yatani. 2022. Gesture-Aware Interactive Machine Teaching with In-Situ Object Annotations. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA, Article 27, 14 pages. https://doi.org/10.1145/3526113.3545648

    Cited By

    View all
    • (2024)Mapping the Design Space of Teachable Social Media Feed ExperiencesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642120(1-20)Online publication date: 11-May-2024
    • (2024)Establishing Heuristics for Improving the Usability of GUI Machine Learning Tools for Novice UsersProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642087(1-19)Online publication date: 11-May-2024
    • (2024)Human-in-the-loop machine learning: Reconceptualizing the role of the user in interactive approachesInternet of Things10.1016/j.iot.2023.10104825(101048)Online publication date: Apr-2024
    • Show More Cited By

    Index Terms

    1. Addressing UX Practitioners’ Challenges in Designing ML Applications: an Interactive Machine Learning Approach

      Recommendations

      Comments

      Please enable JavaScript to view thecomments powered by Disqus.

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      IUI '23: Proceedings of the 28th International Conference on Intelligent User Interfaces
      March 2023
      972 pages
      ISBN:9798400701061
      DOI:10.1145/3581641
      This work is licensed under a Creative Commons Attribution International 4.0 License.

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 27 March 2023

      Check for updates

      Author Tags

      1. Interactive machine learning
      2. UX practice
      3. contextual inquiry
      4. interactive machine teaching

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Conference

      IUI '23
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 746 of 2,811 submissions, 27%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)806
      • Downloads (Last 6 weeks)75
      Reflects downloads up to 14 Aug 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Mapping the Design Space of Teachable Social Media Feed ExperiencesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642120(1-20)Online publication date: 11-May-2024
      • (2024)Establishing Heuristics for Improving the Usability of GUI Machine Learning Tools for Novice UsersProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642087(1-19)Online publication date: 11-May-2024
      • (2024)Human-in-the-loop machine learning: Reconceptualizing the role of the user in interactive approachesInternet of Things10.1016/j.iot.2023.10104825(101048)Online publication date: Apr-2024
      • (2024)From explainable to interactive AI: A literature review on current trends in human-AI interactionInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103301189(103301)Online publication date: Sep-2024
      • (2023)How Do UX Practitioners Communicate AI as a Design Material? Artifacts, Conceptions, and PropositionsProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596101(2263-2280)Online publication date: 10-Jul-2023

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Get Access

      Login options

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media