Just Accepted
A Roadmap of Explainable Artificial Intelligence: Explain to Whom, When, What and How?
Authors: 
Ziming Wang, Changwu Huang, Xin YaoAuthors Info & Claims
Accepted on 21 October 2024
Abstract
Explainable artificial intelligence (XAI) has gained significant attention, especially in AI-powered autonomous and adaptive systems (AASs). However, a discernible disconnect exists among research efforts across different communities. The machine learning community often overlooks “explaining to whom,” while the human-computer interaction community has examined various stakeholders with diverse explanation needs without addressing which XAI methods meet these requirements. Currently, no clear guidance exists on which XAI methods suit which specific stakeholders and their distinct needs. This hinders the achievement of the goal of XAI: providing human users with understandable interpretations. To bridge this gap, this paper presents a comprehensive XAI roadmap. Based on an extensive literature review, the roadmap summarizes different stakeholders, their explanation needs at different stages of the AI system lifecycle, the questions they may pose, and existing XAI methods. Then, by utilizing stakeholders’ inquiries as a conduit, the roadmap connects their needs to prevailing XAI methods, providing a guideline to assist researchers and practitioners to determine more easily which XAI methodologies can meet the specific needs of stakeholders in AASs. Finally, the roadmap discusses the limitations of existing XAI methods and outlines directions for future research.
References
[1]
Zakaria Abou El Houda, Bouziane Brik, and Lyes Khoukhi. 2022. ‘Why should I trust your IDS?’: An explainable deep learning framework for intrusion detection systems in internet of things networks. IEEE Open Journal of the Communications Society 3 (2022), 1164–1176.
[2]
Amina Adadi and Mohammed Berrada. 2018. Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access 6 (2018), 52138–52160.
[3]
Philip Adler, Casey Falk, Sorelle A Friedler, Tionney Nix, Gabriel Rybeck, Carlos Scheidegger, Brandon Smith, and Suresh Venkatasubramanian. 2018. Auditing black-box models for indirect influence. Knowledge and Information Systems 54, 1 (2018), 95–122.
[4]
Yongsu Ahn and Yu-Ru Lin. 2019. Fairsight: Visual analytics for fairness in decision making. IEEE Transactions on Visualization and Computer Graphics 26, 1 (2019), 1086–1095.
[5]
David Alvarez Melis and Tommi Jaakkola. 2018. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems, Vol. 31. 7786–7795.
[6]
Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 291–300.
[7]
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 (2016).
[8]
Anna Markella Antoniadi, Yuhan Du, Yasmine Guendouz, Lan Wei, Claudia Mazo, Brett A Becker, and Catherine Mooney. 2021. Current challenges and future opportunities for XAI in machine learning-based clinical decision support systems: A systematic review. Applied Sciences 11, 11 (2021), 5088.
[9]
Daniel W Apley and Jingyu Zhu. 2020. Visualizing the effects of predictor variables in black box supervised learning models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82, 4 (2020), 1059–1086.
[10]
Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58 (2020), 82–115.
[11]
Vijay Arya, Rachel KE Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C Hoffman, Stephanie Houde, Q Vera Liao, Ronny Luss, Aleksandra Mojsilović, et al. 2019. One explanation does not fit all: A toolkit and taxonomy of AI explainability techniques. arXiv preprint arXiv:1909.03012 (2019).
[12]
Vijay Arya, Rachel KE Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C Hoffman, Stephanie Houde, Q Vera Liao, Ronny Luss, Aleksandra Mojsilovic, et al. 2020. AI explainability 360: An extensible toolkit for understanding data and machine learning models. Journal of Machine Learning Research 21, 130 (2020), 1–6.
[13]
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. 2010. How to explain individual classification decisions. Journal of Machine Learning Research 11 (2010), 1803–1831.
[14]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations.
[15]
Pieter Barnard, Irene Macaluso, Nicola Marchetti, and Luiz A DaSilva. 2022. Resource reservation in sliced networks: An explainable artificial intelligence (XAI) approach. In ICC 2022-IEEE International Conference on Communications. IEEE, 1530–1535.
[16]
John Battelle. 2013. Behind the banner, a visualization of the adtech ecosystem. (2013). https://battellemedia.com/archives/2013/05/behind-the-banner-a-visualization-of-the-adtech-ecosystem.
[17]
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6541–6549.
[18]
Vaishak Belle and Ioannis Papantonis. 2021. Principles and practice of explainable machine learning. Frontiers in Big Data 4 (2021), 688969.
[19]
Adrien Bennetot, Jean-Luc Laurent, Raja Chatila, and Natalia Díaz-Rodríguez. 2019. Towards explainable neural-symbolic visual reasoning. In IJCAI Neural-Symbolic Learning and Reasoning Workshop.
[20]
Vitor Bento, Manoela Kohler, Pedro Diaz, Leonardo Mendoza, and Marco Aurelio Pacheco. 2021. Improving deep learning performance by using explainable artificial intelligence (XAI) approaches. Discover Artificial Intelligence 1, 1 (2021), 1–11.
[21]
Umang Bhatt, Adrian Weller, and José MF Moura. 2021. Evaluating and aggregating feature-based model explanations. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. 3016–3022.
[22]
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. 2020. Explainable machine learning in deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 648–657.
[23]
P. Biecek. 2019. Ceteris paribus plots (what-if plots) for explanations of a single observation. https://github.com/pbiecek/ceterisParibus.
[24]
William J Bingley, Caitlin Curtis, Steven Lockey, Alina Bialkowski, Nicole Gillespie, S Alexander Haslam, Ryan KL Ko, Niklas Steffens, Janet Wiles, and Peter Worthy. 2023. Where is the human in human-centered AI? Insights from developer priorities and user experiences. Computers in Human Behavior 141 (2023), 107617.
[25]
Mariusz Bojarski, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Larry J Ackel, Urs Muller, Phil Yeres, and Karol Zieba. 2018. VisualBackProp: Efficient visualization of CNNs for autonomous driving. In 2018 IEEE International Conference on Robotics and Automation (ICRA). 4701–4708.
[26]
Mariusz Bojarski, Philip Yeres, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Lawrence Jackel, and Urs Muller. 2017. Explaining how a deep neural network trained with end-to-end learning steers a car. arXiv preprint arXiv:1704.07911 (2017).
[27]
Andrea Bontempelli, Stefano Teso, Fausto Giunchiglia, and Andrea Passerini. 2022. Concept-level debugging of part-prototype networks. In Workshop on Trustworthy Artificial Intelligence as a Part of the ECML/PKDD 22 Program.
[28]
Olcay Boz. 2002. Extracting decision trees from trained neural networks. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 456–461.
[29]
Martim Brandao and Daniele Magazzeni. 2020. Explaining plans at scale: Scalable path planning explanations in navigation meshes using inverse optimization. In IJCAI-PRICAI 2020 Workshop on Explainable Artificial Intelligence (XAI).
[30]
Andrea Brennen. 2020. What do people really want when they say they want “explainable AI?” We asked 60 stakeholders. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–7.
[31]
Longbing Cao. 2022. AI in finance: Challenges, techniques, and opportunities. ACM Computing Surveys (CSUR) 55, 3 (2022), 1–38.
[32]
Steffen Castle, Robert Schwarzenberg, and Mohsen Pourvali. 2021. Detecting covariate drift with explanations. In CCF International Conference on Natural Language Processing and Chinese Computing. Springer, 317–322.
[33]
Larissa Chazette and Kurt Schneider. 2020. Explainability as a non-functional requirement: Challenges and recommendations. Requirements Engineering 25, 4 (2020), 493–514.
[34]
Ching-Ju Chen, Ling-Wei Chen, Chun-Hao Yang, Ya-Yu Huang, and Yueh-Min Huang. 2021. Improving CNN-based pest recognition with a post-hoc explanation of XAI. Research Square (2021).
[35]
Dehua Chen, Hongjin Zhao, Jianrong He, Qiao Pan, and Weiliang Zhao. 2021. An causal XAI diagnostic model for breast cancer based on mammography reports. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). 3341–3349.
[36]
Ning Chen, Bernardete Ribeiro, and An Chen. 2016. Financial credit risk assessment: A recent review. Artificial Intelligence Review 45 (2016), 1–23.
[37]
Runpu Chen, Le Yang, Steve Goodison, and Yijun Sun. 2020. Deep-learning approach to identifying cancer subtypes using high-dimensional genomic data. Bioinformatics 36, 5 (2020), 1476–1483.
[38]
Youngwon Choi, Wenxi Yu, Mahesh B Nagarajan, Pangyu Teng, Jonathan G Goldin, Steven S Raman, Dieter R Enzmann, Grace Hyun J Kim, and Matthew S Brown. 2023. Translating AI to clinical practice: Overcoming data shift with explainability. Radiographics 43, 5 (2023), e220105.
[39]
Hotman Christianto, Gary Kee Khoon Lee, Zhou Weigui Jair, Henry Kasim, and Deepu Rajan. 2022. Smart interpretable model (SIM) enabling subject matter experts in rule generation. Expert Systems with Applications 207 (2022), 117945.
[40]
Ching-Hua Chuan, Ruoyu Sun, Shiyun Tian, and Wan-Hsiu Sunny Tsai. 2024. EXplainable artificial intelligence (XAI) for facilitating recognition of algorithmic bias: An experiment from imposed users’ perspectives. Telematics and Informatics 91 (2024), 102135.
[41]
European Commission. 2019. Ethics guidelines for trustworthy AI. https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai.
[42]
Mark Craven and Jude Shavlik. 1995. Extracting tree-structured representations of trained networks. In Advances in Neural Information Processing Systems, Vol. 8. 24–30.
[43]
Mark W Craven and Jude W Shavlik. 1994. Using sampling and queries to extract rules from trained neural networks. In Proceedings of the Eleventh International Conference on International Conference on Machine Learning. 37–45.
[44]
Harry Freitas Da Cruz, Boris Pfahringer, Tom Martensen, Frederic Schneider, Alexander Meyer, Erwin Böttinger, and Matthieu-P Schapranow. 2021. Using interpretability approaches to update “black-box” clinical prediction models: An external validation study in nephrology. Artificial Intelligence in Medicine 111 (2021), 101982.
[45]
Anupam Datta, Shayak Sen, and Yair Zick. 2016. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In 2016 IEEE Symposium on Security and Privacy (SP). 598–617.
[46]
Omer Deperlioglu, Utku Kose, Deepak Gupta, Ashish Khanna, Fabio Giampaolo, and Giancarlo Fortino. 2022. Explainable framework for Glaucoma diagnosis by image processing and convolutional neural network synergy: Analysis with doctor evaluation. Future Generation Computer Systems 129 (2022), 152–169.
[47]
Shipi Dhanorkar, Christine T Wolf, Kun Qian, Anbang Xu, Lucian Popa, and Yunyao Li. 2021. Who needs to know what, when?: Broadening the explainable AI (XAI) design space by looking at explanations across the AI lifecycle. In Designing Interactive Systems Conference 2021. 1591–1602.
[48]
Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. 2018. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems, Vol. 31. 592–603.
[49]
Pedro Domingos. 2012. A few useful things to know about machine learning. Commun. ACM 55, 10 (2012), 78–87.
[50]
Jiqian Dong, Sikai Chen, Mohammad Miralinaghi, Tiantian Chen, Pei Li, and Samuel Labi. 2023. Why did the AI make that decision? Towards an explainable artificial intelligence (XAI) for autonomous driving systems. Transportation Research Part C: Emerging Technologies 156 (2023), 104358.
[51]
Yinpeng Dong, Hang Su, Jun Zhu, and Bo Zhang. 2017. Improving interpretability of deep neural networks with semantic information. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4306–4314.
[52]
Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017).
[53]
Finale Doshi-Velez, Mason Kortz, Ryan Budish, Chris Bavitz, Sam Gershman, David O’Brien, Kate Scott, Stuart Schieber, James Waldo, David Weinberger, et al. 2017. Accountability of AI under the law: The role of explanation. arXiv preprint arXiv:1711.01134 (2017).
[54]
Jeff Druce, Michael Harradon, and James Tittle. 2019. Explainable artificial intelligence (XAI) for increasing user trust in deep reinforcement learning driven autonomous systems. In NeurIPS 2019 Deep RL Workshop.
[55]
Joanita DSouza et al. 2020. Using exploratory data analysis for generating inferences on the correlation of COVID-19 cases. In 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT). IEEE, 1–6.
[56]
Hang Du, Hailin Shi, Dan Zeng, Xiao-Ping Zhang, and Tao Mei. 2020. The elements of end-to-end deep face recognition: A survey of recent advances. ACM Computing Surveys (CSUR) 54, 10s (2020), 1–42.
[57]
Upol Ehsan, Q Vera Liao, Michael Muller, Mark O Riedl, and Justin D Weisz. 2021. Expanding explainability: Towards social transparency in AI systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–19.
[58]
Upol Ehsan, Philipp Wintersberger, Q Vera Liao, Martina Mara, Marc Streit, Sandra Wachter, Andreas Riener, and Mark O Riedl. 2021. Operationalizing human-centered perspectives in explainable AI. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–6.
[59]
Shaker El-Sappagh, Jose M Alonso, SM Islam, Ahmad M Sultan, and Kyung Sup Kwak. 2021. A multilayer multimodal detection and prediction model based on explainable artificial intelligence for Alzheimer's disease. Scientific Reports 11, 1 (2021), 1–26.
[60]
Mica R Endsley. 2017. From here to autonomy: Lessons learned from human–automation research. Human Factors 59, 1 (2017), 5–27.
[61]
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2009. Visualizing higher-layer features of a deep network. University of Montreal 1341, 3 (2009), 1–13.
[62]
Dumitru Erhan, Aaron Courville, and Yoshua Bengio. 2010. Understanding representations learned in deep architectures. Department dInformatique et Recherche Operationnelle, University of Montreal, QC, Canada, Tech. Rep 1355, 1 (2010), 69.
[63]
Gary Ericson, William Anton Rohm, Josée Martens, Kent Sharkey, Craig Casey, Beth Harvey, and Nick Schonning. 2017. Team data science process documentation. Retrieved April 11 (2017), 2019.
[64]
Magnus Falk. 2019. Artificial intelligence in the boardroom. https://www.fca.org.uk/insight/artificial-intelligence-boardroom.
[65]
Fan Fang, Carmine Ventre, Lingbo Li, Leslie Kanthan, Fan Wu, and Michail Basios. 2020. Better model selection with a new definition of feature importance. arXiv preprint arXiv:2009.07708 (2020).
[66]
Juliana Jansen Ferreira and Mateus Monteiro. 2021. The human-AI relationship in decision-making: AI explanation to support people on justifying their decisions. arXiv preprint arXiv:2102.05460 (2021).
[67]
Juliana J Ferreira and Mateus S Monteiro. 2020. What are people doing about XAI user experience? A survey on AI explainability research and practice. In International Conference on Human-Computer Interaction. 56–73.
[68]
Jerome H Friedman. 2001. Greedy function approximation: A gradient boosting machine. Annals of Statistics (2001), 1189–1232.
[69]
Felix Friedrich, Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. 2022. A typology to explore and guide explanatory interactive machine learning. arXiv preprint arXiv:2203.03668 (2022).
[70]
Detlev GABEL and Tim HICKMAN. 2019. GDPR handbook: Unlocking the EU general data protection regulation. White & Case Technology Newsflash (2019). https://www.whitecase.com/insight-our-thinking/gdpr-handbook-unlocking-eu-general-data-protection-regulation.
[71]
Chen Gao, Tzu-Heng Lin, Nian Li, Depeng Jin, and Yong Li. 2023. Cross-platform item recommendation for online social e-commerce. IEEE Transactions on Knowledge and Data Engineering 35, 2 (2023), 1351–1364.
[72]
Julie Gerlings, Millie Søndergaard Jensen, and Arisa Shollo. 2022. Explainable AI, but explainable to whom? An exploratory case study of XAI in healthcare. In Handbook of Artificial Intelligence in Healthcare. Springer, 169–198.
[73]
Julie Gerlings, Arisa Shollo, and Ioanna Constantiou. 2020. Reviewing the need for explainable artificial intelligence (XAI). arXiv preprint arXiv:2012.01007 (2020).
[74]
Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA). 80–89.
[75]
Alex Goldstein, Adam Kapelner, Justin Bleich, and Emil Pitkin. 2015. Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation. Journal of Computational and Graphical Statistics 24, 1 (2015), 44–65.
[76]
Przemyslaw A Grabowicz, Nicholas Perello, and Aarshee Mishra. 2022. Marrying fairness and explainability in supervised learning. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 1905–1916.
[77]
Yulia Grushetskaya, Mike Sips, Reyko Schachtschneider, Mohammadmehdi Saberioon, and Akram Mahan. 2024. HPExplorer: XAI method to explore the relationship between hyperparameters and model performance. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 319–334.
[78]
Venkat Gudivada, Amy Apon, and Junhua Ding. 2017. Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations. International Journal on Advances in Software 10, 1 (2017), 1–20.
[79]
Riccardo Guidotti, Anna Monreale, Fosca Giannotti, Dino Pedreschi, Salvatore Ruggieri, and Franco Turini. 2019. Factual and counterfactual explanations for black box decision making. IEEE Intelligent Systems 34, 6 (2019), 14–23.
[80]
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM Computing Surveys (CSUR) 51, 5 (2018), 1–42.
[81]
Calvin Guillot Suarez. 2022. Human-in-the-loop hyperparameter tuning of deep nets to improve explainability of classifications. Master's thesis. Aalto University. School of Electrical Engineering. http://urn.fi/URN:NBN:fi:aalto-202205223354
[82]
Karthik S Gurumoorthy, Amit Dhurandhar, Guillermo Cecchi, and Charu Aggarwal. 2019. Efficient data representation by selecting prototypes with importance weights. In 2019 IEEE International Conference on Data Mining (ICDM). 260–269.
[83]
Mark Haakman, Luís Cruz, Hennie Huijgens, and Arie van Deursen. 2021. AI lifecycle models need to be revised. Empirical Software Engineering 26, 5 (2021), 1–29.
[84]
David Hardage and Peyman Najafirad. 2020. Hate and toxic speech detection in the context of covid-19 pandemic using XAI: Ongoing applied research. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020. 1–5.
[85]
Michael Harradon, Jeff Druce, and Brian Ruttenberg. 2018. Causal learning and explanation of deep neural networks via autoencoded activations. arXiv preprint arXiv:1802.00541 (2018).
[86]
Alexander Heimerl, Katharina Weitz, Tobias Baur, and Elisabeth Andre. 2020. Unraveling ML models of emotion with nova: Multi-level explainable AI for non-experts. IEEE Transactions on Affective Computing 13, 3 (2020), 1155–1167.
[87]
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016. Generating visual explanations. In European Conference on Computer Vision. Springer, 3–19.
[88]
Andreas Henelius, Kai Puolamäki, Henrik Boström, Lars Asker, and Panagiotis Papapetrou. 2014. A peek into the black box: Exploring classifiers by randomization. Data Mining and Knowledge Discovery 28, 5 (2014), 1503–1529.
[89]
Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608 (2018).
[90]
Fred Hohman, Haekyu Park, Caleb Robinson, and Duen Horng Polo Chau. 2019. Summit: Scaling deep learning interpretability by visualizing activation and attribution summarizations. IEEE Transactions on Visualization and Computer Graphics 26, 1 (2019), 1096–1106.
[91]
Andreas Holzinger, Chris Biemann, Constantinos S Pattichis, and Douglas B Kell. 2017. What do we need to build explainable AI systems for the medical domain? arXiv preprint arXiv:1712.09923 (2017).
[92]
Changwu Huang, Zeqi Zhang, Bifei Mao, and Xin Yao. 2022. An overview of artificial intelligence ethics. IEEE Transactions on Artificial Intelligence (2022), 1–21.
[93]
Christina Humer, Andreas Hinterreiter, Benedikt Leichtmann, Martina Mara, and Marc Streit. 2024. Reassuring, misleading, debunking: Comparing effects of XAI methods on human decisions. ACM Transactions on Interactive Intelligent Systems 14, 3 (2024), 1–36.
[94]
Fatima Hussain, Rasheed Hussain, and Ekram Hossain. 2021. Explainable artificial intelligence (XAI): An engineering perspective. arXiv preprint arXiv:2101.03613 (2021).
[95]
Gandhi Jafta, Alta de Waal, Iena Derks, and Emma Ruttkamp-Bloem. 2022. Evaluation of XAI as an enabler for fairness, accountability and transparency. In Second Southern African Conference for Artificial Intelligence Research. 541–542.
[96]
Helen Jiang and Erwen Senge. 2021. On two XAI cultures: A case study of non-technical explanations in deployed AI system. arXiv preprint arXiv:2112.01016 (2021).
[97]
Anna Jobin, Marcello Ienca, and Effy Vayena. 2019. The global landscape of AI ethics guidelines. Nature Machine Intelligence 1, 9 (2019), 389–399.
[98]
U Johansson, R König, and L Niklasson. 2003. Rule extraction from trained neural networks using genetic programming. In 13th International Conference on Artificial Neural Networks. 13–16.
[99]
Ulf Johansson and Lars Niklasson. 2009. Evolving decision trees using oracle guides. In 2009 IEEE Symposium on Computational Intelligence and Data Mining. IEEE, 238–244.
[100]
Neesha Jothi, Wahidah Husain, et al. 2021. Predicting generalized anxiety disorder among women using shapley value. Journal of Infection and Public Health 14, 1 (2021), 103–108.
[101]
Minsuk Kahng, Pierre Y Andrews, Aditya Kalro, and Duen Horng Chau. 2017. ActiVis: Visual exploration of industry-scale deep neural network models. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 88–97.
[102]
Uday Kamath and John Liu. 2021. Explainable artificial intelligence: An introduction to interpretable machine learning. Springer.
[103]
Amany A Kandeel, Hazem M Abbas, and Hossam S Hassanein. 2021. Explainable model selection of a convolutional neural network for driver's facial emotion identification. In International Conference on Pattern Recognition. Springer, 699–713.
[104]
Ramisetty Kavya, Jabez Christopher, Subhrakanta Panda, and Y Bakthasingh Lazarus. 2021. Machine learning and XAI approaches for allergy diagnosis. Biomedical Signal Processing and Control 69 (2021), 102681.
[105]
Aditya Khamparia, Deepak Gupta, Ashish Khanna, and Valentina E Balas. 2022. Biomedical data analysis and processing using explainable (XAI) and responsive artificial intelligence (RAI). Springer.
[106]
Nadia Khan, Muhammad Nauman, Ahmad S Almadhor, Nadeem Akhtar, Abdullah Alghuried, and Adi Alhudhaif. 2024. Guaranteeing correctness in black-box machine learning: A fusion of explainable AI and formal methods for healthcare decision-making. IEEE Access (2024).
[107]
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. 2018. Textual explanations for self-driving vehicles. In Proceedings of the European Conference on Computer Vision (ECCV). 563–578.
[108]
Miryung Kim, Thomas Zimmermann, Robert DeLine, and Andrew Begel. 2016. The emerging role of data scientists on software development teams. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, 96–107.
[109]
Inna Kolyshkina and Simeon Simoff. 2019. Interpretability of machine learning solutions in industrial decision engineering. In Australasian Conference on Data Mining. Springer, 156–170.
[110]
Matthieu Komorowski, Dominic C Marshall, Justin D Salciccioli, and Yves Crutain. 2016. Exploratory data analysis. Secondary Analysis of Electronic Health Records (2016), 185–203.
[111]
Josua Krause, Adam Perer, and Kenney Ng. 2016. Interacting with predictions: Visual inspection of black-box machine learning models. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 5686–5697.
[112]
R Krishnan, G Sivakumar, and P Bhattacharya. 1999. Extracting decision trees from trained neural networks. Pattern Recognition 32, 12 (1999), 1999–2009.
[113]
Jean-Philippe Kröll, Simon B Eickhoff, Felix Hoffstaedter, and Kaustubh R Patil. 2020. Evolving complex yet interpretable representations: Application to Alzheimer's diagnosis and prognosis. In 2020 IEEE Congress on Evolutionary Computation (CEC). IEEE, 1–8.
[114]
Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of explanatory debugging to personalize interactive machine learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces. 126–137.
[115]
Markus Langer, Daniel Oster, Timo Speith, Holger Hermanns, Lena Kästner, Eva Schmidt, Andreas Sesing, and Kevin Baum. 2021. What do we want from explainable artificial intelligence (XAI)?–A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artificial Intelligence 296 (2021), 103473.
[116]
Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki. 2017. Inverse classification for comparison-based interpretability in machine learning. arXiv preprint arXiv:1712.08443 (2017).
[117]
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. arXiv preprint arXiv:1606.04155 (2016).
[118]
Bruno Lepri, Nuria Oliver, Emmanuel Letouzé, Alex Pentland, and Patrick Vinck. 2018. Fair, transparent, and accountable algorithmic decision-making processes. Philosophy & Technology 31, 4 (2018), 611–627.
[119]
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. 2018. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32. 3530–3537.
[120]
Q Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: Informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–15.
[121]
Q Vera Liao and Kush R Varshney. 2021. Human-centered explainable AI (XAI): From algorithms to user experiences. arXiv preprint arXiv:2110.10790 (2021).
[122]
Brian Y Lim and Anind K Dey. 2010. Toolkit to support intelligibility in context-aware applications. In Proceedings of the 12th ACM International Conference on Ubiquitous Computing. 13–22.
[123]
Brian Y Lim, Anind K Dey, and Daniel Avrahami. 2009. Why and why not explanations improve the intelligibility of context-aware intelligent systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2119–2128.
[124]
Gabriel Lima, Nina Grgić-Hlača, Jin Keun Jeong, and Meeyoung Cha. 2022. The conflict between explainable and accountable decision-making algorithms. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 2103–2113.
[125]
Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. 2020. Explainable AI: A review of machine learning interpretability methods. Entropy 23, 1 (2020), 18.
[126]
Mengchen Liu, Jiaxin Shi, Kelei Cao, Jun Zhu, and Shixia Liu. 2017. Analyzing the training processes of deep generative models. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 77–87.
[127]
Jörn Lötsch, Dario Kringel, and Alfred Ultsch. 2021. Explainable artificial intelligence (XAI) in biomedicine: Making AI decisions trustworthy for physicians and patients. BioMedInformatics 2, 1 (2021), 1–17.
[128]
Yin Lou, Rich Caruana, and Johannes Gehrke. 2012. Intelligible models for classification and regression. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 150–158.
[129]
Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. 2013. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 623–631.
[130]
Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems. 4768–4777.
[131]
Scott M Lundberg, Bala Nair, Monica S Vavilala, Mayumi Horibe, Michael J Eisses, Trevor Adams, David E Liston, Daniel King-Wai Low, Shu-Fang Newman, Jerry Kim, et al. 2018. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature Biomedical Engineering 2, 10 (2018), 749–760.
[132]
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015).
[133]
Avleen Malhi, Samanta Knapic, and Kary Främling. 2020. Explainable agents for less bias in human-agent decision making. In International Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems. Springer, 129–146.
[134]
Nicholas Maltbie, Nan Niu, Matthew Van Doren, and Reese Johnson. 2021. XAI tools in the public sector: A case study on predicting combined sewer overflows. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1032–1044.
[135]
Wilson E Marcílio and Danilo M Eler. 2020. From explanations to feature selection: Assessing SHAP values as feature selection mechanism. In 2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). Ieee, 340–347.
[136]
Andrés R Masegosa, Ana M Martínez, Darío Ramos-López, Helge Langseth, Thomas D Nielsen, and Antonio Salmerón. 2020. Analyzing concept drift: A case study in the financial sector. Intelligent Data Analysis 24, 3 (2020), 665–688.
[137]
Kate Matsudaira. 2015. The science of managing data science. Commun. ACM 58, 6 (2015), 44–47.
[138]
John A McDermid, Yan Jia, Zoe Porter, and Ibrahim Habli. 2021. Artificial intelligence explainability: The technical and ethical dimensions. Philosophical Transactions of the Royal Society A 379, 2207 (2021), 20200363.
[139]
Linhao Meng, Stef Van Den Elzen, and Anna Vilanova. 2022. ModelWise: Interactive model comparison for model diagnosis, improvement and selection. In Computer Graphics Forum, Vol. 41. 97–108.
[140]
Christian Meske, Enrico Bunde, Johannes Schneider, and Martin Gersch. 2022. Explainable artificial intelligence: Objectives, stakeholders, and future research opportunities. Information Systems Management 39, 1 (2022), 53–63.
[141]
Agnieszka Mikołajczyk, Michał Grochowski, and Arkadiusz Kwasigroch. 2021. Towards explainable classifiers using the counterfactual approach-global explanations for discovering bias in data. Journal of Artificial Intelligence and Soft Computing Research 11, 1 (2021), 51–67.
[142]
Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 267 (2019), 1–38.
[143]
Yao Ming, Huamin Qu, and Enrico Bertini. 2018. Rulematrix: Visualizing and understanding classifiers with rules. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 342–352.
[144]
Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter, and Luciano Floridi. 2016. The ethics of algorithms: Mapping the debate. Big Data & Society 3, 2 (2016), 2053951716679679.
[145]
Sina Mohseni, Niloofar Zarei, and Eric D Ragan. 2021. A multidisciplinary survey and framework for design and evaluation of explainable AI systems. ACM Transactions on Interactive Intelligent Systems (TiiS) 11, 3-4 (2021), 1–45.
[146]
Christoph Molnar. 2020. Interpretable machine learning. Lulu. com.
[147]
Isaac Monteath and Raymond Sheh. 2018. Assisted and incremental medical diagnosis using explainable artificial intelligence. In Proceedings of the 2nd Workshop on Explainable Artificial Intelligence. 104–108.
[148]
Pedro A Moreno-Sanchez. 2021. An automated feature selection and classification pipeline to improve explainability of clinical prediction models. In 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI). IEEE, 527–534.
[149]
Edoardo Mosca, Maximilian Wich, and Georg Groh. 2021. Understanding and interpreting the impact of user context in hate speech detection. In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media. 91–102.
[150]
Sakib Mostafa, Debajyoti Mondal, Michael Beck, Christopher Bidinosti, Christopher Henry, and Ian Stavness. 2021. Visualizing feature maps for model selection in convolutional neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1362–1371.
[151]
Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. 2020. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 607–617.
[152]
FG Mourik. 2023. IterSHAP: An XAI feature selection method for small high-dimensional datasets. Master's thesis. University of Twente.
[153]
Shane T Mueller. 2020. Cognitive anthropomorphism of AI: How humans and computers classify images. Ergonomics in Design 28, 3 (2020), 12–19.
[154]
Fatemeh Nargesian, Horst Samulowitz, Udayan Khurana, Elias B Khalil, and Deepak S Turaga. 2017. Learning feature engineering for classification. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2529–2535.
[155]
Shweta Narkar, Yunfeng Zhang, Q Vera Liao, Dakuo Wang, and Justin D Weisz. 2021. Model lineUpper: Supporting interactive model comparison at multiple levels for autoML. In 26th International Conference on Intelligent User Interfaces. 170–174.
[156]
Tien N Nguyen and Raymond Choo. 2021. Human-in-the-loop XAI-enabled vulnerability detection, investigation, and mitigation. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1210–1212.
[157]
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. 2018. The building blocks of interpretability. Distill 3, 3 (2018), e10.
[158]
Julian D Olden and Donald A Jackson. 2002. Illuminating the “black box”: A randomization approach for understanding variable contributions in artificial neural networks. Ecological Modelling 154, 1-2 (2002), 135–150.
[159]
Lara O’Reilly. 2014. Here's one way to find out which advertisers are tracking you across the internet. (2014). https://www.businessinsider.com/floodwatch-ad-tracking-chrome-extension-2014-10.
[160]
Dieudonne N Ouedraogo. 2021. Interpretable machine learning model selection for breast cancer diagnosis based on k-means clustering. Applied Medical Informatics 43, 3 (2021), 91–102.
[161]
James Overton. 2011. Scientific Explanation and Computation. In ExaCt. 41–50.
[162]
Frederik Pahde, Maximilian Dreyer, Wojciech Samek, and Sebastian Lapuschkin. 2023. Reveal to revise: An explainable AI life cycle for iterative bias correction of deep models. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 596–606.
[163]
Iam Palatnik de Sousa, Marley MBR Vellasco, and Eduardo Costa da Silva. 2021. Explainable artificial intelligence for bias detection in covid CT-Scan classifiers. Sensors 21, 16 (2021), 5657.
[164]
Raja Parasuraman, Thomas B Sheridan, and Christopher D Wickens. 2008. Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs. Journal of Cognitive Engineering and Decision Making 2, 2 (2008), 140–160.
[165]
Ankita Ramjibhai Patel, Jaganmohan Chandrasekaran, Yu Lei, Raghu N Kacker, and D Richard Kuhn. 2022. A combinatorial approach to fairness testing of machine learning models. In 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 94–101.
[166]
Junfeng Peng, Kaiqiang Zou, Mi Zhou, Yi Teng, Xiongyong Zhu, Feifei Zhang, and Jun Xu. 2021. An explainable artificial intelligence framework for the deterioration risk prediction of hepatitis patients. Journal of Medical Systems 45, 5 (2021), 1–9.
[167]
Jeremy Petch, Shuang Di, and Walter Nelson. 2022. Opening the black box: The promise and limitations of explainable machine learning in cardiology. Canadian Journal of Cardiology 38, 2 (2022), 204–213.
[168]
Nicola Pezzotti, Thomas Höllt, Jan Van Gemert, Boudewijn PF Lelieveldt, Elmar Eisemann, and Anna Vilanova. 2017. Deepeyes: Progressive visual analytics for designing deep neural networks. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 98–108.
[169]
Anna Polzer, Jürgen Fleiß, Thomas Ebner, Philipp Kainz, Christoph Koeth, and Stefan Thalmann. 2022. Validation of AI-based information systems for sensitive use cases: Using an XAI approach in pharmaceutical engineering. In Proceedings of the 55th Hawaii International Conference on System Sciences. 1500–1509.
[170]
Romila Pradhan, Jiongli Zhu, Boris Glavic, and Babak Salimi. 2022. Interpretable data-based explanations for fairness debugging. In Proceedings of the 2022 International Conference on Management of Data. 247–261.
[171]
Alun Preece, Dan Harborne, Dave Braines, Richard Tomsett, and Supriyo Chakraborty. 2018. Stakeholders in explainable AI. arXiv preprint arXiv:1810.00184 (2018).
[172]
Pearl Pu and Li Chen. 2007. Trust-inspiring explanation interfaces for recommender systems. Knowledge-Based Systems 20, 6 (2007), 542–556.
[173]
Jaber Rad, Karthik K Tennankore, Amanda Vinson, and Syed Sibte Raza Abidi. 2022. Extracting surrogate decision trees from black-box models to explain the temporal importance of clinical features in predicting kidney graft survival. In International Conference on Artificial Intelligence in Medicine. Springer, 88–98.
[174]
Enayat Rajabi and Kobra Etminani. 2021. Towards a knowledge graph-based explainable decision support hystem in Healthcare. Stud Health Technol Inform 281 (2021), 502–503.
[175]
Gabrielle Ras, Ning Xie, Marcel van Gerven, and Derek Doran. 2022. Explainable deep learning: A field guide for the uninitiated. Journal of Artificial Intelligence Research 73 (2022), 329–397.
[176]
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. ‘Why should I trust you?’ Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1135–1144.
[177]
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: High-precision model-agnostic explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32. 1527–1535.
[178]
Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. 2020. Interpretations are useful: Penalizing explanations to align neural networks with prior knowledge. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 8116–8126.
[179]
Avi Rosenfeld and Ariella Richardson. 2019. Explainability in human–agent systems. Autonomous Agents and Multi-Agent Systems 33, 6 (2019), 673–705.
[180]
Patrik Sabol, Peter Sinčák, Pitoyo Hartono, Pavel Kočan, Zuzana Benetinová, Alžbeta Blichárová, L’udmila Verbóová, Erika Štammová, Antónia Sabolová-Fabianová, and Anna Jašková. 2020. Explainable classifier for improving the accountability in decision-making for colorectal cancer diagnosis from histopathological images. Journal of Biomedical Informatics 109 (2020), 103523.
[181]
Waddah Saeed and Christian Omlin. 2023. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems 263 (2023), 110273.
[182]
Wojciech Samek and Klaus-Robert Müller. 2019. Towards explainable artificial intelligence. Springer, 5–22.
[183]
Wojciech Samek, Thomas Wiegand, and Klaus-Robert Müller. 2017. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arXiv:1708.08296 (2017).
[184]
Salih Sarp, Murat Kuzlu, Emmanuel Wilson, Umit Cali, and Ozgur Guler. 2021. The enlightening role of explainable artificial intelligence in chronic wound classification. Electronics 10, 12 (2021), 1406.
[185]
NB Sarter and DD Woods. 1995. Autonomy, authority, and observability: The evolution of critical automation properties and their impact on man-machine coordination and cooperation. In Proceedings of the 6th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Man-Machine Systems.
[186]
Benajmin Schmidt. 2016. A public exploratory data analysis of gender bias in teaching evaluations. In Proceedings of Workshop on Visualization for the Digital Humanities (Vis4DH 16).
[187]
Tjeerd AJ Schoonderwoerd, Wiard Jorritsma, Mark A Neerincx, and Karel Van Den Bosch. 2021. Human-centered XAI: Developing design patterns for explanations of clinical decision support systems. International Journal of Human-Computer Studies 154 (2021), 102684.
[188]
Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. 2020. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence 2, 8 (2020), 476–486.
[189]
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision. 618–626.
[190]
Rudy Setiono and Huan Liu. 1995. Understanding neural networks via rule extraction. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, Vol. 1. 480–485.
[191]
Arash Shaban-Nejad, Martin Michalowski, John S Brownstein, and David L Buckeridge. 2021. Guest editorial explainable AI: Towards fairness, accountability, transparency and trust in healthcare. IEEE Journal of Biomedical and Health Informatics 25, 7 (2021), 2374–2375.
[192]
Hetan Shah. 2018. Algorithmic accountability. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 376, 2128 (2018), 20170362.
[193]
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning. PMLR, 3145–3153.
[194]
Andrew Silva, Mariah Schrum, Erin Hedlund-Botti, Nakul Gopalan, and Matthew Gombolay. 2023. Explainable artificial intelligence: Evaluating the objective and subjective impacts of XAI on human-agent interaction. International Journal of Human–Computer Interaction 39, 7 (2023), 1390–1404.
[195]
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations. Citeseer.
[196]
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. 2017. SmoothGrad: Removing noise by adding noise. arXiv preprint arXiv:1706.03825 (2017).
[197]
Peter Sollich. 2002. Bayesian methods for support vector machines: Evidence and predictive class probabilities. Machine Learning 46, 1 (2002), 21–52.
[198]
Thilo Spinner, Daniel Fürst, and Mennatallah El-Assady. 2024. iNNspector: Visual, interactive deep model debugging. arXiv preprint arXiv:2407.17998 (2024).
[199]
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. 2014. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 (2014).
[200]
Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. 2021. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3619–3629.
[201]
Pierre Stock and Moustapha Cisse. 2018. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. In Proceedings of the European Conference on Computer Vision (ECCV). 498–512.
[202]
Jungyo Suh, Sangjun Yoo, Juhyun Park, Sung Yong Cho, Min Chul Cho, Hwancheol Son, and Hyeon Jeong. 2020. Development and validation of an explainable artificial intelligence-based decision-supporting tool for prostate biopsy. BJU International 126, 6 (2020), 694–703.
[203]
Shimon Sumita, Hiroyuki Nakagawa, and Tatsuhiro Tsuchiya. 2023. Xtune: An XAI-based hyperparameter tuning method for time-series forecasting using deep learning. (2023).
[204]
Dong Sun, Zezheng Feng, Yuanzhe Chen, Yong Wang, Jia Zeng, Mingxuan Yuan, Ting-Chuen Pong, and Huamin Qu. 2020. DFSeer: A visual analytics approach to facilitate model selection for demand forecasting. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.
[205]
Harini Suresh, Steven R Gomez, Kevin K Nam, and Arvind Satyanarayan. 2021. Beyond expertise and roles: A framework to characterize the stakeholders of interpretable machine learning and their needs. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–16.
[206]
Jianrong Tao, Yu Xiong, Shiwei Zhao, Yuhong Xu, Jianshi Lin, Runze Wu, and Changjie Fan. 2020. XAI-driven explainable multi-view game cheating detection. In 2020 IEEE Conference on Games (CoG). IEEE, 144–151.
[207]
Stefano Teso and Kristian Kersting. 2019. Explanatory interactive machine learning. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 239–245.
[208]
Patanamon Thongtanunam, Xin Yang, Norihiro Yoshida, Raula Gaikovina Kula, Ana Erika Camargo Cruz, Kenji Fujiwara, and Hajimu Iida. 2014. Reda: A web-based visualization tool for analyzing modern code review dataset. In 2014 IEEE International Conference on Software Maintenance and Evolution. IEEE, 605–608.
[209]
Gabriele Tolomei, Fabrizio Silvestri, Andrew Haines, and Mounia Lalmas. 2017. Interpretable predictions of tree-based ensembles via actionable feature tweaking. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 465–474.
[210]
Richard Tomsett, Dave Braines, Dan Harborne, Alun Preece, and Supriyo Chakraborty. 2018. Interpretable to whom? A role-based model for analyzing interpretable machine learning systems. arXiv preprint arXiv:1806.07552 (2018).
[211]
Geoffrey G Towell and Jude W Shavlik. 1993. Extracting refined rules from knowledge-based neural networks. Machine Learning 13, 1 (1993), 71–101.
[212]
Sandhya Tripathi, N Hemachandra, and Prashant Trivedi. 2020. Interpretable feature subset selection: A Shapley value based approach. In 2020 IEEE International Conference on Big Data (Big Data). IEEE, 5463–5472.
[213]
Stéphane Tufféry. 2011. Data mining and statistics for decision making. John Wiley & Sons.
[214]
European Union. 2018. General data protection regulation (GDPR). https://gdpr-info.eu/.
[215]
Jasper van der Waa, Tjeerd Schoonderwoerd, Jurriaan van Diggelen, and Mark Neerincx. 2020. Interpretable confidence measures for decision support systems. International Journal of Human-Computer Studies 144 (2020), 102493.
[216]
Corne Van Zyl, Xianming Ye, and Raj Naidoo. 2024. Harnessing eXplainable artificial intelligence for feature selection in time series energy forecasting: A comparative analysis of Grad-CAM and SHAP. Applied Energy 353 (2024), 122079.
[217]
Sahil Verma, John Dickerson, and Keegan Hines. 2020. Counterfactual explanations for machine learning: A review. arXiv preprint arXiv:2010.10596 (2020).
[218]
Tom Vermeire, Thibault Laugel, Xavier Renard, David Martens, and Marcin Detyniecki. 2021. How to choose an explainability method? Towards a methodical implementation of XAI in practice. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 521–533.
[219]
Trent W Victor, Emma Tivesten, Pär Gustavsson, Joel Johansson, Fredrik Sangberg, and Mikael Ljung Aust. 2018. Automation expectation mismatch: Incorrect prediction despite eyes on threat and hands on wheel. Human Factors 60, 8 (2018), 1095–1116.
[220]
Marina M-C Vidovic, Nico Görnitz, Klaus-Robert Müller, and Marius Kloft. 2016. Feature importance measure for non-linear learning algorithms. arXiv preprint arXiv:1611.07567 (2016).
[221]
Giulia Vilone and Luca Longo. 2020. Explainable artificial intelligence: A systematic review. arXiv preprint arXiv:2006.00093 (2020).
[222]
Klaus Virtanen. 2022. Using XAI tools to detect harmful bias in ML models.
[223]
Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harv. JL & Tech. 31 (2017), 841.
[224]
Syed Wali and Irfan Khan. 2021. Explainable AI and random forest based reliable intrusion detection system. TechRxiv Preprint (2021).
[225]
Dakuo Wang, Justin D Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray. 2019. Human-AI collaboration in data science: Exploring data scientists’ perceptions of automated AI. In Proceedings of the ACM on Human-Computer Interaction, Vol. 3. 1–24.
[226]
Maonan Wang, Kangfeng Zheng, Yanqing Yang, and Xiujuan Wang. 2020. An explainable machine learning framework for intrusion detection systems. IEEE Access 8 (2020), 73127–73141.
[227]
Tong Wang. 2019. Gaining free or low-cost interpretability with interpretable partial substitute. In International Conference on Machine Learning. PMLR, 6505–6514.
[228]
Ziming Wang, Changwu Huang, Yun Li, and Xin Yao. 2024. Multi-objective feature attribution explanation for explainable machine learning. ACM Transactions on Evolutionary Learning and Optimization 4, 1 (2024), 1–32.
[229]
Ziming Wang, Changwu Huang, and Xin Yao. 2023. Feature attribution explanation to detect harmful dataset shift. In 2023 International Joint Conference on Neural Networks (IJCNN). IEEE, 1–8.
[230]
Ziming Wang, Changwu Huang, and Xin Yao. 2024. Procedural fairness in machine learning. arXiv preprint arXiv:2404.01877 (2024).
[231]
Geoffrey I Webb, Loong Kuan Lee, François Petitjean, and Bart Goethals. 2017. Understanding concept drift. arXiv preprint arXiv:1704.00362 (2017).
[232]
Katharina Weitz, Dominik Schiller, Ruben Schlagowski, Tobias Huber, and Elisabeth André. 2019. ‘Do you trust me?’ Increasing user-trust by integrating virtual agents in explainable AI interaction design. In Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents. 7–9.
[233]
Adrian Weller. 2017. Challenges for transparency. arXiv preprint arXiv:1708.01870 (2017).
[234]
Maximilian Wich, Jan Bauer, and Georg Groh. 2020. Impact of politically biased data on hate speech classification. In Proceedings of the Fourth Workshop on Online Abuse and Harms. 54–64.
[235]
Andrew C Wicks, Shawn L Berman, and Thomas M Jones. 1999. The structure of optimal trust: Moral and strategic implications. Academy of Management Review 24, 1 (1999), 99–116.
[236]
Gesa Wiegand, Matthias Schmidmaier, Thomas Weber, Yuanting Liu, and Heinrich Hussmann. 2019. I drive-you trust: Explaining driving behavior of autonomous cars. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1–6.
[237]
Rüdiger Wirth and Jochen Hipp. 2000. CRISP-DM: Towards a standard process model for data mining. In Proceedings of the 4th International Conference on the Practical Applications of Knowledge Discovery and Data Mining, Vol. 1. Manchester, 29–39.
[238]
Kanit Wongsuphasawat, Daniel Smilkov, James Wexler, Jimbo Wilson, Dandelion Mane, Doug Fritz, Dilip Krishnan, Fernanda B Viégas, and Martin Wattenberg. 2017. Visualizing dataflow graphs of deep learning models in tensorflow. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 1–12.
[239]
Yao Xie, Melody Chen, David Kao, Ge Gao, and Xiang’Anthony’ Chen. 2020. CheXplain: Enabling physicians to explore and understand data-driven, AI-enabled medical imaging analysis. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.
[240]
Zhi Yang, Ziming Wang, Changwu Huang, and Xin Yao. 2023. An explainable feature selection approach for fair machine learning. In International Conference on Artificial Neural Networks. Springer, 75–86.
[241]
Qinghao Ye, Jun Xia, and Guang Yang. 2021. Explainable AI for COVID-19 CT classifiers: An initial comparison study. In 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS). IEEE, 521–526.
[242]
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. 2015. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 (2015).
[243]
Mohammad Zaeri-Amirani, Fatemeh Afghah, and Sajad Mousavi. 2018. A feature selection method based on shapley value to false alarm reduction in ICUs a genetic-algorithm approach. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 319–323.
[244]
Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European Conference on Computer Vision. Springer, 818–833.
[245]
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6720–6731.
[246]
Chanyuan Abigail Zhang, Soohyun Cho, and Miklos Vasarhelyi. 2022. Explainable artificial intelligence (XAI) in auditing. International Journal of Accounting Information Systems (2022), 100572.
[247]
Junzhe Zhang and Elias Bareinboim. 2018. Fairness in decision-making—the causal explanation formula. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 2037–2045.
[248]
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. 2018. Top-down neural attention by excitation backprop. International Journal of Computer Vision 126, 10 (2018), 1084–1102.
[249]
Ke Zhang, Jun Zhang, Pei-Dong Xu, Tianlu Gao, and David Wenzhong Gao. 2021. Explainable AI in deep reinforcement learning models for power system emergency control. IEEE Transactions on Computational Social Systems 9, 2 (2021), 419–427.
[250]
Quanshi Zhang, Ruiming Cao, Ying Nian Wu, and Song-Chun Zhu. 2017. Mining object parts from CNNs via active question-answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 346–355.
[251]
Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu, and Song-Chun Zhu. 2018. Interpreting CNN knowledge via an explanatory graph. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 4454–4463.
[252]
Quanshi Zhang, Ruiming Cao, Ying Nian Wu, and Song-Chun Zhu. 2017. Growing interpretable part graphs on convnets via multi-shot learning. In Proceedings of 31st the AAAI Conference on Artificial Intelligence. 2898–2906.
[253]
Quanshi Zhang, Jie Ren, Ge Huang, Ruiming Cao, Ying Nian Wu, and Song-Chun Zhu. 2020. Mining interpretable AOG representations from convolutional networks via active question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 11 (2020), 3949–3963.
[254]
Quanshi Zhang, Xin Wang, Ruiming Cao, Ying Nian Wu, Feng Shi, and Song-Chun Zhu. 2020. Extraction of an explanatory graph to interpret a CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 11 (2020), 3863–3877.
[255]
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. 2018. Interpretable convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8827–8836.
[256]
Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. 2019. Interpreting CNNs via decision trees. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6261–6270.
[257]
Yu Zhang, Peter Tiňo, Aleš Leonardis, and Ke Tang. 2021. A survey on neural network interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence 5, 5 (2021), 726–742.
Index Terms
- A Roadmap of Explainable Artificial Intelligence: Explain to Whom, When, What and How?
Recommendations
A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods
FAccT '22: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and TransparencyThe recent surge in publications related to explainable artificial intelligence (XAI) has led to an almost insurmountable wall if one wants to get started or stay up to date with XAI. For this reason, articles and reviews that present taxonomies of XAI ...
Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI
Highlights- We review concepts related to the explainability of AI methods (XAI).
- We comprehensive analyze the XAI literature organized in two taxonomies.
- We identify future research directions of the XAI field.
- We discuss potential ...
AbstractIn the last few years, Artificial Intelligence (AI) has achieved a notable momentum that, if harnessed appropriately, may deliver the best of expectations over many application sectors across the field. For this to occur shortly in Machine ...
Explainable Artificial Intelligence: Requirements for Explainability
SIGSIM-PADS '22: Proceedings of the 2022 ACM SIGSIM Conference on Principles of Advanced Discrete SimulationTo date, many reasons have been suggested for making explainable artificial intelligence (XAI) models. However, it is unclear when the XAI suggested content is considered an explanation. This paper conducts a survey to determine the requirements for ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
EISSN:1556-4703
Table of ContentsCopyright © 2024 Copyright held by the owner/author(s).
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Online AM: 05 November 2024
Accepted: 21 October 2024
Revised: 20 October 2024
Received: 30 September 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 192Total Downloads
- Downloads (Last 12 months)192
- Downloads (Last 6 weeks)192
Reflects downloads up to 10 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in