Jeff Sauro and my friends at MeasuringU tested whether #ChatGPT can be used for #thematicanalysis of subjective user questionnaire responses. Answer: it does almost as well as human UX professionals, at least on the metric of inter-rater reliability. The mean agreement between ChatGPT and humans was .632 across three case studies, whereas it was .704 between the humans. We don’t know whether the humans were necessarily right and whether the small amount by which ChatGPT disagreed more with the humans than they disagreed among each other reflects *poorer* analysis by ChatGPT (my biased guess because I root for team human, being a member myself) or *better* AI-driven analysis (absolutely possible because grouping user comments into themes is a distinctly inaccurate art). Caveat: this doesn’t mean that ChatGPT can analyze user *behavior*, only that it is likely a time-saver in grouping non-behavioral questionnaire responses. (ChatGPT is known to be great at summarizing and classifying text without understanding what it means.) One more interesting finding from this study: it took the researchers several attempts to write an appropriate prompt to get good themes from ChatGPT. This again confirms that prompt-driven intent-based outcome specification has poor usability and isn’t for the faint of heart. Full writeup: https://lnkd.in/g4RNpGWE #userexperience #userresearch #artificialintelligence #ux #ai Follow me on LinkedIn for more updates like this: https://lnkd.in/gfffgNcr
It’s a bit of a click-baity title, no? The conclusions are sound but the title is so over-hyped most people will react instead of reading the dang article.
Canvs.ai and other large-scale text-based Qual analysis tools have had fairly fast sentiment coding & categorization available for a while. They become worth the ramp-up costs & learning curve if you deal with n=5000+ open ends per month in a habitual research cycle. Otherwise, you can actually templatize Excel files to format cells and output COUNTIF based on certain keywords and phrases.
Very interesting! I have also compared thematic analysis between human coders and GPT-4 and my Kappa coefficient was around .801 for GPT and humans, and .842 between humans. I used the themes gathered and applied PCA and other techniques to better understand how tokens would constitute themes that had a greater match between humans and GPT, and then compare it to some literature on UX Research and thematic analysis to understand if GPT model-training has an impact on bias. What I have found so far is that, given literature on how themes are recognized and noted by researchers, models need a constant training in order to not reduce then to what the model recognizes. This is interesting when research subjects are fairly newer than when the model was trained. If you ask researchers to do thematic analysis on people’s tweets when it comes to twitter privacy before and after Musk bought it, the same tokens and/or groups of them will bring about different codes, because chatGPT was trained up to the end of 2021 and doesn’t have trained data about musk owning twitter, at least by the date I conducted the experiments.
This is not replacing, this is helping with the heavy lifting
Gökhan Besen
1y
Great analysis! ChatGpt can be effectively used as a leverage to do the leg work for the them, so they can focus on the outcome. It will also help scale small research teams. As in other professions, i agree researchers who utilize chatgpt will replace researchers who don’t. It’s like researchers on steroids ;)
Siniša Šašić
1y
Excellent insights. “Replacement” is also quite a broad representation of what can happen in the interplay of AI and humans. Even though it’s not a smooth way to use such a powerful tool (yet), making most out of AI with proper prompts is still a considerably handy skill to have. I’m not sure if anybody should base their career on that skill, but perhaps it would be foolish to dismiss using and practicing it for the time being.
I hope not cause that is my fav part of the UX field!
Dean Peters
1y
Tools such as ChatGPT can serve as a 'coaching network' to help UX practitioners level up their decision making at the speed of change.
UX Designer | Mobile & Web | Google Certified UX Designer | Using user-centered designs to improve conversion rates by 10%
1yI believe the role of a UX researcher will be complemented by ChatGPT, not necessarily replaced.