Fostering Data Worker Inclusion and Well-Being: Identifying Barriers and Designing Interventions
Pages 425 - 428
Abstract
In recent years, we have seen great advancements in artificial intelligence. Given this new “Age of AI”, an increase in annotated data for training advanced models requires increased support of a large, diverse, and robust population of workers. Fostering inclusivity and well-being within this space of data work must then be a top priority. In my thesis, I start with identifying barriers to inclusion and well-being across three groups of data workers with differing challenges. First, I study crowd workers whose disabilities encumber their ability to participate in data work. Next, I explore the effectiveness of a strategy for training crowd workers to complete complex tasks, as advanced AI models may elevate the required complexity of future data work. Lastly, I will briefly discuss a work-in-progress characterizing the productivity and well-being of a trained annotation team. Through these studies, I find a common shared barrier: a lack of shared knowledge of productive behaviors among workers. I conclude with future work involving designing interventions to mitigate this barrier.
References
[1]
Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77–91.
[2]
Rocío Calvo, Shaun K Kane, and Amy Hurst. 2014. Evaluating the accessibility of crowdsourcing tasks on Amazon’s mechanical turk. In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility. 257–258.
[3]
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. 2021. All That’s ‘Human’Is Not Gold: Evaluating Human Evaluation of Generated Text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 7282–7296.
[4]
Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on computer supported cooperative work. 1013–1022.
[5]
Claudia Flores-Saviaga, Yuwen Li, Benjamin Hanrahan, Jeffrey Bigham, and Saiph Savage. 2020. The challenges of crowd workers in rural and urban America. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8. 159–162.
[6]
Bill Gates. 2023. The Age of AI has begun. https://www.gatesnotes.com/The-Age-of-AI-Has-Begun
[7]
Young-Ho Kim, Eun Kyoung Choe, Bongshin Lee, and Jinwook Seo. 2019. Understanding personal productivity: How knowledge workers define, evaluate, and reflect on their productivity. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
[8]
Gary P Latham and Travor C Brown. 2006. The effect of learning vs. outcome goals on self-Efficacy, satisfaction and performance in an MBA program. Applied Psychology 55, 4 (2006), 606–623.
[9]
Angli Liu, Stephen Soderland, Jonathan Bragg, Christopher H. Lin, Xiao Ling, and Daniel S. Weld. 2016. Effective Crowd Annotation for Relation Extraction. In HLT-NAACL.
[10]
David Martin, Benjamin V Hanrahan, Jacki O’Neill, and Neha Gupta. 2014. Being a turker. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 224–235.
[11]
Abraham Harold Maslow. 1943. A theory of human motivation.Psychological review 50, 4 (1943), 370.
[12]
Tanushree Mitra, Clayton J Hutto, and Eric Gilbert. 2015. Comparing person-and process-centric strategies for obtaining quality data on amazon mechanical turk. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1345–1354.
[13]
Bernd Rechel, James Buchan, and Martin McKee. 2009. The impact of health facilities on healthcare workers’ well-being and performance. International journal of nursing studies 46, 7 (2009), 1025–1034.
[14]
Amy Rechkemmer and Ming Yin. 2020. Motivating Novice Crowd Workers through Goal Setting: An Investigation into the Effects on Complex Crowdsourcing Task Training. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8. 122–131.
[15]
Amy Rechkemmer and Ming Yin. 2022. Understanding the Microtask Crowdsourcing Experience for Workers with Disabilities: A Comparative View. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–30.
[16]
Scale. 2023. Creating large volumes of training data without sacrificing quality.https://scale.com/customers/toyota
[17]
Gerard H Seijts, Gary P Latham, Kevin Tasa, and Brandon W Latham. 2004. Goal setting and goal orientation: An integration of two different yet related literatures. Academy of management journal 47, 2 (2004), 227–239.
[18]
Neville Stanton, Paul M Salmon, and Laura A Rafferty. 2013. Human factors methods: a practical guide for engineering and design. Ashgate Publishing, Ltd.
[19]
Saiganesh Swaminathan, Kotaro Hara, and Jeffrey P Bigham. 2017. The crowd work accessibility problem. In Proceedings of the 14th Web for All Conference on The Future of Accessible Work. 1–4.
[20]
Toon W Taris and Paul JG Schreurs. 2009. Well-being and organizational performance: An organizational-level test of the happy-productive worker hypothesis. Work & Stress 23, 2 (2009), 120–136.
[21]
Antonio Torralba and Alexei A Efros. 2011. Unbiased look at dataset bias. In CVPR 2011. IEEE, 1521–1528.
[22]
Stephen Uzor, Jason T Jacques, John J Dudley, and Per Ola Kristensson. 2021. Investigating the Accessibility of Crowdwork Tasks on Mechanical Turk. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.
[23]
Haiyi Zhu, Steven P Dow, Robert E Kraut, and Aniket Kittur. 2014. Reviewing versus doing: learning and performance in crowd assessment. Conference on Computer Supported Cooperative Work (2014), 1445–1455.
[24]
Kathryn Zyskowski, Meredith Ringel Morris, Jeffrey P Bigham, Mary L Gray, and Shaun K Kane. 2015. Accessible crowdwork? Understanding the value in and challenge of microtask employment for people with disabilities. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing. 1682–1693.
Index Terms
- Fostering Data Worker Inclusion and Well-Being: Identifying Barriers and Designing Interventions
Recommendations
Co-worker transparency in a microtask marketplace
CSCW '13: Proceedings of the 2013 conference on Computer supported cooperative workWorkers in microtask work environments such as Mechanical Turk typically do not know if or how they fit into a workflow. The research question we posed here was whether displaying information about the number of other workers doing the same task would ...
Accessible Crowdwork?: Understanding the Value in and Challenge of Microtask Employment for People with Disabilities
CSCW '15: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social ComputingWe present the first formal study of crowdworkers who have disabilities via in-depth open-ended interviews of 17 people (disabled crowdworkers and job coaches for people with disabilities) and a survey of 631 adults with disabilities. Our findings ...
Quantifying the Invisible Labor in Crowd Work
CSCW2Crowdsourcing markets provide workers with a centralized place to find paid work. What may not be obvious at first glance is that, in addition to the work they do for pay, crowd workers also have to shoulder a variety of unpaid invisible labor in these ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2023
596 pages
ISBN:9798400701290
DOI:10.1145/3584931
Copyright © 2023 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 14 October 2023
Check for updates
Author Tags
Qualifiers
- Abstract
- Research
- Refereed limited
Conference
CSCW '23
Sponsor:
CSCW '23: Computer Supported Cooperative Work and Social Computing
October 14 - 18, 2023
MN, Minneapolis, USA
Acceptance Rates
Overall Acceptance Rate 2,235 of 8,521 submissions, 26%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 97Total Downloads
- Downloads (Last 12 months)97
- Downloads (Last 6 weeks)4
Reflects downloads up to 04 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format