Course meeting time and place:

Tuesdays 1:30 - 4:00 pm, GSB Room M107 (McClelland 107, across from/South of Arbuckle dining hall)

Instructors:

Jennifer L. Eberhardt
Email: jleberhardt@stanford.edu
Office Hours: Mondays 11:30 – 12 noon, Mondays 3:00- 3:30 pm, or by appointment

Dan Jurafsky
Email: jurafsky@stanford.edu
Class Office Hours: Tuesdays 4:00 – 5:00 pm, or by appointment

Teaching Partner:

Myra Cheng
Email: myra1@stanford.edu
Office Hours: Tuesdays 4:00 – 5:00 pm, or by appointment

Course Description:

Course Description: This course is an interdisciplinary practicum designed to get students with computational backgrounds and students with social science backgrounds to collaborate on building practical language-related systems that address societal issues related to race and inequity. Readings will be drawn broadly from across the social sciences and computer science. Students will work with large, complex datasets and participate in research involving community partnerships relevant to race and natural language processing. Prerequisite: Graduate standing and instructor permission required. Limited enrollment.

Course Requirements

Due Dates:

Course Topics:

Preparation, Attendance, and Participation:

It is important that you attend each session and complete the readings prior to class. The discussion and interaction during class time will be an integral part of the course.

Throughout, we will engage challenging questions about human behavior and society that intersect with our own lives. It is especially important that we all remain open to the expression of views that differ from our own and that we are willing to reconsider our own views. Maintaining humility about the rightness of one’s own views and appreciation for the insights of others are both essential to our being able to learn together. Of course, even as we challenge each other’s ideas and arguments, we want to maintain an atmosphere of respect and collegiality.

Reaction Papers:

You will be required to write two short reaction papers during the quarter. These papers should be approximately 2 pages (double spaced). The papers can be written for any week of your choosing, as long as they are posted to Canvas the day before the class meets to discuss the topic/readings about which you have written. These papers should not be descriptive summaries of the readings. Instead, they should offer a critical analysis. For example, you may choose to discuss a problem or limitation with one of the readings (or with several of the readings) and offer a better approach or method. You may propose a specific study to conduct. You may discuss a theme that seems to cut across all of the readings. You may propose a new theoretical framework for understanding a phenomenon discussed in the readings. There are many options. Regardless of the option you choose, you should strive to lay out a coherent, well-defended argument.

Discussion Questions:

Each week you will be asked to generate at least two discussion questions based on the readings. These questions will be factored into your participation grade. The questions will help us to understand common points of interest. And we will use them to help guide the discussion in class. Again, these questions should not be descriptive. Instead, they should be probing, analytical, and thought-provoking. Indeed, the quality of our discussions will critically depend on your contributions in this regard. Your questions should be posted on Canvas no later than 5 PM each Monday.

Discussion Leadership:

Once during the quarter, you will be asked to help lead the discussion for the week. You are expected to meet with your group partner(s) beforehand to agree on the questions and issues you will use to frame and guide the discussion. You should be prepared to point out big themes. We will rely on you to place the research in context. What is the value of the work? Why does this work matter to the fields of linguistics, computer science, psychology, organizational behavior, comparative studies, or to the public at large? Does the research have policy implications worth exploring? Bring in your own expertise on the issues to help inform the class and to push us beyond the readings.

Final Project:

The final project is a chance to apply NLP to a societal issue related to race. During the course of the quarter, we will discuss projects that developed from partnerships with leaders in industry, schools, and police departments (with whom we have built ties over the years) as well as repos of social media data. The projects you propose could involve using text data to conduct analytic studies of race and inequality in partner domains, testing NLP-powered interventions like content moderation or training, or building other practical tools. For areas where data requires IRBs or long-term contracting, the project can be a proof-of-concept proposal, with an implementation applied to pilot or sample data. The projects can be done individually or with a partner; if with a partner, we recommend a cross-disciplinary mix of computer scientists and social scientists. Project milestones include:

1. Proposal: Based on an initial investigation of community partner or other data, propose a high-level plan for an experimental study involving an NLP system
2. Rough Draft: A first draft of the project, including progress on a first study.
3. In-Class Presentation of Projects
4. Project Writeup

Assigned Readings

Week 1: September 24
Course Introduction, Student Introductions, Ice Breakers and a Taste of What’s Ahead

Week 2: October 1
The Transmission of Bias and the Mechanics of Inequality

• Wilkerson, I. (2020). America’s enduring caste system, New York Times Magazine, July 1.
  

• Richeson, J. A. (2020). Americans are determined to believe in Black progress: Whether it is happening or not. The Atlantic, September issue.
  

• Hofmann, V., Kalluri, P.R., Jurafsky, D., and King, S. 2024. AI generates covertly racist decisions about people based on their dialect. Nature, August. 2024.

• Anjalie Field, Su Lin Blodgett, Zeerak Waseem, and Yulia Tsvetkov. 2021. A Survey of Race, Racism, and Anti-Racism in NLP. Proceedings of ACL 2021.
  

• Dallas Card, Serina Chang, Chris Becker, Julia Mendelsohn, Rob Voigt, Leah Boustan, Ran Abramitzky, & Dan Jurafsky. 2022. Computational analysis of 140 years of US political speeches reveals more positive but increasingly polarized framing of immigration. Proceedings of the National Academy of Sciences, 119.

Week 3: October 8
How We Police

• Voigt, R., Camp, N. P., Prabhakaran,V., Hamilton, W. L., Hetey, R. C., Griffiths, C. M., Jurgens, D., Jurafsky, D., & Eberhardt, J. L. (2017). Language from police body camera footage shows racial disparities in officer respect. Proceedings of the National Academy of Sciences, 114, 6521-6526.

• Camp. N. P., Voigt, R., Hamedani, M. G., Jurafsky, D., Eberhardt, J. L. (2024). Leveraging body-worn camera footage to assess the effects of training on officer communication during traffic stops. PNAS Nexus 3 (9).

• Eugenia H. Rho, Maggie Harrington, Yuyang Zhong, Reid Pryzant, Nicholas P. Camp, Dan Jurafsky, and Jennifer L. Eberhardt. 2023. Escalated police stops of Black men are linguistically and psychologically distinct in their earliest moments. Proceedings of the National Academy of Sciences 120 (23).

• Caleb Ziems and Diyi Yang. 2021. To Protect and To Serve? Analyzing Entity-Centric Framing of Police Violence. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 957–976,

Week 4: October 15
How We Judge

• Spruill, Mikaela, & Lewis, Neil A., Jr. (2022). Legal descriptions of police officers affect how citizens judge them. Journal of Experimental Social Psychology, 101, 104306.

• Rickford, John R., and Sharese King. 2016. Language and linguistics on trial: Hearing Rachel Jeantel (and other vernacular speakers) in the courtroom and beyond. Language (2016): 948-988.

• Sharese King and Samantha Jacobs. 2024. Talk about testimony: courtroom dialogue as racialized interactions. Linguistics Vanguard.

• Shastri, Ishana, Shomik Jain, Barbara Engelhardt, and Ashia Wilson. 2024. "Automating Transparency Mechanisms in the Judicial System Using LLMs: Opportunities and Challenges." arXiv preprint arXiv:2408.08477 (2024).

Week 5: October 22
How We Teach

• Walton, G., Okonofua, J.A., Cunningham, K. R., Hurst, D., Pinedo, A., Weitz, E., Ospina, J. P., Tate, H., Eberhardt, J. L. (2021). Lifting the bar: A relationship-orienting intervention reduces recidivism among children reentering school from juvenile detention. Psychological Science, 32 (11), 1747-1767.

• Darling-Hammond, S., Ruiz, M., Eberhardt, J.L., & Okonofua, J. A. (2023). The dynamic nature of student discipline and discipline disparities. Proceedings of the National Academy of Sciences.

• Mei Tan, Dorottya Demszky (2023) Sit Down Now: How Teachers’ Language Reveals the Dynamics of Classroom Management Practices. Working Paper.

• Lucy Li, Camilla Griffiths, Claire Ying, JJ Kim-Ebio, Sabrina Baur, Sarah Levine, Jennifer L. Eberhardt, David Bamman, Dora Demszky. Racial and ethnic representation in literature taught in U. S. high schools. (under review).

Week 6: October 29
How We Work

• Haozhe An, Christabel Acquaye, Colin Wang, Zongxia Li, and Rachel Rudinger. 2024. Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender? Proceedings of ACL 2024, 386–397

• Kraus, M. W., Torrez, B., Park, J. W., and Ghayebi, F. (2019). Evidence for the reproduction of social class in brief speech. Proceedings of the National Academy of Sciences, 116, 17225-17230.

• Amit Haim, Alejandro Salinas, Julian Nyarko. 2024. What's in a Name? Auditing Large Language Models for Race and Gender Bias. Manuscript

• Lyons-Padilla, S. Markus, H. R., Monk, A. Radhakrishna, S., Shah, R., Dodson, N. A., & Eberhardt, J. L. (2019) Race influences professional investors’ financial judgments. Proceedings of the National Academy of Sciences, 116, 17225-17230.

Week 7: November 5 (NO CLASS – Democracy Day)

Week 8: November 12
How We Connect

• Lee, C., Gligorić, K. Kalluri, P. R., Harrington, M., Durmus, E., Sanchez, K. L., San, Y., Tes, D., Zhao, X., Hamedane, M. G., Markus, H. R., Jurafsky, D., and Eberhardt, J. L. (2024). People who share encounters with racism are silenced online by humans and machines, but a guideline-reframing intervention holds promise. Proceedings of the National Academy of Sciences.

• Shaikh, Omar, Valentino Emil Chai, Michele Gelfand, Diyi Yang, and Michael S. Bernstein. 2024. "Rehearsal: Simulating conflict to teach conflict resolution." In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-20.

• Argyle, Lisa P., Christopher A. Bail, Ethan C. Busby, Joshua R. Gubler, Thomas Howe, Christopher Rytting, Taylor Sorensen, and David Wingate. 2023. Leveraging AI for democratic discourse: Chat interventions can improve online political conversations at scale. Proceedings of the National Academy of Sciences 120, no. 41 (2023): e2311627120.

• Reddan, M.C., Garcia, S., Golarai, G., Eberhardt, J. L., Zaki, J. (2024). A film intervention increases understanding of formerly incarcerated people and support for criminal justice reform. Proceedings of the National Academy of Sciences.

Week 9: November 19
How We Advance

• Field, Anjalie, Amanda Coston, Nupoor Gandhi, Alexandra Chouldechova, Emily Putnam-Hornstein, David Steier, and Yulia Tsvetkov. "Examining risks of racial biases in NLP tools for child protective services." In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp. 1479-1492. 2023.

• Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, and David G. Robinson. 2020. Roles for Computing in Social Change. In Conference on Fairness, Accountability, and Transparency (FAT* ’20), January 27–30, 2020, Barcelona, Spain.

• Faiz Surani∗, Mirac Suzgun∗, Vyoma Raman, Christopher D. Manning, Peter Henderson, and Daniel E. Ho. 2024. AI for Scaling Legal Reform: Mapping and Redacting Racial Covenants in Santa Clara County. Draft.

• Blueprint for an AI Bill of Rights

November 26 (NO CLASS – Thanksgiving Recess)

Week 10: December 3
Presentations of Final Projects