Discovering the syntax and strategies of natural language programming with generative language models
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2022•dl.acm.org
In this paper, we present a natural language code synthesis tool, GenLine, backed by 1) a
large generative language model and 2) a set of task-specific prompts that create or change
code. To understand the user experience of natural language code synthesis with these new
types of models, we conducted a user study in which participants applied GenLine to two
programming tasks. Our results indicate that while natural language code synthesis can
sometimes provide a magical experience, participants still faced challenges. In particular …
large generative language model and 2) a set of task-specific prompts that create or change
code. To understand the user experience of natural language code synthesis with these new
types of models, we conducted a user study in which participants applied GenLine to two
programming tasks. Our results indicate that while natural language code synthesis can
sometimes provide a magical experience, participants still faced challenges. In particular …
In this paper, we present a natural language code synthesis tool, GenLine, backed by 1) a large generative language model and 2) a set of task-specific prompts that create or change code. To understand the user experience of natural language code synthesis with these new types of models, we conducted a user study in which participants applied GenLine to two programming tasks. Our results indicate that while natural language code synthesis can sometimes provide a magical experience, participants still faced challenges. In particular, participants felt that they needed to learn the model’s “syntax,” despite their input being natural language. Participants also struggled to form an accurate mental model of the types of requests the model can reliably translate and developed a set of strategies to debug model input. From these findings, we discuss design implications for future natural language code synthesis tools built using large generative language models.