thesis

Building Gaze-aware Programming Environments

Programming is a cognitively demanding exercise. Artificial Intelligence (AI) as a disruptive technology is redefining the practice of programming and transforming software engineering. As AI is evolving to a multimodal version that accommodates not only texts but also speech, images, and more, we see an opportunity to design eye-tracking based assistance to support programmers. Since AI has taken the heavy lifting of producing code, we speculate that programmers will read and understand a larger amount of code and thereby spend more of their time reading it. We deem this a promising problem domain where eye-tracking can be of assistance.

To explore this inquiry, we undertook two mapping studies to establish the problem and solution constructs. We then surveyed 68 professional developers to understand this representative cohort and gather concrete, situated problems from them. After that, we co-developed multiple versions of design artifacts with a variety of groups of programmers. Finally, we employed a mixed-methods approach, including a pre-experiment survey, a controlled experiment, and post-experiment interviews with 40 novice programmers, to evaluate the proof-of-concept GazePrinter.

From the first study, we found that eye-tracking so far is used mostly for education-oriented studies in the research community focused on software development. There is a need to bring it closer to practitioners. From the second study, we identify that the gaze data produced by eye trackers has been explored with a collection of machine learning techniques. However, these models were trained with small samples that might carry bias and insufficiency. Contemporary machine learning techniques may be able to compensate for that. From the survey, we learned that developers have already adopted AI assistance, and they are mostly positive about it despite room for greater accuracy and capability. As eye-tracking is relatively novel to them, most developers are unsure about how it can help them. From the design study, we realized that programmers are intrigued by gaze-based assistance in a programming environment. Lastly, from the evaluation study, we found that our designed intervention, GazePrinter, can nudge novice programmers to read code in a manner that more closely aligns with experts.

For future work, we invite research exploring adaptive, gaze-driven assistance and interactions in AI native environments for different programmers.

Thesis

Included Papers

Supervisors

• Associate Professor Emma Söderberg (main supervisor), Lund University
• Professor Martin Höst (co-supervisor), Malmö University
• Dr. Diederick C. Niehorster (co-supervisor), Humanities Lab, Lund University

Opponent

• Associate Professor Andrew Begel, Carnegie Mellon University, USA

Examination Committee Members

• Professor Yvonne Dittrich, IT University of Copenhagen, Denmark
• Professor Emilia Mendas, Aarhus University, Denmark
• Professor Veronica Sundstedt, Blekinge Institute of Technology, Sweden
• Dr. Sara Willhammar (deputy member), Lund University, Sweden