thesis

Building Intelligent, Gaze-aware Programming Environments for Better Accessibility, Interactivity, and Productivity

Abstract

Background: Code comprehension is a cognitively demanding Software Engineering (SE) task. Studies report that developers can spend up to 60% of their working hours reading code. For instance, software engineers onboarding at Google can take up to five months to become familiar with a new codebase and stabilize their productivity. Meanwhile, eye-tracking (ET) has been employed for empirical SE studies for decades. While it has contributed to understanding programmers’ gaze behavior and cognitive activities to some extent, only a few studies have explored integrating fine-grained gaze-based assistance directly into the programming environment. In particular, there is a lack of participatory design approaches, which emphasize incorporating end-users’ perspectives from the earliest stages. Furthermore, the effectiveness of gaze-driven assistance has rarely been examined with code stumuli at scale in real-world programming environments.

Objectives: In this thesis, our objectives are to: (1) understand the state of the art regarding how eye-tracking has been used in SE studies. (2) assess what machine learning techniques have been applied to gaze data in the context of software development. (3) gather data on current problems professional developer encounter and what types of tool support they desire. (4) co-design new tool support with programmers and develop design artifacts. (5) develop a proof-of-concept to evaluate whether gaze-based assistance can support programmers during onboarding new codebases.

Method: At a high level, we employed the Design Science Model for Empirical Software Engineering. This methodology consists of a problem–solution pair (horizontally) and a theory–practice pair (vertically). Our studies correspond to the Problem Construct (Paper I), Solution Construct (Paper II), Problem Instance (Paper III), Solution Instance (Paper IV), and Empirical Evaluation (Paper V) components, which map onto the quadrants of this methodology when visualized. Specifically, to understand how ET is used in SE research, we conducted a systematic mapping study including 71 papers. To examine how ML has been applied to leverage gaze data in SE, we synthesized findings from 10 papers. To identify problems faced by programmers, we surveyed 68 professional developers and academics with substantial industrial experience. To sketch solutions for the pinpointed problems, we organized a co-design workshop and subsequently conducted user testing with four and five programmers, respectively. Finally, to evaluate whether our proof-of-concept enhances programmers’ reading and understanding of code at scale, we conducted a controlled experiment with 40 novices.

Results: By consolidating studies in the areas of interest, we found that experiments were primarily conducted with toy code snippets and novice participants, and that ML techniques have mainly been used to predict programmer expertise and task difficulty in program comprehension contexts. Our survey revealed that code comprehension remains a major pain point among professional developers. Through co-designing solutions for this problem, we identified a promising opportunity for gaze-driven assistance, as programmers preferred it over voice-based support. Comparing the control and experimental groups to each other and to experts, we found that gaze-based assistance can influence programmers’ attention-distribution strategies, leading them to read fewer peripheral files and focus more effectively on relevant code.

Conclusion: In summary, our work comprises a series of mixed-methods, evidence-based studies grounded in the theory of participatory design. It provides fine-grained evidence that both corroborates and extends findings regarding the use of gaze as a tool to support programmers. Moreover, it aligns with the growing importance of gaze as a communication medium for intelligent human–AI collaboration in future Software Engineering.

Included Papers

Supervisors

• Emma Söderberg (main supervisor)
• Martin Höst (co-supervisor)
• Diederick C. Niehorster (co-supervisor)

Opponent & Committee Members

• To be listed soon