thesis

Building Intelligent, Gaze-aware Programming Environment for Better Accessibility, Interactivity, and Productivity

Abstract

Background: Code comprehension is a cognitively demanding Software Engineering (SE) task. Studies report that developers can spend up to 60% of their working hours reading code. For instance, software engineers newly onboarding at Google can take up to five months to become familiar with a new codebase and stabilize their productivity. Meanwhile, eye-tracking (ET) has been applied in empirical SE studies for decades. While it has contributed to understanding programmers’ gaze behavior and cognitive activities to some extent, only a few studies have explored integrating gaze-based assistance at a fine-grained level into the programming environment as a form of support. In particular, there lacks the employment of the participatory design approach, which emphasizes the perspectives of end-users from the early stages. Furthermore, the effectiveness of gaze-driven assistance has seldom been investigated with code stumuli at scale in a real-world programming environment

Objectives: In this thesis, our objective is to: (1) understand the state of the art regarding how eye-tracking has been used in SE studies. (2) understand what machine learning techniques have been applied to gaze data in relation to software development. (3) gather data on current problems professional developer encounter and what type of tool support they want. (4) co-design the new tool support with programmers and develop design artifacts. (5) develop a proof-of-concept to evaluate whether gaze-based assistance can support programmers in onboarding new codebases.

Method: At a high level, we employed the Design Science Model for Empirical Software Engineering. This methodology consists of a problem–solution pair (horizontally) and a theory–practice pair (vertically). Our studies correspond to the Problem Construct (Paper I), Solution Construct (Paper II), Problem Instance (Paper III), Solution Instance (Paper IV), and Empirical Evaluation (Paper V) components, which map onto the quadrants of this methodology when visualized. Specifically, to understand how ET is used in SE research, we conducted a systematic mapping study including 71 papers. To examine how ML has been employed to leverage gaze data in SE, we synthesized findings from 10 papers. To identify problems faced by programmers, we surveyed 68 professional developers and academics with substantial industrial experience. To sketch solutions for the pinpointed problems, we organized a co-design workshop and subsequently conducted user testing with four and five programmers, respectively. Finally, to evaluate whether our proof-of-concept influences programmers’ reading and understanding of code at scale, we conducted a controlled experiment with 40 novices.

Resulsts: By consolidating studies in the areas of interest, we found that experiments were primarily conducted with toy code snippets and novice participants, and that ML techniques have mainly been used to predict programmer expertise and task difficulty in the context of program comprehension. Our survey revealed that code comprehension remains a major pain point among professional developers. Through co-designing solutions for this problem, we identified a promising opportunity for gaze-driven assistance, as programmers preferred it over voice-based support. Comparing the control and experimental groups to each other and to experts, we found that gaze-based assistance can influence programmers’ attention-distribution strategies, leading them to read fewer peripheral files and focus more effectively on relevant code.

Conclusion: In summary, our work comprises a series of systematic, evidence-based studies grounded in the theory of participatory design. It provides fine-grained evidence that both corroborates and extends findings regarding the use of gaze as a tool to support programmers. Moreover, it aligns with the growing importance of gaze as a communication medium for intelligent human–AI collaboration in future Software Engineering.

Included Papers

Supervisors

• Emma Söderberg
• Martin Höst
• Diederick C. Niehorster.

Opponent & Committee Members

• To be listed soon