2026-07-17: Summer Research Internship at the Harvard Graduate School of Education
This summer, I had the opportunity to work as a visiting student at the Harvard Graduate School of Education (HGSE), where I worked in the Learning, Innovation, and Technology Lab (LIT Lab) supervised by Dr. Bertrand Schneider. The LIT Lab investigates how people learn and collaborate in learning environments. They use advanced sensing technologies and data-driven methods to measure attention, behavior, and interactions. HGSE is a leading institution dedicated to advancing educational research and innovation through interdisciplinary approaches that combine education, technology, and learning sciences. The internship provided an opportunity to explore new methods, tools, and measures used to study collaborative learning, while also contributing to a new research project.
My internship was an eight-week program, and during my visit I collaborated closely with Dr. Schneider on the project “Exploring the trade-offs between wearable eye-tracking and computer vision technologies for studying joint visual attention (JVA)”. We investigated the strengths and limitations of these approaches in real-world collaborative environments. I participated in regular one-on-one research meetings, which were collaborative development sessions where Dr. Bertrand and I shared progress, discussed findings, and set goals for the coming week. These interactions provided valuable opportunities to refine research methodologies, learn to look at the problem with new perspectives, and to understand the applications and contributions of the work in different domains. The experience allowed me to study how JVA is observed and used in learning science, meet new people and make connections, and explore new opportunities.
Project Overview
Our project explores how people share their attention while working together in real-world activities. There are various definitions of JVA in the literature, but for this study, we define JVA as two people looking at the same object at the same time. JVA can help us better understand collaboration, communication, learning, and social interactions.
Our study investigates the trade-offs between wearable eye tracking and computer vision for detecting JVA in collaborative settings. Wearable eye-tracking technology provides measurements of where individuals are looking from an egocentric point of view, but requires specialized equipment for data collection. In contrast, computer vision models estimate a person's visual attention using head orientation and other visual cues in an image or a video recording from the third-person view, offering a less intrusive and more scalable alternative. By comparing these two approaches across different collaborative tasks, the project aims to identify the situations in which computer vision provides sufficiently accurate estimates of JVA and those in which eye tracking remains necessary. The findings will help researchers choose the most appropriate method for studying human attention in real-world environments and support the development of more reliable and practical approaches for future research.
Approach
We collected data from pairs of participants while they worked together on different collaborative activities using both egocentric eye-tracking glasses and an exocentric video camera. The activities included reading together, building a small LEGO model, making simple electrical circuits, crafting, and some more tasks that naturally require people to share their attention on a third object or area. We varied several factors in the activity setting, including the number of objects involved in the task, the distance between the exocentric camera and the activity area, and the positions of the participants, to evaluate how these conditions affect the performance of the computer vision models.
During each activity, both participants wore Project Aria glasses, which recorded where each person was looking. The glasses captured egocentric (first-person) video and eye-tracking data. At the same time, a smartphone camera recorded the entire activity from a third-person view. This gave us two different views of the same interaction. We applied two state-of-the-art computer vision models, Sharingan: A Transformer Architecture for Multi-Person Gaze Following (Tafasca et al., 2024) and MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction (Gupta et al., 2024), to the third-person videos to estimate where each participant was looking and whether they were paying attention to the same object. We then compared these predictions with the eye-tracking data collected from the Project Aria glasses.
We compared the two methods by sampling video frames at corresponding timestamps covering the entire recording. For each sampled frame obtained from the eye-tracking sequence, we manually compared it with the predicted frame from both computer vision models. We analyzed cases where both eye-tracking and computer vision approaches identified JVA correctly, as well as cases where one method detected JVA while the other method could not, and we explored the factors affecting the failed cases. We were able to find instances where computer vision models performed well and instances where wearable eye tracking is still necessary. These findings will help researchers choose the best method for studying human attention in real-world collaborative environments based on the conditions of the activity.
Interesting Cases Identified During Manual Inspection
Comparing the computer vision model predictions with eye-tracking data revealed several interesting findings where the model predictions and eye-tracking data did not agree on detecting JVA. We identified the main reasons for the disagreements and organized them into the following categories:
Viewpoint Dependency
Computer vision models often estimate the gaze positions based on participants' head orientation. This works well in many cases, but it can fail when people move only their eyes without moving their head. Sometimes, when there is more than one object in the scene, the computer vision techniques may not provide a precise location of the gaze. Also, the objects of interest can be occluded in the exocentric view. These are some factors affecting incorrect predictions of attention in computer vision models. Figure 1 illustrates this limitation. From the third-person view, the Sharingan model predicts that both participants are looking at the same object. However, the egocentric views captured by the eye-tracking glasses reveal that each participant is actually attending to two different objects.
 |
| Figure 1: The Sharingan model predicts shared attention in the third-person view (left), whereas the egocentric views (middle and right) from the eye-tracking glasses reveal that the two participants are actually attending to different objects. |
Scene Complexity and Small Objects
The computer vision models perform well when there is one clear object of interest in the activity that everyone is attending to. But we identified some mispredictions when multiple objects are close together in the scene or when participants look at small objects. Figure 2 shows an activity with a single object, and computer vision models performed well in detecting JVA. Figure 3 shows an activity with small LEGO pieces in which the eye-tracking method performed better than the computer vision approach in detecting JVA.
 | ||
Figure 2: A less complex scene in which two individuals are watching a video on a laptop. Both the computer vision model and the eye-tracking data consistently identify the laptop as the shared object of attention.
|
Visibility and Occlusion
Performance of computer vision models decreases when faces are blocked by objects, people turn away from the camera, or the object being viewed is outside the exocentric camera's field of view. With less visual information, the models are more likely to predict the wrong gaze target. In Figure 4, the exocentric camera is placed too far from the activity, and it makes head detection and identifying head orientation difficult for computer vision techniques. Therefore, the attention predictions are incorrect when compared with the eye-tracking data.
 | ||
Figure 4: An example where the exocentric camera is positioned too far from the activity, resulting in inaccurate gaze predictions by the computer vision model. In contrast, the eye-tracking data reveal that each participant is attending to a different object held in their hands.
Figure 5 illustrates an instance where the face of one participant is blocked by the object in front. This leads to incorrect gaze predictions by computer vision models.
|
Tracking and Multi-person Association
Computer vision models struggle to maintain consistent tracking of participants when they move out of and later re-enter the camera's field of view, when heads overlap in the scene, or when temporary occlusions interrupt person tracking. In these cases, the models reassign participant identities after an occlusion event or after a participant leaves and later re-enters the camera's field of view, resulting in inconsistent identity assignments across frames. Figure 6 illustrates four video frames captured at different timestamps where participants were assigned different identities following these events. The changes are indicated by the change in head detection colors.
Although these identity switches do not directly affect JVA detection in our approach, since JVA estimation does not rely on persistent participant IDs, they reveal limitations in the temporal consistency of the tracking process. Such inconsistencies can affect participant-specific JVA analysis, longitudinal behavioral measurements, and applications that require maintaining a continuous identity for each individual throughout an interaction.
 |
| Figure 6: An example from a movement-intensive collaborative activity where frequent motion and changes in participant visibility challenge computer vision–based gaze-following models. As participants move out of the camera's field of view and later re-enter the scene, the models may assign them new identities instead of maintaining consistent tracking. |
Model Biases
Computer vision–based gaze-following models may exhibit inherent biases that influence their predictions in social interaction scenarios. For example, the MTGS model occasionally predicts that participants are looking at each other even when eye-tracking data indicate that their attention is directed elsewhere. Similarly, the model tends to infer shared attention toward prominent or commonly occurring objects in the scene, even when the actual gaze target differs. These biases can reduce accuracy in complex collaborative activities involving multiple objects, subtle gaze shifts, or task-specific attention. Figure 7 shows the tendency of the model to overestimate looking at humans. Figure 8 is an example of a misprediction due to a predefined model threshold.
 | ||
Figure 7: The MTGS model predicts that one participant is looking at the other person, even though her actual attention is directed toward the board.
|
What We Have Learned So Far
As a part of this ongoing work, we are currently expanding our manually annotated dataset to include a wider range of collaborative activities and interaction scenarios. Our preliminary findings reveal that computer vision models perform well in simple scenarios where participants share a clear, visible object of attention and their faces are easily seen by the exocentric camera. However, the prediction accuracy decreases in more complex situations involving multiple similar objects, occlusions, participant movement, or limited camera viewpoints. In comparison, wearable eye tracking provides more accurate measurements of where people are actually looking, making it better suited for studies that require object-level accuracy. However, this approach typically involves higher costs, requires participants to wear specialized eye-tracking devices, and can be more intrusive than camera-based methods, making it less practical for large-scale or naturalistic studies. Overall, our results indicate that the most appropriate method depends on the task and research goals, and that combining computer vision with eye tracking has strong potential for studying real-world collaborative interactions by leveraging the strengths of both approaches.
Experience at HGSE and LIT Lab
Working at the LIT Lab at HGSE was a great experience that contributed significantly to my professional and personal growth. The environment allowed me to interact with fellow students from diverse backgrounds, providing opportunities to exchange ideas and learn from their expertise. Everyone in the lab was welcoming, supportive, and always willing to help. Having access to the Innovation Studio further enriched my experience, and I learnt how advanced technologies, tools, and creative research spaces enhance interdisciplinary collaboration and hands-on experimentation. The experience strengthened my confidence as a researcher, expanded my professional network, and gave me new perspectives on conducting impactful human-centered research in collaborative environments. I’m sincerely grateful to Dr. Schneider for this opportunity and support throughout my internship.
 |
| LIT lab members and friends
|
Exploring Harvard and Boston
During my internship, I had the opportunity to explore Harvard University and experience the rich history, architecture, and vibrant atmosphere of the campus and its environs. I appreciated the historic buildings, beautiful surroundings, and the unique academic environment that has shaped generations of scholars. Beyond the university, I explored different popular areas around Boston, discovering the city's blend of historic landmarks and attractions. The following are a few snapshots from my visit.
Acknowledgments
I would like to express my gratitude to my advisor, Dr. Sampath Jayarathna, for taking the initiative to make this incredible internship opportunity at HGSE possible. I appreciate his continuous guidance, encouragement, and support throughout my academic journey. His mentorship has played a significant role in my academic and professional growth.
Kumushini Thennakoon
PhD Student | Department of Computer Science,
Old Dominion University, Norfolk, VA 23529
Email: kthen001@odu.edu
Web : https://www.cs.odu.edu/~cs_kthen001/
Comments
Post a Comment