2025-10-10: Six Years, Countless Experiments, One Framework: The Story of Multi-Eyes

By -
In 2019, I packed my bags and flew from Sri Lanka to Virginia to begin my Ph.D. in Computer Science at Old Dominion University. I did not have a clear roadmap or any prior research experience; all I had was the hope that I would be able to figure things out along the way. After six years, I found myself diving deep into eye-tracking, human-computer interaction, and machine learning; eventually completing my dissertation in multi-user eye-tracking using commodity cameras, with the support of my advisor, Dr. Sampath Jayarathna, NIRDS Lab, and ODU Web Science and Digital Libraries Research group. 
 

 
 
When I started my Ph.D. at ODU, I had limited knowledge and experience in eye tracking and computer vision research. After learning about ongoing research at the lab on cognitive load using eye tracking, I was fascinated by how we could use technology to better understand humans in terms of their intentions, focus, attention, and interactions with the world. That curiosity, combined with my liking for working with hardware, eventually led me to eye-tracking research.

Early on, I realized that most eye-tracking studies focused on single users, highly controlled environments, and expensive hardware. That works for lab studies, but the real world is messy, as we experienced during our first event participation, STEAM on Spectrum at VMASC. Our demo application for eye tracking was successful for a single user in the laboratory environment, but it did not perform well in the real world. Also, since we had only one eye tracker for the demo, only one person could experience eye tracking, while the others had to wait in line away from the tracker. These problems led us to question how we could enable two or more people to interact with an eye tracker while also measuring their joint attention, which a traditional eye tracker could not do. That was when the idea for Multi-Eyes started to take shape.

 

First, we started with the trivial approach of having a dedicated eye tracker for each user. It worked well until all the users started moving, which sometimes prevented the eye trackers from capturing valid eye tracking data, giving us incorrect values. Movement constraints and the high cost of eye trackers made the setup very expensive and difficult to use in real-life applications. It may be disadvantageous for eye tracking when participants are physically together. Still, it worked best when they were online, which we later published in CHIIR 2023, "DisETrac: Distributed Eye-tracking for Online Collaboration." 


 

Due to the limitations of this approach, mainly the need for a dedicated device for each participant, we attempted to create Multi-Eyes using low-cost, commodity cameras, such as webcams, thereby eliminating the need for specialized eye-tracking hardware. Although modern eye trackers made the process appear simple, there were numerous challenges to overcome when building Multi-Eyes.

The first challenge was developing a gaze estimation model that can identify where a person is looking in various environments, such as poorly lit rooms, different camera hardware, extreme head angles, and different facial features. To address this, we developed a gaze model that utilizes unsupervised domain adaptation techniques, providing robust gaze estimates across a wide range of environmental conditions. Additionally, we focused on achieving parameter efficiency through existing model architectures. We validated this through a series of experiments on publicly available gaze estimation datasets, with our approach and findings published in IEEE IRI 2024 (Multi-Eyes: A Framework for Multi-User Eye-Tracking using Webcameras), and IEEE IRI 2025 (Unsupervised Domain Adaptation for Appearance-based Gaze Estimation). 

Beyond gaze estimates, we had to solve the problem of mapping each user’s gaze direction onto a shared display, a commonly discussed scenario in multi-user interaction within human-computer interaction. The mapping process required transforming gaze information from the user coordinate frame into the display coordinate frame. We designed a simple yet effective learnable mapping function, eliminating the need for complex setup procedures. Our approach achieved on-screen gaze locations with horizontal and vertical gaze errors of 319 mm and 219 mm, respectively, using 9-point 9-sample calibration. Considering large shared displays, the error is sufficient and stable for gaze classification or coarse-grained gaze estimation tasks.

By combining these approaches, we developed a prototype application that can run at ~17 gaze samples per second on commodity hardware, without utilizing GPU acceleration or a specialized installation. We replicated an existing study in the literature using a setup that traditionally requires expensive hardware, demonstrating that Multi-Eyes could serve as a viable low-cost alternative. 

Throughout the Multi-Eyes project, we contributed to advancements in the field of eye tracking through conference presentations and publications. Notably, our review paper on eye tracking and pupillary measures helped us set the requirements for Multi-Eyes, which later received the Computer Science Editor’s Pick award. We first proposed the Multi-Eyes architecture at ETRA 2022 and then refined the approach, showcasing its feasibility at IEEE IRI 2024. Along with the papers, we also published our research on gaze estimation approaches, capsule-based gaze estimation at Augmented Human 2020, parameter-efficient gaze estimation at IEEE IRI 2024, and parameter-efficient gaze estimation with domain adaptation in IEEE IRI 2025. 

Beyond the main framework, Multi-Eyes sparked several spin-off projects. Our work, utilizing a dedicated eye tracker-based approach, resulted in published research in ACM CHIIR 2023, IEEE IRI 2023, and IJMDEM 2024. In addition, through my work with eye trackers, I contributed to several publications on visual search patterns, published in JCDL 2021, ETRA 2022, and ETRA 2025, as well as drone navigation, published in Augmented Humans 2023. 

 


 

Throughout my Ph.D., I also contributed to the research community by serving as a program committee member and a reviewer for conferences, including ACM ETRA, ACM CUI, ACM CIKM, ACM/IEEE JCDL, ACM CHI, and ACM CHIIR. In addition, I participated in various university events and summer programs, including ODU CS REU, STRS: Student ThinSat Research Camp, Trick or Research, Science Connection Day, and STEAM on Spectrum. 

Looking back, I’m grateful that my work has had a positive impact on the broader community by advancing research in eye tracking and making the technology more accessible. After a journey of over five years, I’m starting a new chapter as a lecturer at the Department of Computer Science at ODU. While teaching is my primary role, I plan to continue my research, exploring new directions in eye tracking and human-computer interaction. 

While I have documented most of my research findings, I am adding a few tips for myself, in case I ever happen to do it again or travel through time, which someone else might find helpful. 

  • Collaboration is key: Collaborators can bring together the missing pieces of the puzzle, offering fresh perspectives that may lead to new ideas. Additionally, they can serve as your free reviewer before rejection ;). 
  • Embrace rejections: Every 'no' is a part of the process, as all of my work comes from ideas that were initially rejected but later accepted after refinement.
  • Prototype early, fail fast: Building something tangible, even if it’s not perfect, helps you identify problems sooner and will aid in your next step. 
  • Document everything: A half-forgotten experiment is as good as lost. Notes and version control have saved me many times, especially when refining after a rejection. You will thank yourself for explaining why you have used that weird design or that random number. 

I am immensely grateful to my dissertation committee members and mentors: Dr. Sampath Jayarathna, Dr. Michael Nelson, Dr. Michele Weigle, Dr. Vikas Ashok, and Dr. Yusuke Yamani for their invaluable feedback, which greatly contributed to my success. I also owe my heartfelt thanks to my family, friends, and research collaborators, whose encouragement kept me going through the highs and lows of this journey. 

--Bhanuka (@mahanama94)

Comments

Post a Comment