2019-06-19: Use of Cognitive Memory to Improve the Accessibility of Digital Collections

Eye Tracking Scenario
(source - https://imotions.com/blog/eye-tracking-work/)

Since I joined ODU, I have been working with eye tracking data recorded when completing a Working Memory Capacity (WMC) measure to predict a diagnosis of Attention-Deficit/Hyperactivity Disorder (ADHD). People with ADHD could be restless and hyperactive with distinct behavioral symptoms such as difficulty in paying attention and controlling impulsive behaviors. WM is a cognitive system, which makes it possible for human  to hold and manipulate information simultaneously. Greater WMC means greater ability to use attention to avoid distraction. Theoretically, adults with ADHD have reduced working memory when compared with their peers, demonstrating significant differences in WMC.

Among many tasks (O-Span, R-Span, N-Back) to measure the WMC, the reading span task (R-Span) is used as a valid measure of working memory yielding a WMC score. In R-Span, participants are asked to read a sentence and letter they see on a computer screen. Sentences are presented in varying sets of 2-5 sentences. Participants are asked to judge sentence coherency by saying 'yes' or 'no' at the end of each sentence. Then, participants are asked to remember the letter printed at the end of the sentence. After a 2-5 sentence set, participants are asked to recall all the letters they can remember from that set. R-Span scores are generated based on the number of letters accurately recalled, divided by the total number of possible letters recalled in order. This task represents a person’s ability to hold and manipulate information simultaneously.

We investigated eye gaze metrics collected during this task to differentiate the performance of adults with and without ADHD. This was important as it reveals an important eye movements feature differences between atypical and complex attention systems. The precise measurements of eye movements during cognitively demanding tasks provide a window into underlying brain systems affected by ADHD or other learning disabilities.

Fig 1: Comparison of Eye Fixations for ADHD (Left) and Non-ADHD (Right) participant during WMC Task (source -https://www.igi-global.com/chapter/predicting-adhd-using-eye-gaze-metrics-indexing-working-memory-capacity/227272)

We chose standard information retrieval evaluation metrics such as  precision, recall, f-measure, and accuracy to evaluate our work. We developed three detailed saccades (rapid changes of gaze) and fixation feature sets. Saccades are eye movements used to jump rapidly from one point to another. Fixations are the times which our eyes stop scanning and hold the vision in place to process what is being looked at. Feature includes the qualifiers: gender, number of fixations, fixation duration measured in milliseconds, average fixation duration in milliseconds, fixation standard deviation in milliseconds, pupil diameter left, pupil diameter right, and diagnosis label or class. The three feature sets categorized according to metric type:

1) fixation feature set

2) saccade feature set

3) saccade and fixation combination feature set

Fig 2: Classification of Eye Saccade Features during WMC (source - https://www.igi-global.com/chapter/predicting-adhd-using-eye-gaze-metrics-indexing-working-memory-capacity/227272)

Fig 3: Classification of Eye Fixation and Saccade Features during WMC (source - https://www.igi-global.com/chapter/predicting-adhd-using-eye-gaze-metrics-indexing-working-memory-capacity/227272)

The purpose of our research is to determine if eye gaze patterns during a WMC task would help us create an objective measuring system to differentiate a diagnosis of ADHD for adults. We identified six of the top performing classifiers for each of the three feature sets: J48, LMT, RandomForest, REPTree, K Star, and Bagging. While fixation features, saccade features, and a combination of saccade and fixation features accurately predicted the classification of ADHD with an accuracy of greater than 78%, saccade features were the best predictors with an accuracy of 91%. 

We published our work at IGI Global book chapter.

Anne M. P. Michalek, *Gavindya Jayawardena, and Sampath Jayarathna. "Predicting ADHD Using Eye Gaze Metrics Indexing Working Memory Capacity", Computational Models for Biomedical Reasoning and Problem Solving, IGI Global, pp. 66-88. 2019

An extended version of the paper is published at arXiv that elaborates more on the use of area of interest (AOI) during the ADHD diagnosis with eye tracking measures.

Use of Working Memory Capacity in the Wild...

Research shows that learning disabilities may be present in one's life either from birth or develop later in life due to dementia or injuries. Regardless of their declined cognitive abilities, people are interested in learning new things. For instance, older adults love to read books and learn new things after retirement to make use of their free time. But, physical disabilities and declined cognitive abilities might restrict people from accessing library materials. The Library of Congress Digital Collection is an excellent place for people to do their research as all they need is a computer and an internet connection. Therefore it is essential to make these public digital collection accessible.

Fig 4: The Library Of Congress Digital Collection Home Page.

Most of the times, web developers focus on regular users, and tend to forget how to cater to all types of users. Digital Collections requires careful considerations for the web UI to make it accessible, and we believe, based on our eye tracking research on WMC, we can help content creators such as Library of Congress Digital Collections achieve that. 
In Dr. Jayarathna's HCI course, I learned, to understand people, to be careful of different perspectives, and to design for clarity, and consistency. But as you can see, the application of these rules may differ with requirements. Since we predicted a diagnosis of ADHD with an accuracy greater than 78% using eye gaze data, there is a potential where we could identify people with and without declined cognitive abilities. This allow us to dynamically determine the complexity of the attentional system (whether typical or complex) of users and provide variations of the UI (similar to how a language translator works, click of a button to change the UI to be accessible). 

In the Future...

Consider an example scenario when a person with ADHD views the content of the Library of Congress Digital Collection. With a click of a button, web UI can change the presentation of the content. If the person has ADHD or some other learning disability, the content could be arranged in a different layout which allows the user to interact with it differently. 

Expanding on our results, we set our goal to explore how can we generalize our study to improve content accessibility for the people with learning disabilities without overloading their cognitive memory. We plan to use the Library of Congress or other similar online platform to start our exploration.

There is a real opportunity for us to help content creators of digital collection to be make these collections accessible for people regardless of their cognitive abilities.

-- Gavindya Jayawardena