2022-07-17: Research Capacity-Building Planning Grant Awarded by NSF CISE Minority-Serving Institutions Program Grant

We are excited to receive 2022 National Science Foundation (NSF) Computer and Information Science and Engineering Minority-Serving Institutions (CISE-MSI) Research Expansion Program award worth $400,000 for two years for collaborative research towards "Understanding Human Emotions Associated with Attention Activities in Temporal Domain Utilizing Multimodal Data.” 

The goal of the CISE-MSI program is to broaden participation by increasing the number of CISE-funded research projects from MSIs and to develop research capacity toward successful submissions to core CISE programs. MSIs are central to inclusive excellence: they foster innovation, cultivate current and future undergraduate and graduate computer and information science and engineering talent, and bolster long-term U.S. competitiveness. MSI designated schools in this program are Historically Black Colleges and Universities (HBCUs), Hispanic-Serving Institutions (HSIs), and Tribal College & Universities (TCUs). 

This proposed research partnership with Old Dominion University (Carnegie R1 Very High Research University) and Bowie State University (HBCU) will establish a synergistic advancement in initiating collaborative research projects by building student-engaged research activities, improving student research skills, and expanding the faculty-lead research activities.

Figure 1: Dr. Jayarathna presenting eye tracking research to high school visitors. 

Emotions impact human behaviors because they are attached to our daily lives closely. Thus, understanding emotions have considerable effect on cognitive functions, as they are closely linked to learning, decision-making, and memory. Since the importance of recognizing emotions has been gradually increased, identifying emotions have been studied in various domains such as therapy, audience understanding, and clinical. In neurodevelopmental disorder clinical domains, emotion analysis has been studied to help the individuals with attention-deficit/hyperactivity disorder (ADHD). People with ADHD have difficulty attending to important details, planning task completion, modulating responses, and complex learning. Understanding emotions considering the changes of emotions over time with young adults with ADHD has not been fully studied in the past. The benefits of this research are building research capacity at a minority serving institution such as Bowie State University, by preparing student researchers to collaborate with faculty on cutting edge research such as Old Dominion University. Also, the research will provide an interdisciplinary solution in understanding and adjusting emotional behaviors to enhance human performances.

This research involves designing a forecast model that can predict emotions for the near future by examining the emotions associated with attention activities over time analyzing multimodal data such as eye-tracking, electroencephalogram, skin temperature, galvanic skin response, and oxygen saturation. The project will investigate stream-based multimodal data acquisition using a scientific workflow system that leverages both metadata standards and visual programming. Next, it will collect data from young adults with ADHD to generate the multimodal metrics during an emotionality analysis task and extract emotion-attention features associated with attention activities. Subsequently, the research will focus on generating a forecast model to predict emotions to enhance students' learning in data science. Furthermore, the project results can be used to monitor emotion changes for various applications such as attention training, interventions in social-emotional adjustments, and classroom-based learning. Moreover, the success of this project can contribute to developing real-time emotion monitoring utilizing wearable sensor devices.

Research Team:

Dr. Soo-Yeon Ji (PI) is an Associate Professor in the Department of Computer Science at Bowie State University. Her research specialty is data analytics including emotion detection, cyber attack detection, risk attack estimation, computer-aided decision making for health and medicine applications, and visual analytics.

Dr. Sampath Jayarathna (Co-PI) is an Assistant Professor of Computer Science at Old Dominion University. He is a PI of multiple active NSF awards including 2021 CAREER Award, S&CC Program Award, and REU Site Program Award. He has been actively performing research in the areas of machine learning, eye tracking, and multimodal sensing.

Dr. Anne M. Perrotti (Co-PI) is an Associate Professor of Communication Disorders and Special Education at Old Dominion University. Her research uses an inter-professional approach and design thinking for the design and development of innovative solutions which improve outcomes for underserved and special student populations. Specifically, her work has focused on understanding and objectifying working memory capacity using eye tracking technology for young adults with and without ADHD.

Dr. Katrina Kardiasmenos (Co-PI) is a professor of the Psychology Department at Bowie State University. Her research addresses all forms of executive functioning, with a specific emphasis on deficits in attention. Specifically, her research focuses on the effects of ADHD on learning in college students.

-- Sampath (@openmaze)