2022-05-06: ACM International Conference on Intelligent User Interfaces (IUI) Trip Report

By -

Figure: Virtual cover for ACM IUI’22 conference


The premier showcase of outstanding research and development on intelligent user interfaces was hosted by the University of Helsinki, Finland. The 27th edition of the annual ACM Intelligent User Interface (IUI) conference was virtually held on March 21-25, 2022. There were 386 registered participants and 62 papers presented worldwide. For those who could not attend the conference, pre-recorded videos are available on the ACM SIGCHI YouTube channel, and the conference proceedings are available online.

This year, I was fortunate that one of our papers was accepted and represented our research group, Web Science and Digital Libraries Research Group (WSDL), at this conference. My paper, "InSupport: Proxy Interface for Enabling Efficient Non-Visual Interaction with Web Data Records" was co-authored by Dr. Sampath Jayarathna and Dr. Vikas Ashok from Old Dominion University, USA, and Hae-Na Lee from Stony Brook University, USA. This conference's presentations were pre-recorded and shown during the conference instead of presenting the papers submitted. After the presentation, there were two minutes given to each presentation for Q&A sessions with participants.

Day - 1

The first day of the conference began with five workshops and one Doctoral Consortium. I attended the “HEALTHI Workshop”, which offers a forum that brings academics and industry researchers together and seeks submissions broadly related to the design of intelligent user interfaces for promoting health. It builds on the fields of psychology, behavioral health, human-computer interaction, ubiquitous computing, and artificial intelligence. The workshop session covered intelligent user interfaces such as screens, wearables, voice assistants, and chatbots to support health, health behavior, and wellness in an accessible and equitable manner. The organizer for the session was Katrin Hänsel from Yale University, Michael Sobolev from Cornell Tech, Tobias Kowatsch from ETH Zurich, University of St.Gallen, and Rafael A Calvo from Imperial College London. This session has two keynotes: one from academia, presented by Dr. Timothy Bickmore from Northeastern University, and another from industry, presented by Steve Downs, a co-founder at Building H.
For the rest of the day, the conference arranged four workshops:
  1. APEx-UI Workshop
  2. HAI-GEN Workshop
  3. HUMANIZE Workshop
  4. TExSS Workshop

Day - 2

Health, Well-being, and Accessibility

This session was the first day of the paper session. In this session, I presented our work on behalf of my co-authors. Our work aims to minimize the usability gap between sighted and blind users to instant access in exploring important auxiliary segments of web browsers.



Min Lee (Carnegie Mellon University) began the session by presenting "Towards Efficient Annotations for a Human-AI Collaborative, Clinical Decision Support System: A Case Study on Physical Stroke Rehabilitation Assessment." He talked about how the development of ML-based decision support systems is challenging due to the expensive process of collecting an annotated dataset. Therefore, they propose a solution to augment an ML model with a rule-based (RB) model from therapists in assessing physical stroke rehabilitation exercises.



Many studies have revealed that mandala color benefits mental well-being. "Colorbo: Envisioned Mandala Coloring through Human-AI Collaboration," presented by Eunseo Kim from Hanyang University, shows selecting harmonious colors/areas and envisioning how each selection affects the final output.


In "The "Artificial" Colleague: Evaluation of Work Satisfaction in Collaboration with Non-human Coworkers," presented an online experiment where Shadan Sadeghian from the University of Siegen talked about the perception of work meaningfulness and relationship with the collaborator across different task distributions and collaborators.



Blind and Visually Impaired (BVI) individuals rely on captions to understand images; it takes more time to read for the more extended captions, but short captions may impair a user's ability to comprehend the image's content completely. Carlos A Aguirre, a Ph.D. student from Johns Hopkins University, presented "Crowdsourcing Thumbnail Captions Using Time-Constrained Methods," where he talked about how the time-constrained method effectively collects thumbnail captions while preserving the correctness of the captions.


Remote-sighted assistance (RSA) is popular among visual impairment users. Sooyeon Lee from Pennsylvania State University discussed the challenges that adversely affect the agent-user interaction in RSA services in "Opportunities for Human-AI Collaboration in Remote Sighted Assistance."


The final paper of this session was "FitNibble: A Field Study to Evaluate the Utility and Usability of Automatic Diet Monitoring in Food Journaling using an Eyeglasses-based Wearable" where a research scientist from Apple (Abdelkareem Bedri) presented a preliminary evaluation of automatic diet monitoring systems utility and usability using an end-to-end system.

Recommender Systems and Decision-Making

This session has seven papers. I didn't join this session. But I include all the presentations from the SIGCHI website. The chair of this session was Bart Knijnenburg from Clemson University.

Jeroen Ooge from KU Leuven presented "Explaining Recommendations in E-Learning, Effects on Adolescents' Trust".







Thi Ngoc Trang Tran from the Graz University of Technology presented “TiiS: Humanized Recommender Systems: State-of-the-Art and Research Issues”.



Wanqi Ma from Shenzhen University presented “TiiS: Holistic Transfer to Rank for Top-N Recommendation”.


Day - 3

Explainable AI (XAI) 1

The first session of the third day of the conference was Explainable AI, where most of the papers addressed the challenges of the spread of AI into the world outside of research labs that brought pressures and requirements. The chair of this session was Simone Stumpf from the University of Glasgow. Clarice Wang from The Harker School talked about how the challenges arise when humans interact with unbiased artificial intelligence in "Do Humans Prefer Debiased AI Algorithms? A Case Study in Career Recommendation". Most research in the AI and ML domain focuses on developing fair AI algorithms.


Lack of relevant knowledge results in people's inappropriate usage of machine learning technologies. In "Exploring the Effects of Machine Learning Literacy Interventions on Laypeople's Reliance on Machine Learning Models," Chun-Wei Chiang from Purdue University discussed an ML-assisted decision-making setting and conducted a human-subject randomized experiment to explore how providing different types of user tutorials as the machine learning literacy interventions can influence laypeople's reliance on ML models, on both in-distribution and out-of-distribution examples.
There is an urgent need to design effective methods to increase people's machine learning literacy, as the lack of relevant knowledge may result in people's inappropriate usage of machine learning technologies. In "Explaining Call Recommendations in Nursing Homes: A User-Centered Design Approach for Interacting with Knowledge-Based Health Decision Support Systems," Francisco Gutiérrez from KU Leuven presented a design about .recommender engine to support call suggestions that may help the safety and quality of care of people.

Machine Learning models can output confident but incorrect predictions. In "Deep Learning Uncertainty in Machine Teaching," Téo Sanchez, a Ph.D. student from Université Paris Saclay, investigated two types of uncertainty — aleatory and epistemic — to help non-expert users understand the strengths and weaknesses of a classifier in an interactive setting.

A person decides on which of several AI-powered sequential decision-making systems to rely on some choices. In "How Do People Rank Multiple Mutant Agents?" Jonathan Dodge described a novel explanation of sequential decision-making.

In "Investigating Explainability of Generative Models for Code through Scenario-based Design," Jiao Sun explored users' explainability needs for GenAI in three software engineering use cases: natural language to code, code translation, and code auto-completion.
Significant ethical and trust issues arise when a user does not adequately understand algorithmic and system processes. In "TiiS: Explaining conversational agents for use in criminal investigations," Sam Hepenstal from Middlesex University London demonstrated an AI system for intelligence analysis that tackles these issues.


Alternative Input Modes

The day's second session started with "Emotion Recognition in Conversations using Brain and Physiological Signal," presented by Nastaran Saffaryazdi from the Auckland Bioengineering Institute.


Lisa-Marie Vortmann talked about "Differentiating Endogenous and Exogenous Attention Shifts Based on Fixation-Related Potentials." In her talk, she presented that changes in visual-spatial attention are usually accompanied by eye movements and a fixation on the new center of attention.


In "Brainwave-Augmented Eye Tracker: High-Frequency SSVEPs Improves Camera-Based Eye Tracking Accuracy", presented by Alexandre Armengol-Urpi (MIT), the authors proposed a framework for visual attention to improve the accuracy of a commercial eye tracker through the analysis of electroencephalography (EEG) waves.



"Robust and Deployable Gesture Recognition for Smartwatches" presented by Utkarsh Kunwar, suggested that previous understanding of the causes behind everyday variability and false positives should be exploited in developing recognizers.


Specific valuable interaction cues, such as impromptu group polling, are less optimally executed when compared to in-person meetings because it is more challenging to gauge meeting participants who are not co-located, and it requires prior appointment set up to set up built-in polling tools for automated counting. In “Show of Hands: Leveraging Hand Gestural Cues in Virtual Meetings for Intelligent Impromptu Polling Interactions,” Jung In Koh presented a novel intelligent user interface approach for prevalent virtual meeting software, which allows for more seamlessly impromptu polling interactions similarly to in-person meetings.



"Hazard Notifications for Cyclists: Comparison of Awareness Message Modalities in a Mixed Reality Study" concentrated on possible collisions caused by car doors opening in a bicycle's movement route, resulting in catastrophic injury to the rider. This paper was presented by Tamara von Sawitzky from Johannes Kepler University.



The final paper for this session was "Developing Persona Analytics Towards Persona Science," presented by Bernard J Jansen from Hamad Bin Khalifa University. He talked about how users interact with data-driven personas.

Tools for AI Developers

Just like the other two sessions, seven papers were presented in this session. The session began with "GridBook: Natural Language Formulas for the Spreadsheet Grid", presented by Sruti Srinivasa Ragavan from Microsoft Research.



Generative source code using a machine learning model is an exciting area of research now. Justin D. Weisz from IBM research presented "Better Together? An Evaluation of AI-Supported Code Translation", which examined whether such imperfect outputs are helpful in Java-to-Python code translation.


"ODEN: Live Programming for Neural Network Architecture Editing" demonstrated an always-on visualization and the use of live programming approaches in neural network architecture editing. The paper was presented by Chunqi Zhao from The University of Tokyo.


"Expressive Communication: Evaluating Developments in Generative Models and Steering Interfaces for Music Creation". HCI researchers have focused on designing steering interfaces that support user control and ownership. In this paper, Ryan Louie presented a common framework of how developments in both models and user interfaces are essential for empowering co-creation.

Model designers frequently need to compare across many embedding spaces to describe model defects and pick an acceptable representation, a challenging analytical endeavor assisted by few current tools. Venkatesh Sivaraman, a Ph.D. student from Carnegie Mellon University, talked about "Emblaze: Illuminating Machine Learning Representations through Interactive Comparison of Embedding Spaces," a new technology that incorporates embedding space comparison into a computational notebook environment.


Gary Ang from Singapore Management University proposed a novel Heterogeneous Attention-based Multimodal Positional graph neural network model in "Learning User Interface Semantics from Heterogeneous Networks with Multimodal and Positional Attributes" that will demonstrate the UI semantics well for different UI annotation, search, and evaluation tasks.

All graphical user interfaces consist of one or more screens that may show the user depending on their interactions. Shirin Feiz from Stony Brook University presented "Understanding Screen Relationships from Screenshots of Smartphone Applications," where she talks about how it is difficult to deal with screen relationships as instances of the same screen may have visual and structural variation.

Day - 4

Mobiles and Wearables

The first session of day 4 of the conference began with "Estimating 3D Finger Pose via 2D-3D Fingerprint Matching", which was presented by Yongjie Duan from Tsinghua University. He talked about a finger-specific algorithm for estimating 3D finger pose, including roll, pitch, and yaw from fingerprint images.

Maozheng Zhao, a Ph.D. student from Stony Brook University, presented "EyeSayCorrect: Eye Gaze and Voice Based Hands-free Text Correction for Mobile Devices", an eye gaze and voice-based hands-free text correction method for mobile devices.


Modern in-cabin sensor technologies, particularly vision-based techniques, have significantly increased user interaction within the car, opening the path for new natural user interaction applications. By tracking the movements of eye-gaze, head, and finger, Abdul Rafey Aftab working in BMW Group, presented "Multimodal Fusion for Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle", a multimodal fusion architecture based on a deep neural network to precisely determine the driver's addressing purposes.
Due to existing constraints in the automobile, new interaction paradigms are required to enable productive office work. The car needed a wide range of non-driving functions for those who will drive self-driving vehicles in the future. Clemens Schartmüller from Johannes Kepler University introduced "Multimodal Error Correction for Speech-to-Text in a Mobile Office Automated Vehicle: Results From a Remote Study", and talked about a technique for determining whether the driver's desired item is inside or outside the vehicle based on the expected pointing direction.
Advanced data processing, including machine learning techniques necessary for gesture recognition. While these approaches can reach high gesture recognition accuracy, using artificial neural networks requires many gesture templates for training, and calibration is radar-specific. To address these challenges, Arthur Sluÿters presented "Hand Gesture Recognition for an Off-the-Shelf Radar by Electromagnetic Modeling and Inversion", a novel data processing pipeline for hand gesture recognition that combines advanced full-wave electromagnetic modeling (EM) and inversion with machine learning.

In "Mind-proofing Your Phone: Navigating the Digital Minefield with GreaseTerminator", Siddhartha Datta, a Ph.D. student from the University of Oxford, presented a framework named "GreaseTerminator" which enables researchers to quickly build, implement, and test various interventions against interface-based digital harms, establishing the ground truth about which interventions work for end-users.

Being able to assess habitual smartphone use, in particular automatically, might have different applications, e.g., to design better "digital wellbeing" solutions for mitigating meaningless traditional use. To close this gap, Alberto Monge Roffarello presented "TiiS: Understanding, Discovering, and Mitigating Habitual Smartphone Use in Young Adults", a data analytic methodology based on clustering and association rules mining to discover complex smartphone habits from mobile usage data automatically.

Interacting with Machine Learning

The first paper, "Building Trust in Interactive Machine Learning via User Contributed Interpretable Rules", talked about a user-centric evaluation framework to create a comprehensive structural model to understand how an explanation-driven interactive machine learning mechanism improves users' satisfaction with the machine learning system. Lijie Guo from Clemson University presented this work on behalf of the co-author. The chair of the second paper session of the day was Alison Renner from Dataminr.


Based on integration testing concepts in software development, Quan Ze Chen from the University of Washington presented "HINT: Integration Testing for AI-based features with Humans in the Loop", a crowd-based framework for testing AI-based experiences integrated with a humans-in-the-loop workflow.


The critical barrier to training a useful supervised model for many automated classification tasks is collecting labeled data. Interfaces for interactive labeling tighten the loop of labeled data collection and model development, enabling a subject-matter expert to quickly establish the feasibility of a classifier for addressing a problem of interest. In "Trade-offs in Sampling and Search for Early-stage Interactive Machine Learning", Zachary Levonian demonstrated a trade-off between improving a text classifier's performance and computing unbiased performance estimates.

The excellent performance of models poses whether it is possible to reduce further the effort required to label training data by adopting a human-in-the-loop interface. "Efficiently correcting machine learning: considering the role of example ordering in human-in-the-loop training of image classification models", presented by Geoff Holmes from the University of Waikato, looks at how to order the data in this iterative training scheme to achieve the highest model performance while minimizing the effort needed to correct misclassified examples.

In "TiiS: MI3: Machine-Initiated Intelligent Interaction for Interactive Classification and Data Reconstruction", Yu Zhang from the University of Oxford presented a novel technique for the machine to initiate intelligent interactions to reduce the user's interaction cost in interactive classification tasks.

When humans interact with intelligent systems, their causal responsibility for outcomes becomes equivocal. In "TiiS: Theoretical, Measured and Subjective Responsibility in Aided Decision Making", Joachim Meyer talks about a newly developed responsibility quantification model (ResQu) to predict actual human responsibility and perceptions of duty in the interaction with intelligent systems.
Understanding urban areas of interest (AOIs) is essential in many real-life scenarios, and such AOIs can be computed based on the geographic points that satisfy user queries. In "TiiS: AOI-shapes: supporting interactive visualization of user-defined urban areas of interest", Mingzhao Li from RMIT University showed a problem of efficient and effective visualization of user-defined urban AOIs interactively.

Learning and Playing

We came to the final paper session of the day named "Learning and Playing". The chair of this session was Oznur Alkan from IBM Research.

Apps like Robinhood and Webull have been increasingly popular among investors due to their ability to trade stocks, options, and other instruments. Non-expert investors who use these apps, on the other hand, frequently make poor investment judgments due to behavioral issues. In “Robinhood's Forest: An Idle Game to Improve Investor Behavior”, presented by Sayan Chaudhry from Carnegie Mellon University, Robinhood's Forest is an idle game that encourages better investing habits.


The key to decreasing the impact of disruptive occurrences is to optimize operations on critical infrastructure networks. In "Rather Solve the Problem from Scratch: Gamesploring Human-Machine Collaboration for Optimizing the Debris Collection Problem", Aybike Ulusan from Northeastern University discussed how humans and algorithms might collaborate to solve this complex problem.

Shiwali Mohan from Palo Alto Research Center talked about how intelligent collaborative agents that are human-aware can design adaptive coaching interactions to help people develop sustainable healthy behaviors in “TiiS: Exploring the Role of Common Model of Cognition in Designing Adaptive Coaching Interactions for Health Behavior Change”.

Microlearning is very popular now. However, there are some challenges to maintain, such as scheduled reminders. Fiona Draxler from LMU Munich presented "Agenda- and Activity-Based Triggers for Microlearning", where she addressed a schedule-based and an activity-based trigger for microlearning.

Free-form Game-Design (GD) environments can be challenging to some students due to their highly open-ended nature. Özge Nilay Yalçın from the University of British Columbia expressed a long-term goal to ease challenges with agents that can monitor the student's engagement with the environment in "An Intelligent Pedagogical Agent to Foster Computational Thinking in Open-Ended Game Design Activities".

Numerous people use commercial software that requires instructions to make tutorial videos. However, this software necessitates frequent back-and-forth between the two, incurring cognitive overhead. Saelyne Yan, a Ph.D. student from KAIST, presented "SoftVideo: Improving the Learning Experience of Software Tutorial Videos with Collective Interaction Data". This prototype system assists users in planning ahead of time before watching each step in tutorial films and providing feedback and assistance to users on their progress.

In large-scale classes, it is difficult to identify with those who require help in the class during lectures. “TiiS: A Real-time Interactive Visualizer for Large Classroom” was presented by Ujjwal Biswas from the Indian Institute of Technology Guwahati, where he addressed an idea to handle this challenge by using a two-level visualizer.

Day - 5

The conference's final day has three paper sessions and a closing session. Each paper session has seven paper presentations.

Applications and Tools

Yining Cao from the University of California, San Diego, presented “VideoSticker: A Tool for Active Viewing and Visual Note-taking from Videos”.


Ryoichi Shibata from Keio University presented “Utilizing Core-Query for Context-Sensitive Ad Generation Based on Dialogue”.

Explainable AI (XAI) 2



Jonathan Dodge from Oregon State University presented “TiiS: After-Action Review for AI (AAR/AI)”.


Natural Language















Closing & Keynote

The closing session briefly presented the conference's organization and announced the 2023 conference. Simone Stumpf, one of the program chairs, thanked all the participants and the attendees who had made the conference possible. On behalf of the other two chairs (Tuukka Ruotsalo, Krzysztof Gajos), Simone Stumpf thanked all the chairs, reviewers, student volunteers, and sponsors for their contributions. Stuart Russell gave the final keynote.

This year 62 papers were presented, and 14 Transactions on Interactive Intelligent Systems (TiiS) papers were presented; three keynote sessions were held (Ana Paiva, Munmun De Choudhury, Stuart Russell) during the conference.

This year two papers won “Best Paper”:
  1. Deep Learning Uncertainty in Machine Teaching (Téo Sanchez, Baptiste Caramiaux, Pierre Thiel, Wendy E. Mackay)
  2. Hand Gesture Recognition for an Off-the-Shelf Radar by Electromagnetic Modeling and Inversion (Arthur Sluÿters, Sébastien Lambot, Jean Vanderdonckt)
Three papers received “Honorable mention”:
  1. Embedding Comparator: Visualizing Differences in Global Structure and Local Neighborhoods via Small Multiples (Angie Boggust, Brandon Carter, Arvind Satyanarayan)
  2. Estimating 3D Finger Pose via 2D-3D Fingerprint Matching (Yongjie Duan, Ke He, Jianjiang Feng, Jiwen Lu, Jie Zhou)
  3. Learning User Interface Semantics from Heterogeneous Networks with Multimodal and Positional Attributes (Gary Ang, Ee Peng Lim)
The best poster and demo:
  1. The Diversity of Music Recommender Systems" by Ian Baracskay, Donald J Baracskay III, Mehtab Iqbal, Bart Knijnenburg
After this, the chair said goodbye to IUI 2022 and welcomed ACM IUI 2023. ACM IUI 2023 will be held in Sydney, Australia, from March 27th to 31st, 2023. The organizers are now planning for a five-day face-to-face event.


Figure: Image taken from ACM IUI’22 live YouTube streaming



Acknowledgment

I want to express my gratitude to Dr. Michael Nelson for his assistance in reviewing this blog article.


–– Md Javedul Ferdous (@jaf_ferdous)


Comments

Post a Comment