2023-11-17: Taming User-Interface Heterogeneity with Uniform Overlays for Blind Users

 

Figure 1: Example of AccessBolt with File Explorer application. (Figure 1 from Utku et al.)

Introduction

User interface (UI) automation is the process of programmatically controlling and testing the UI of applications. In our last blog on UI automation, we talk about the ideas of using it and walkthrough of these interesting tools using the Python programming language. The main purposes of UI automation are to simulate user interactions for testing, increase test coverage, and reduce repetitive manual testing. In this blog, we are going to talk about a recently published paper named “Taming User-Interface Heterogeneity with Uniform Overlays for Blind Users,” published at the UMAP 2022 conference, where Utku et al. explored how to improve the accessibility of desktop application interfaces for blind individuals who rely on screen readers. The paper introduces a novel system called AccessBolt that aims to make interacting with desktop software easier and more efficient using UI automation for blind users (accessing option in desktop applications is shown in Figure 1). The dataset and all the supplementary material for the project are available online.

The Challenges Faced by Blind Screen Reader Users

Today, most computer applications have graphical user interfaces (GUIs) predominantly designed for sighted users. These GUIs have controls like menus, buttons, and options in a two-dimensional visual format. However, blind users who rely on screen readers can only access content sequentially in a linear, one-dimensional manner. This makes navigating and understanding the overall interface layout extremely difficult and tedious.

Blind users must memorize different keyboard shortcuts and navigation strategies corresponding to each application. For example, the shortcuts needed to access Microsoft Word options differ from those in Adobe Photoshop. This imposes a heavy cognitive burden on blind users. Furthermore, exploring an unfamiliar interface using repetitive screen reader key presses can be extremely frustrating. As a result, blind users take much longer to accomplish common tasks in computer applications than sighted individuals.

Prior Accessibility Solutions and Their Limitations

Previous research has tried to tackle challenges through various approaches—some solutions involved using commands, interactive interfaces, and novel input modalities like tactile devices. However, most of these focused largely on improving web accessibility. Very few works have looked at enhancing desktop application usability for blind users in a generalizable manner.

The existing approaches also need to fully address core issues like eliminating shortcut memorization and simplifying navigation of interface layouts. Many require additional expensive hardware devices. Others support only proprietary assistive technologies and have compatibility issues. So, there is a need for an efficient, cost-effective, and interoperable solution that can work across diverse desktop applications.

The AccessBolt System and How It Works

To overcome the limitations of prior tools, we designed a new system called AccessBolt that can make interacting with any desktop application significantly easier for blind individuals. It has an automated technique to extract all the controls and commands from an application's interface. It utilizes the accessibility APIs built into modern operating systems that expose the underlying interface components of software. AccessBolt programmatically analyzes the interface tree and identifies functional controls using machine learning algorithms.

It then provides the identified controls via a uniform popup overlay interface. This overlay acts as a centralized dashboard for accessing all commands and options regardless of their original location in the application layout. Users can simply navigate between choices linearly using arrow keys or tab without memorizing any application-specific shortcuts.

AccessBolt also enables users to search quickly for commands by name. Whenever the user selects an option in AccessBolt's overlay, it automatically triggers the action on the actual control in the application in the background. This preserves context and enables seamless transitions between the overlay and application.

Evaluation of AccessBolt System

We evaluated AccessBolt via a user study involving 12 blind participants. The participants had to complete typical tasks in desktop applications under two conditions:

• Using only their preferred screen reader (baseline)
• Using AccessBolt overlay along with the screen reader

The study showed that with AccessBolt, participants could complete the tasks with 40% fewer keystrokes on average compared to the baseline condition (Table 1). AccessBolt reduced the amount of repetitive navigation substantially (Baseline 178.35, whereas AccessBolt 72.18). The users also rated their perceived workload nearly four times lower with AccessBolt.

Table 1: Average number of key presses for baseline and AccessBolt conditions (Table 5 from Utku et al.).

In qualitative feedback, the participants praised AccessBolt's simplicity and minimal learning required to use it effectively. They appreciated how it made accessing application commands much quicker than brute-force exploration with screen readers. The study empirically demonstrates AccessBolt's benefits in enhancing desktop computing accessibility.

Summary and future work

This work highlights how intelligent, universally designed interfaces can make computers more usable for those with disabilities. AccessBolt overcomes the disparity faced by blind individuals when interacting with visual interfaces. Its model of automatically extracting interface components and delivering them via consistent alternative modalities can potentially transform accessibility.

The study also offers insights into minimizing the gulf of execution between user goals and interface actions. By reducing the cognitive load and physical effort required, AccessBolt enables blind users to achieve their objectives much more efficiently. The overlay concept demonstrates one way to bridge the gap between 2D visual and 1D non-visual interfaces.

AccessBolt effectively decouples the front-end presentation from the back-end functionality. This approach aligns with the W3C's accessibility principles of perceivable, operable, understandable, and robust interfaces. It underscores the need for customizable user experiences that cater to diverse needs and abilities.

In future work, we express exploring these scopes:

• Design of future inclusive interfaces and assistive technologies.
• With intelligent human-centered design, leverage technology by creating computing experiences that empower those with disabilities rather than hamper them.
• Plan to improve AccessBolt's capabilities further and conduct more extensive user evaluations.

Acknowledgment

I am writing to express my gratitude to Dr. Michael Nelson for providing extremely helpful feedback on my blog post. His deep knowledge and perceptive comments significantly enhanced the article.

-- Md Javedul Ferdous (@jaf_ferdous)

Reference

Uckun, U., Tumkur Suresh, R., Ferdous, M.J., Bi, X., Ramakrishnan, I.V. and Ashok, V., 2022, July. "Taming User-Interface Heterogeneity with Uniform Overlays for Blind Users." In Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization (pp. 212-222). DOI:10.1145/3503252.3531317