2023-07-29: InSupport: Proxy Interface for Enabling Efficient Non-Visual Interaction with Web Data Records

Figure 1: The virtual cover of the IUI '22 conference.

In 2022, there were 268 million online shoppers in the United States. This number will increase to almost 285 million online shoppers in 2025. It’s become a much easier way to shop, considering that it doesn’t require you to leave your house. People every day go to online websites to buy their goods based on their preferences. An e-commerce website like Walmart or Amazon consists of web data records that are integral components of many websites [Figure 2]. These websites present structured information by querying from a database. Examples include a list of products or services, such as items on a shopping website, available hotel rooms on a travel website, and so on. In this blog, I will discuss our work titled “InSupport: Proxy Interface for Enabling Efficient Non-Visual Interaction with Web Data Records.” which was published at the 27th International Conference on Intelligent User Interfaces. The work focuses on improving the interaction experience for blind screen reader users by using machine learning-based algorithms to identify auxiliary segments automatically. The approach significantly outperformed state-of-the-art alternatives in a usability evaluation research with 14 blind individuals, reducing completion time and keypresses.

Figure 2: Data record of Amazon website

Introduction

Most websites offer support tools like data filters, sort options, search forms, and page links for easy interaction with data records. However, for blind users who rely on keyboard-driven screen reader assistive technology, accessing auxiliary segments are less convenient. Users must manually move focus between data records and auxiliary segments, and variations in shortcut patterns may increase frustration and time spent on screen readers. To provide convenient access to auxiliary segments, we designed a popup interface, namely InSupport, containing dummy controls that serve as proxies to the actual auxiliary segment controls present on the webpage.

Framework

We implemented InSupport as a browser extension so that it can provide support for any arbitrary webpage containing data records. InSupport has two core components: a Segment Extractor and a Proxy Interface [Figure 3]. The Segment Extractor analyzes the webpage DOM and extracts the auxiliary segments using custom machine learning-based identification algorithms. The content of the identified auxiliary segments is then duplicated and presented to a user via the Proxy Interface, as shown in the figure. All user selections in the Proxy Interface, e.g., "sort by price," are automatically translated by InSupport into equivalent actions on the actual auxiliary segment controls on the webpage. 

Figure 3: InSupport architectural workflow

We devised four separate algorithms for extracting the four types of auxiliary segments. These algorithms all have a similar workflow. They first extract all the candidate DOM nodes by referring to a predefined list. This predefined list was compiled after a manual analysis of 100 web pages from top-visited websites belonging to different domains such as shopping, travel, job search, and classifieds. Then the algorithms extract a set of handcrafted features from the DOM subtrees of all candidates. The extracted features representing each candidate are then fed to custom-built machine learning models that use classification to determine whether the candidate is the intended auxiliary segment. The keyboard shortcuts for navigating the InSuport popup interface mostly consist of basic navigation shortcuts such as arrow and Tab keys, as shown in the table. The special shortcut to invoke the popup interface was set to CTRL+SHIFT+Z. 

User Study

To evaluate InSupport, we conducted a user study with 14 screen reader users proficient in web browsing and the JAWS screen reader. The participants did two types of representative tasks on travel and shopping websites under three conditions. The participants varied in age between 26 and 64 years [Table 1]. All participants reported no motor impairments that affected their ability to use screen-reader shortcuts. Proficiency with web screen-reading and Chrome web browser constituted our inclusion criteria. All participants stated that they regularly accessed a wide range of e-commerce websites for doing activities such as shopping, searching for jobs, and browsing classifieds.

Table 1: Participant demographics for the user study

The results for task completion times and the number of input actions favored the InSupport condition compared to the other two study conditions. These differences in the performance metrics were also statistically significant. Overall, in the InSupport condition, the participants limited their focus to the popup interface and therefore spent less time exploring the web pages as they did in the other two conditions. For the standard System Usability Scale questionnaire to measure usability, the participants rated positive and negative statements about each study condition on a Likert scale from 1 - strongly disagree to 5 - strongly agree. The System Usability Scale scores measuring usability were significantly higher on average for the InSupport condition than others. Similarly, we also did the NASA task load index, which allows you to determine the mental workload of a participant while they are performing a task. the NASA TLX scores measuring task workload were significantly lower for InSupport, and the ratings for the individual TLX subscales were more uniform with no apparent patterns. In the exit interview, most participants attributed their favorable ratings to the instant access feature of InSupport, which, they said, reduced the number of key presses for navigating between webpage segments.

Finding and Discussion

Almost all participants stated that having a separate proxy interface for accessing the auxiliary segments was vital because it helped them "separate their concerns," that is, using screen reader shortcuts only for navigating within the data records and not worrying about how to navigate to the auxiliary segments. Most participants also stated that with a screen reader, they could not explore many data records due to fatigue, so they often missed out on the “best deals.” These participants expressed that fatigue and frustration were significantly lower with InSupport, so they could explore more data records and get better deals.  In other qualitative feedback, five participants expressed a desire to remember the past selection of filters and then automatically apply them in future interactions involving the same or similar data records, to mitigate duplication of effort. Half the participants wished to have all web data records on one page, as distribution over multiple pages only increased their interaction burden. Our work had a few limitations. First, the dataset for building and evaluating the algorithms were relatively small, and the evaluation was not performed "in the wild" on arbitrary web pages. The evaluation too was restricted to a specific environment, such as Chrome browser, English language webpages, and JAWS screen reader. Further work is therefore needed to ensure the generalizability of the proposed ideas. Informed by the study findings, automatic filter selection and reordering based on prior user data can further reduce the interaction effort for screen reader users, so we also intend to pursue this idea in our future work. 

Conclusion

• We presented InSupport, a browser extension that automatically extracts the important auxiliary segments from webpages containing data records, making them instantly accessible to blind screen readers.
• Evaluation of InSupport in a user study with blind participants showed that InSupport significantly reduced the interaction effort when compared with a state-of-the-art solution and also the participants’ preferred screen readers.
• The entire project, including the dataset, is publicly available on GitHub.

References

Ferdous, Javedul, Hae-Na Lee, Sampath Jayarathna, and Vikas Ashok. "InSupport: Proxy Interface for Enabling Efficient Non-Visual Interaction with Web Data Records." In 27th International Conference on Intelligent User Interfaces, pp. 49-62. 2022.

-- Md Javedul Ferdous (@jaf_ferdous)