2020-12-30: UI Automation: A walkthrough with UI Automation

Fig: a demonstration of UI Automation in Notepad

Essential criteria for accessibility assistance in the application are programmatic access and keyboard access. To test accessibility for people with different disabilities and limitations or those who prefer to use a keyboard, it is important that you test the accessibility of your Windows applications, assistive technology (AT) tools, and user interface (UI) frameworks. You will not be able to use your application for users with visual, learning, dexterity/mobility, and with language/commune impairments or disabilities, without appropriate access through AT such as screen readers and on-screen keyboards.

In the last blog, I talked about some high-level concepts on UI Automation (UIA). In this walkthrough, I’ll give a snippet on how to use UIA on Notepad. You can write your script using C++, C#, or Python. Microsoft uses C++ and C# for their UI Automation and all kinds of testing. However, most of the developers use Python, since it is convenient to use as well as it can easily pipeline with another machine learning-based work. I’ll use two UIA based Python libraries for this demonstration: 

Uiautomation library supports UI automation for the applications which implemented UI automation provider, such as MFC, Windows Form, WPF, Modern UI(Metro UI), Qt, and Firefox. If you run it for the first time on Python, you need to run the below command first:

Notepad is a generic Microsoft Windows text editor that allows you to create, open, and read files with plaintext. I will automate this Notepad with UIA by using the uiautomation library. My plan is to write a Python script to select and customize font, write a sentence, check with each keyboard character, and finally, save the Notepad file without using a mouse and keyboard.

A video demonstration of code is shown here:


(An alternative version of the video)

In the video, you can see, once I clicked the compiled button on Visual Studio Code, it will open a Notepad. In the Notepad, it will search the “Format” menu item. Once it finds it, it will click and go to the “Font…” element. A dialog box will popup and within the dialog, you can see all the font-related configuration is shown for a Notepad. After selecting the recommended font configuration, it will start to write on a Notepad. Once it writes the first sentence, it will change the font configuration again to show even working on a sentence or paragraph, you can still change your font configuration. After changing it, it will test with all the character from the keyboard, finally, it will save the Notepadon the desktop and name it "UIA". This is one approach to work with the UIA library.

Pywinauto, another Python library, automates the Microsoft Windows UI. It allows you to send mouse and keyboard actions to windows dialog and control type simplest way for both Windows and Linux. If you are run Pywinauto for the first time, you need to install this library by executing this command first:

Pywinauto needs a few dependencies to support its work:
  1. pyWin32
  2. comtypes
  3. six
  4. Pillow
After installing dependency and the package, you are ready to write your first script. 

The first part of the script is to import the required module. Let's begin with the import of the Pywinauto module application class. Application for each automation process is the starting point.

from pywinauto.application import Application

You also have to import the following code if you want to call basic user input functions like the mouse and the keyboard:

import pywinauto.mouse as mouse
import pywinauto.keyboard as keyboard

The next step is to connect the application instance to automate it. There are two ways you can do it.

notepadapp = Application(backend="uia").start("notepad.exe")
or
notepadapp = Application(backend="uia").connect(path=r"C:\Windows\system32\notepad.exe")

"start()" is used while you're not running the program and need to start it. It accepts a string that can contain the argument for the command line and a timeout. Only when your program takes too long to start is the timeout parameter needed whereas "connect()" is used while a running procedure is being attempted. To specify an already running application, you must assign one of the following: process, handle, path, or any combination of the parameters that specify a window.

The next phase is to recognize all available controls by running the code below.
notepad_window = notepadapp.window(title='Untitled - Notepad')
notepad_window.print_control_identifiers()

It will print our all the available control type in a parent-child hierarchical tree format for each node where "Untitled - Notepad" is the root. You will automatically use various forms of controls. "Button", "Edit", and "MenuItem" are the most common buttons to use. Each control has a function call of its own. For further detail, see the documentation.


Fig: Tree representation of Notepad 

Let's test with some sentences with the keyboard.

dialog = notepadapp['Untitled - Notepad']
dialog.type_keys('Hello{SPACE}from{SPACE}PYWINAUTO')
dialog.type_keys('{ENTER}Lets test with keyboard')
dialog.type_keys('{ENTER}0123456789')
dialog.type_keys('{ENTER}ABCDEFGHIJKLMNOPQRSTUVWXYZ')
dialog.type_keys('{ENTER}abcdefghijklmnopqrstuvwxyz')
dialog.type_keys('{ENTER}`~!@#$%^&*()-_=+')
dialog.type_keys('{ENTER}[]{{}{}}\\|;:\'\",<.>/?')

If you execute it, you will find that "Hello from PYWINAUTO" is the final text appearing on the Notepad, where there is no place for both spaces. This is typical since the keyboard entry is based on this module. This can quickly be overcome by inserting the correct key codes in the string. For further information please visit the Pywinauto documentation.

Similarly, you can use specific coordination to move the mouse to a particular position. Here, I will the close button of the Notepad by clicking the cross-section of the Notepad. To get a specific location, you need to run this code below:

notepad_window["Close"].rectangle()

The output will show the top, bottom, left, right positions of the "Close" button.

To click this, you need to focus the dialog first, then use this coordination to click the "Close" button.

dialog.set_focus()
mouse.click(coords=(1608, 267, 1655, 297))

If you want to move from one menu item or browse within it, you have to use "menu_select" to proceed. For example, like the previous example of "uiautomation", if we are looking for font configuration, we need to run this code:

fmenu = dialog.menu_select('Format->Font...')
fdlg = notepadapp.Font
ftype = fdlg.ComboBox
ftype.select('Trebuchet MS')
ftype = fdlg.ComboBox2
ftype.select('Bold Italic')
fsize = fdlg.ComboBox3
fsize.type_keys('28')
fdlg.OK.click() 

A quick recap of what we have so far. First, we install the "uiautomation" and "Pywinauto" module and demonstrate the installation processes of each module. We also examined in detail the fundamental use of the module. I used both libraries in Notepad to automate its core elements as an example in this case. We have also written some code for automating basic acts like clicking and selecting the menu. Next, we have attempted manual automation using the integrated mouse and the keyboard that offers greater versatility than normal. Thank you for reading this post, and I look forward to seeing you again!

my GitHub repo: https://github.com/javedulferdous/UIABlog

-- Md Javedul Ferdous(@jaf_ferdous)

Comments