The Power of UI Automation

What if you needed to get a list of all the open browser tabs in some browser? In the (very) old days you might assume that each tab is its own window, so you could find a main browser window (using FindWindow, for example), and then enumerate child windows with EnumChildWindows to locate the tabs. Unfortunately, this approach is destined to fail. Here is a screenshot of WinSpy looking at a main window of Microsoft Edge:

MS Edge showing only two child windows

The title of the main window hints to the existence of 26 tabs, but there are only two child windows and they are not tabs. The inevitable conclusion is that the tabs are not windows at all. They are being “drawn” with some technology that the Win32 windowing infrastructure doesn’t know about nor cares.

How can we get information about those browsing tabs? Enter UI Automation.

UI Automation has been around for many years, starting with the older technology called “Active Accessibility“. This technology is geared towards accessibility while providing rich information that can be consumed by accessibility clients. Although Active Accessibility is still supported for compatibility reasons, a newer technology called UI Automation supersedes it.

UI Automation provides a tree of UI automation elements representing various aspects of a user interface. Some elements represent “true” Win32 windows (have HWND), some represent internal controls like buttons and edit boxes (created with whatever technology), and some elements are virtual (don’t have any graphical aspects), but instead provide “metadata” related to other items.

The UI Automation client API uses COM, where the root object implements the IUIAutomation interface (it has extended interfaces implemented as well). To get the automation object, the following C++ code can be used (we’ll see a C# example later):

CComPtr<IUIAutomation> spUI;
auto hr = spUI.CoCreateInstance(__uuidof(CUIAutomation));
if (FAILED(hr))
	return Error("Failed to create Automation root", hr);

The client automation interfaces are declared in <UIAutomationClient.h>. The code uses the ATL CComPtr<> smart pointers, but any COM smart or raw pointers will do.

With the UI Automation object pointer in hand, several options are available. One is to enumerate the full or part of the UI element tree. To get started, we can obtain a “walker” object by calling IUIAutomation::get_RawViewWalker. From there, we can start enumerating by calling IUIAutomationTreeWalker interface methods, like GetFirstChildElement and GetNextSiblingElement.

Each element, represented by a IUIAutomationElement interface provides a set of properties, some available directly on the interface (e.g. get_CurrentName, get_CurrentClassName, get_CurrentProcessId), while others hide behind a generic method, get_CurrentPropertyValue, where each property has an integer ID, and the result is a VARIANT, to allow for various types of values.

Using this method, the menu item View Automation Tree in WinSpy shows the full automation tree, and you can drill down to any level, while many of the selected element’s properties are shown on the right:

WinSpy automation tree view

If you dig deep enough, you’ll find that MS Edge tabs have a UI automation class name of “EdgeTab”. This is the key to locating browser tabs. (Other browsers may have a different class name). To find tabs, we can enumerate the full tree manually, but fortunately, there is a better way. IUIAutomationElement has a FindAll method that searches for elements based on a set of conditions. The conditions available are pretty flexible – based on some property or properties of elements, which can be combined with And, Or, etc. to get more complex conditions. In our case, we just need one condition – a class name called “EdgeTab”.

First, we’ll create the root object, and the condition (error handling omitted for brevity):

int main() {
	::CoInitialize(nullptr);

	CComPtr<IUIAutomation> spUI;
	auto hr = spUI.CoCreateInstance(__uuidof(CUIAutomation));

	CComPtr<IUIAutomationCondition> spCond;
	CComVariant edgeTab(L"EdgeTab");
	spUI->CreatePropertyCondition(UIA_ClassNamePropertyId, edgeTab, &spCond);

We have a single condition for the class name property, which has an ID defined in the automation headers. Next, we’ll fire off the search from the root element (desktop):

CComPtr<IUIAutomationElementArray> spTabs;
CComPtr<IUIAutomationElement> spRoot;
spUI->GetRootElement(&spRoot);
hr = spRoot->FindAll(TreeScope_Descendants, spCond, &spTabs);

All that’s left to do is harvest the results:

int count = 0;
spTabs->get_Length(&count);
for (int i = 0; i < count; i++) {
	CComPtr<IUIAutomationElement> spTab;
	spTabs->GetElement(i, &spTab);
	CComBSTR name;
	spTab->get_CurrentName(&name);
	int pid;
	spTab->get_CurrentProcessId(&pid);
	printf("%2d PID %6d: %ws\n", i + 1, pid, name.m_str);
}

Try it!

.NET Code

A convenient Nuget package called Interop.UIAutomationClient.Signed provides wrappers for the automation API for .NET clients. Here is the same search done in C# after adding the Nuget package reference:

static void Main(string[] args) {
    const int ClassPropertyId = 30012;
    var ui = new CUIAutomationClass();
    var cond = ui.CreatePropertyCondition(ClassPropertyId, "EdgeTab");
    var tabs = ui.GetRootElement().FindAll(TreeScope.TreeScope_Descendants, cond);
    for (int i = 0; i < tabs.Length; i++) {
        var tab = tabs.GetElement(i);
        Console.WriteLine($"{i + 1,2} PID {tab.CurrentProcessId,6}: {tab.CurrentName}");
    }
}

More Automation

There is a lot more to UI automation – the word “automation” implies some more control. One capability of the API is providing various notifications when certain aspects of elements change. Examples include the IUIAutomation methods AddAutomationEventHandler, AddFocusChangedEventHandler, AddPropertyChangedEventHandler, and AddStructureChangedEventHandler.

More specific information on elements (and some control) is also available with more specific interfaces related to controls, such as IUIAutomationTextPattern, IUIAutomationTextRange, and manu more.

Happy automation!