Pixel-Based Reverse Engineering of Graphical Interfaces

Abstract

User interface software is often large, complex, rigid, and difficult to implement. In fact, most of the development time and effort in a typical application is dedicated to the user interface. Not only do interfaces require the programmer to deal with complex input and output, asynchrony, and rapid feedback to the user that is context-dependent, but they also need to run on many different devices and serve a wide variety of people with complex needs. For example, applications may be required to run on the desktop, on mobile touchscreen devices, both for people with motor impairments and able-bodied users, and in multiple languages. As a result, most of today’s interfaces require an immense amount of resources to address the increasingly complex ecosystem of devices and users. This dissertation focuses on unlocking existing user interfaces to overcome these challenges. Similar to how Wikipedia opened encyclopedia articles for anyone to edit, we envision a similar democratization of user interfaces, where anybody can modify the graphical interface of any application. Any single company or group of developers cannot single-handedly address people’s wide variety of needs, but unlocking existing interfaces could diffuse the cost and effort across many people. As a first step towards realizing this vision, this dissertation explores how to modify existing graphical interfaces by reverse engineering their pixels. Working from the pixels enables developers and end-users to modify existing interfaces without compliance from the underlying application. For example, we use our pixel-based methods to implement many different modifications including accessibility enhancements, improved input on mobile devices, interface translation for improved localization, and end-user customization. Researchers have explored pixel-based reverse engineering for decades, but this dissertation presents the first real-time methods for interpreting an interface’s pixels and using that reverse engineered structure to modify an application’s apparent behavior.