Runtime Repair and Enhancement of Mobile App Accessibility

Abstract

Mobile devices and applications (apps) have become ubiquitous in daily life. Ensuring full access to the wealth of information and services they provide is a matter of social justice. Unfortunately, many capabilities and services offered by apps today remain inaccessible for people with disabilities. Built-in accessibility tools rely on correct app implementation, but app developers often fail to implement accessibility guidelines. In addition, the absence of tactile cues makes mobile touchscreens difficult to navigate for people with visual impairments. This dissertation describes the research I have done in runtime repair and enhancement of mobile app accessibility. First, I explored a design space of interaction re-mapping, which provided examples of re-mapping existing inaccessible interactions into new accessible interactions. I also implemented interaction proxies, a strategy to modify an interaction at runtime without rooting the phone or accessing app source code. This strategy enables third-party developers and researchers to repair and enhance mobile app accessibility. Second, I developed a system for robust annotations on mobile app interfaces to make the accessibility repairs reliable and scalable. Third, I built Interactiles, a low-cost, portable, and unpowered system to enhance tactile interaction on touchscreen phones for people with visual impairments. The thesis of this dissertation is: An interaction remapping strategy can enable third-party developers and researchers to robustly repair and enhance the accessibility of mobile applications at runtime, while preserving the platform’s security model and accessibility infrastructure.