Toward Safer Augmented Reality: Securing Input, Output, and Interaction

Abstract

Augmented Reality (AR) technologies have evolved significantly over the years. Once considered niche and expensive research prototypes, AR devices are becoming increasingly accessible and powerful. In addition to the hardware advancements, the application development ecosystems and AI capabilities integrated into these devices have also rapidly expanded. These advancements in AR will soon empower individuals to use AR on an everyday basis. As millions of users begin to explore AR technologies and incorporate them into their daily lives, safeguarding users’ security and privacy from unwanted threats becomes ever more imperative. Due to AR devices’ ability to alter users’ perceptions of the physical world, the nature of their three-dimensional user interface, and multi-modal sensing capabilities, many of these threats are fundamentally different from known risks of non-immersive technologies like web and mobile interfaces. In this dissertation, I identify critical security and privacy risks, evaluate these risks in cutting-edge AR systems, and propose mitigation solutions to enhance user safety. My approach centers on analyzing the three core phases of the AR system data flow -- input, output, and interaction -- each of which introduces distinct classes of vulnerabilities. For threats related to AR input, I investigated the emerging sensory permission models, such as eye-tracking and hand-tracking, for three major AR platforms (HoloLens 2, Oculus Quest Pro, and Vision Pro). My collaborators and I surveyed 280 participants on Prolific to investigate their comfort, perceived and actual comprehension, and decision factors. We explicitly recruited participants who had no prior experience with AR, in order to capture people’s comfort and comprehension on their first exposure to these permission-granting flows, rather than relying on their past experiences. Based on the results, we identify design principles for how future AR platforms can better communicate existing privacy protections, enhance privacy-preserving designs, and more effectively communicate potential risks. For threats related to AR output, I present my work that formalizes the security-related properties of the 3D UI output in AR. My collaborators and I demonstrate the security implications of different instantiations of these properties through five proof-of-concept attacks between distrusting AR application components (i.e., a main app and an included library), including a clickjacking attack and an object erasure attack. We then empirically investigate these UI security properties on five current AR platforms: ARCore (Google), ARKit (Apple), HoloLens (Microsoft), Oculus (Meta), and WebXR (browser), finding that all platforms enable at least three of our proof-of-concept attacks to succeed. We provide concrete recommendations for platform developers, including adaptations of existing 2D UI security measures and novel AR-specific defense techniques to prevent these attacks. For threats related to AR interaction, I describe my work that characterizes perceptual manipulation attacks (PMA) in AR, which involves manipulating users’ multi-sensory (e.g., visual, auditory, haptic) perceptions of the world when users are interacting with AR content. Through immersive adversarial overlaid content, PMA influence users’ judgments and following actions to induce incorrect perception, cognition, or resulting reaction. To provide a foundation for understanding and addressing PMA, my collaborators and I conducted an in-person study with 21 participants with three PMA that attacked different perceptions: visual, auditory, and situational awareness. Our findings reveal the effectiveness of these attacks and inform design guidelines for defending against PMA in AR environments. Together, this thesis represents significant theoretical and empirical progress toward secure, privacy-preserving, and trustworthy AR systems for mainstream adoption.