Extending Human Capabilities with Deep Learning-Powered Wearables

Abstract

Deep learning-powered wearables have the potential to seamlessly extend human capabilities by enhancing perception and interaction in everyday environments. In this dissertation, I present three wearable systems that integrate deep learning neural networks with custom hardware to enable real-time audio enhancement, vision-based smart interactions, and visual intelligence through wireless earbuds. First, I present ClearBuds, a wireless earbud system that performs real-time speech enhancement using a synchronized binaural microphone array and a lightweight dual-channel neural network. The system achieves high-precision synchronization and low-latency processing on mobile devices, enabling robust noise suppression and background speech removal in diverse real-world conditions. Second, I introduce IRIS, a vision-enabled smart ring that fits within the size and power constraints of the ring form factor to enable context-aware smart home interactions. By combining scene semantics with detected objects, IRIS achieves instance-level device recognition and outperforms voice commands in speed, precision, and social acceptability. Third, and as the final contribution, I present VueBuds, the first vision-enabled wireless earbuds integrating low-power cameras with vision language model interaction. VueBuds addresses fundamental challenges in embedding cameras into earbuds—strict power and form-factor constraints, facial occlusion from ear-level positioning, and real-time multimodal processing over Bluetooth. Through a stereo camera system operating at under 5 mW and end-to-end system optimizations, VueBuds achieves visual question-answering performance comparable to commercial smart glasses while leveraging a significantly more ubiquitous form factor. Together, these systems demonstrate how deep learning powered wearables can extend human capabilities with on-the-go intelligence, establishing new platforms for intuitive, responsive, and enhanced human-computer interaction.