Towards Tactile Intelligence: Grounding Robot Manipulation Models in Multisensory Contact

Abstract

Touch comes before sight, before speech. In today’s AI landscape, this Margaret Atwood quote is playing out in reverse; despite touch being a crucial modality for physical interaction, its integration into robot manipulation remains secondary. While significant progress has been made in spatial and semantic understanding for robot manipulation, achieving physical robustness, reactivity, and dexterity remains a fundamental challenge. Current paradigms often rely on exocentric vision to infer the complex dynamics of hand-object-environment interactions. This task is frequently ill-posed due to visual occlusions and the latent nature of contact characteristics such as friction, stiffness, and force. To achieve truly fluent manipulation skills, robots must not only understand the visual cues of their environment but also learn to speak the primary language of interaction: contacts. Unlike vision, which is largely captured through structured images, touch is inherently multisensory. It can be perceived through elastomer deformations, magnetic field perturbations, high-frequency vibrations, pressure, and other signals. This richness introduces significant complexity in abstracting meaningful contact information, which is often sparse, for downstream tasks. This thesis proposes a unified framework for tactile perception for incorporating contact feedback into robot decision-making. The research is organized into three hierarchical thrusts. First, we address the observability gap in contact-rich tasks by introducing Neural Contact Fields (NCF), a method for tracking arbitrary extrinsic contact interactions that are typically occluded from vision. We demonstrate that providing robots with explicit spatial awareness of contact enables robust zero-shot sim-to-real transfer for complex insertion tasks. Second, to address the lack of standardized tactile encoders and the prohibitive cost of labeled data collection, we introduce the Sparsh family of models. We adapt self-supervised learning techniques to the tactile domain to learn general-purpose representations from unlabeled data using masked objectives. These representations capture both static contact properties (such as material, 3-axis forces) and dynamic interaction cues (such as slip, grasp stability, and pose), offering a rich latent space that can be leveraged by policies in the same manner as pre-trained vision encoders. Our framework comprises Sparsh for vision-based tactile sensors, Sparsh-Skin for magnetic skins, and Sparsh-X, a multisensory backbone that fuses image, vibration, motion, and pressure signals into a shared latent space. Finally, we integrate these tactile priors into a generative framework through Visuo-Tactile World Models (VT-WM). By grounding a robot’s internal “imagination” in local tactile feedback, we demonstrate that the model is less prone to hallucinations, maintains object permanence under heavy occlusion, and adheres to physical laws during planning. Together, these contributions provide a path toward data-effcient, contact-aware robot manipulation, demonstrating that the synergy between global context and local interaction is essential for reliable, fine-grained dexterity in the real world.