Locating and identifying objects in vision-denied environments is a critical challenge for intelligent robot systems. We present a tactile-only method for object search and recognition using custom tactile skin sensors on robot hands. The system employs a novel two-phase classification process: an initial single-handed classification by pushing, followed by a bimanual verification stage that incorporates size measurement.
To support this, we introduce the HAS (Hide-and-Seek) dataset, a large-scale multimodal tactile dataset of 1.12 million frames. Our system achieves a 91.1% object classification accuracy and a 61.4% success rate in real-world identification experiments.
The framework consists of four main phases: (1) Bimanual Tactile Exploration for incremental 3D mapping; (2) Push and Classify for initial object identification; (3) Bimanual Verification to reduce errors via shape and size cues; (4) Grasp and Pick for target retrieval.
Demos — full pipeline execution: exploration, push-and-classify, bimanual verification, and retrieval
The Hide-and-Seek (HAS) Dataset covers 61 object-weight categories with 1.12 million frames. It includes five synchronized modalities: tactile point clouds, virtual skin wrenches, proprioceptive poses, raw skin data, and wrist F/T wrenches.
Data is collected via two interaction primitives that mirror the system's online pipeline. In the Push phase, the robot performs a single-handed pushing motion to make initial tactile contact with the object. In the Grasp phase, both hands simultaneously grasp the object to capture bimanual tactile signals for size and shape verification.
Push phase — single-handed tactile contact
Grasp phase — bimanual tactile verification