Human-in-the-loop Teleoperation for Tactile-driven Physical Interactions in Unstructured Environments

Book excerpt: The unpredictability of unstructured environments poses a significant challenge to fully autonomous systems. A human in the loop can provide high-level decisions that leverage experience, context, and human creativity. Allowing humans to operate remotely through teleoperation improves flexibility and safety in hazardous conditions. Our work addresses human-in-the-loop teleoperation applications in assistive robotics and hazardous environments with granular materials. We demonstrate the critical role that haptic feedback plays in enabling humans and robots to operate in unstructured environments effectively. In study #1, we developed a tactile-based proximity sensing framework for estimating the distance to buried objects in granular materials by exploiting granular media jamming. As a sensorized robot finger displaces granular material towards a buried object, jamming granular particles leads to increased fingertip contact forces. A tactile sensor effectively acts as a proximity sensor when moved through granular materials. Using our framework, the distance to a buried object can be estimated using solely haptic feedback. In study #2, we developed a structured approach for estimating the orientation and diameter of cables buried in sand using a deformable, vision-based tactile sensor. Our orientation estimation method identifies the contact region of the cable by employing a proximity function that takes extracted image features as input. Our approach is compared with existing image feature extraction methods, including PCA and maximally stable extremal regions. Our framework demonstrates the effectiveness of tactile sensing when an object is partially or fully occluded in granular materials. In study #3, we designed a telepresence system with real-time auditory, visual, and haptic feedback. The system comprises a modified Stretch mobile manipulator with additional controllable degrees of freedom in the wrist and fingers. The operator interface consists of a virtual reality headset and a haptic glove. The operator controls the remote robot using intuitive controls based on motion tracking. To convey tactile information, the operator receives electrical impulses to stimulate the afferent sensory nerves of each finger driven by tactile feedback from the robot hand. In summary, we developed tactile-based systems and approaches to enhance human-in-the-loop teleoperation for applications in unstructured environments, such as in the home and in granular materials.