Data-driven Robotic Manipulation of Deformable Objects Using Tactile Feedback

Book excerpt: Perceiving and manipulating deformable objects with the sense of touch are essential skills in everyday life. However, it remains difficult for robots to autonomously manipulate deformable objects using tactile sensing because of numerous perception, modeling, planning, and control challenges. We believe this is partially due to two fundamental challenges: (1) Establishing a physics-based model describing physical interactions between deformable tactile sensors and deformable objects is difficult; (2) Modern tactile sensors provide high-dimensional data, which is beneficial for perception but impedes the development of practical planning and control strategies. To address these challenges, we developed systematic frameworks for the tactile-driven manipulation of deformable objects that integrates state-of-the-art tactile sensing with well-established tools used by other robotics communities. In Study \#1, we showed how a robot can learn to manipulate a deformable, thin-shell object via tactile sensor feedback using model-free reinforcement learning methods. A page flipping task was learned on a real robot using a two-stage approach. First, we learned nominal page flipping trajectories by constructing a reward function that quantifies functional task performance from the perspective of tactile sensing. Second, we learned adapted trajectories using tactile-driven perceptual coupling, with an intuitive assumption that, while the functional page flipping trajectories for different task contexts (page sizes) might differ, similar tactile sensing feedback should be expected. In Study \#2, we showed how a robot can use tactile sensor feedback to control the pose and tension of a deformable linear object (elastic cable). For a cable manipulation task, low-dimensional latent space features were extracted from high-dimensional raw tactile sensor data using unsupervised learning methods, and a dynamics model was constructed in the latent space using supervised learning methods. The dynamics model was integrated with an optimization-based, model predictive controller for end-to-end, tactile-driven motion planning and control on a real robot. In summary, we developed frameworks for the tactile-driven manipulation of deformable objects that either circumvents sensor modeling difficulties or constructs a dynamics model directly from tactile feedback and uses the model for planning and control. This work provides a foundation for the further development of systematic frameworks that can address complex, tactile-driven manipulation problems.