Tactile Sensing, Skill Learning, and Robotic Dexterous Manipulation

Chapter 12: A hand-arm teleoperation system for robotic dexterous manipulation

Shuang Li^a; Qiang Li^b; Jianwei Zhang^a    aUniversität Hamburg, Hamburg, Germany^bUniversity of Bielefeld, Bielefeld, Germany

Abstract

To perform dexterous robotic manipulation, we set up a multimodal teleoperation system, which combines vision-based control for the robotic hand and IMU-based control for the robot arm. A novel vision-based neural network, Transteleop, is proposed to estimate the joint commands of the robot hand. Integrating the image-to-image translation approach, Transteleop bridges the kinematic gap between the robot hand and the human hand. Combining with wearable IMU on the human arm, the robot hand-arm coordinately tracking is performed to follow the ...