Continuous Image Generation from Low-Update-Rate Images and Physical Sensors through a Conditional GAN for Robot Teleoperation
Dae-Kwan Ko ; Dong-Han Lee ; Soo-Chul Lim
When a robot is teleoperated, its operator control is based on transmitted images. Network limitations and/or a remote distance usually cause delays or interruptions of the image transmission, which is one of the reasons for the instability of teleoperation systems. In this work, we propose a high-update-rate image generation method using past low update image and current grip position and electrical motor current of gripper received by sensors during teleoperation via a conditional generative adversarial network. The main challenge is that such a network can generate current high-update-rate images from past low-update-rate one, the current high-update-rate grip force, and the grip angle. We equipped a robot gripper with a camera and a grip force sensor and collected a large dataset of robot vision, grip force, and grip angle sequences; objects with deformation, including irregular deformation, and rigid objects were tested in the experiment to verify the possibility of high-update-rate image generation under various grip conditions. We found that the proposed network allows the generation of current images with high update rate. Here, we present the results of real-time image generation from delayed images and sensor data along with the root-mean-square value, the peak signal-to-noise ratio, and the structural similarity index measure and compare the images generated by the proposed network with those from other networks.
Published in: IEEE Transactions on Industrial Informatics ( Early Access )
Page(s): 1 – 1
Date of Publication: 06 May 2020