Open Access

Execution monitoring is essential for robots to detect and respond to failures. Since it is impossible to enumerate all failures for a given task, we learn from successful executions of the task to detect visual anomalies during runtime. Our method learns to predict the motions that occur during the nominal execution of a task, including camera and robot body motion. A probabilistic U-Net architecture is used to learn to predict optical flow, and the robot's kinematics and 3D model are used to model camera and body motion. The errors between the observed and predicted motion are used to calculate an anomaly score. We evaluate our method on a dataset of a robot placing a book on a shelf, which includes anomalies such as falling books, camera occlusions, and robot disturbances. We find that modeling camera and body motion, in addition to the learning-based optical flow prediction, results in an improvement of the area under the receiver operating characteristic curve from 0.752 to 0.804, and the area under the precision-recall curve from 0.467 to 0.549.

The dataset contains the following data from successful and failed executions of the Toyota HSR robot placing a book on a shelf. RGB images from the robot's head camera Depth images from the robot's head camera Rendered images of the robot's 3D model from the point of view of the robot's head camera Force-torque readings from a wrist-mounted force-torque sensor Joint efforts, velocities and positions extrinsic and intrinsic camera calibration parameters frame-level anomaly annotations The anomalies that occur during execution include: the manipulated book falling down books on the shelf being disturbed significantly camera occlusions robot being disturbed by an external collision The dataset is split into a train, validation and test set with the following number of trials: Train: 48 successful trials Validation: 6 successful trials Test: 60 anomalous trials and 7 successful trials

This dataset contains multimodal data recorded on robots performing robot-to-human and human-to-robot handovers. The intended application of the dataset is to develop and benchmark methods for failure detection during the handovers. Thus the trials in the dataset contain both successful and failed handover actions. For a more detailed description of the dataset, please see the included Datasheet.