Evaluating Uncertainty Estimation Methods on 3D Semantic Segmentation of Point Clouds
- Deep learning models are extensively used in various safety critical applications. Hence these models along with being accurate need to be highly reliable. One way of achieving this is by quantifying uncertainty. Bayesian methods for UQ have been extensively studied for Deep Learning models applied on images but have been less explored for 3D modalities such as point clouds often used for Robots and Autonomous Systems. In this work, we evaluate three uncertainty quantification methods namely Deep Ensembles, MC-Dropout and MC-DropConnect on the DarkNet21Seg 3D semantic segmentation model and comprehensively analyze the impact of various parameters such as number of models in ensembles or forward passes, and drop probability values, on task performance and uncertainty estimate quality. We find that Deep Ensembles outperforms other methods in both performance and uncertainty metrics. Deep ensembles outperform other methods by a margin of 2.4% in terms of mIOU, 1.3% in terms of accuracy, while providing reliable uncertainty for decision making.
Document Type: | Preprint |
---|---|
Language: | English |
Author: | Swaroop Bhandary K, Nico Hochgeschwender, Paul Plöger, Frank Kirchner, Matias Valdenegro-Toro |
Number of pages: | 12 |
DOI: | https://doi.org/10.48550/arXiv.2007.01787 |
ArXiv Id: | http://arxiv.org/abs/2007.01787 |
Publisher: | arXiv |
Date of first publication: | 2020/07/03 |
Copyright: | Copyright 2020 by the author(s). |
Note: | Presented at the ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning |
Departments, institutes and facilities: | Fachbereich Informatik |
Institut für Cyber Security & Privacy (ICSP) | |
Dewey Decimal Classification (DDC): | 0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik |
Entry in this database: | 2020/07/17 |