Volltext-Downloads (blau) und Frontdoor-Views (grau)

Towards Robust and Interpretable Practical Applications of Automatic Mental State Analysis Using a Dynamic and Hybrid Facial Action Estimation Approach

  • This dissertation presents a probabilistic state estimation framework for integrating data-driven machine learning models and a deformable facial shape model in order to estimate continuous-valued intensities of 22 different facial muscle movements, known as Action Units (AU), defined in the Facial Action Coding System (FACS). A practical approach is proposed and validated for integrating class-wise probability scores from machine learning models within a Gaussian state estimation framework. Furthermore, driven mass-spring-damper models are applied for modelling the dynamics of facial muscle movements. Both facial shape and appearance information are used for estimating AU intensities, making it a hybrid approach. Several features are designed and explored to help the probabilistic framework to deal with multiple challenges involved in automatic AU detection. The proposed AU intensity estimation method and its features are evaluated quantitatively and qualitatively using three different datasets containing either spontaneous or acted facial expressions with AU annotations. The proposed method produced temporally smoother estimates that facilitate a fine-grained analysis of facial expressions. It also performed reasonably well, even though it simultaneously estimates intensities of 22 AUs, some of which are subtle in expression or resemble each other closely. The estimated AU intensities tended to the lower range of values, and were often accompanied by a small delay in onset. This shows that the proposed method is conservative. In order to further improve performance, state-of-the-art machine learning approaches for AU detection could be integrated within the proposed probabilistic AU intensity estimation framework.

Export metadata

Additional Services

Search Google Scholar Check availability


Show usage statistics
Document Type:Doctoral Thesis
Author:Teena Hassan
Number of pages:xxviii, 159
Date of exam:2020/05/27
Contributing Corporation:Otto-Friedrich-Universit├Ąt Bamberg
Date of first publication:2020/10/29
Award:Promotionspreis der Otto-Friedrich-Universit├Ąt Bamberg
Keyword:Machine learning; distraction detection; facial action units; facial expression analysis; information fusion; interpretability; mental state analysis; pain detection; state estimation
Dewey Decimal Classification (DDC):0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 006 Spezielle Computerverfahren
Entry in this database:2023/04/13
Licence (German):License LogoCreative Commons - CC BY - Namensnennung 4.0 International