Facial Emotion Recognition Using Deep Learning to Identify the Problems Related to Mental Health

Abstract

Facial emotions, such as neutral, happiness, sadness, fear, annoyance, anger, and surprise, are interpreted similarly across different cultures. These expressions indicate a person's emotional state, signal approval or disapproval of others' behaviors in social situations, and reveal mental or neurological disorders, if any. With the advent of high computing systems, image processing, and deep learning algorithms in the recent decade, facial expression recognition (FER) technology has achieved high accuracy in recognizing emotions from facial images. In many applications, the current technologies have achieved better performance than humans. However, identifying emotions from facial images under constrained circumstances remains challenging due to obstructions, poor or improper lighting conditions, and different positions of the head/body. Moreover, identifying mental or neurological disorders requires knowledge of the problem domain, appropriate deep-learning models, and labeled data sets. This research aims to enhance the performance of facial expression/emotion recognition models and apply them to real-world scenarios, such as identifying autism spectrum disorder in children, which is increasing rapidly, particularly in the post-COVID era. The major contributions of this thesis present several novel classifier methods for identifying and discriminating facial emotions of individuals into seven different emotions: neutral, happiness, sadness, fear, annoyance, anger, and surprise, as endorsed by the American Psychological Association in 2008. Individuals with strong emotional well-being may still have a range of physical and mental issues. The proposed deep learning-based Artificial intelligence models generate sophisticated decisions using real-time data, a key strength that is highly relevant in identifying mental health problems. Emotion recognition is complicated in children with Autism spectrum disorder (ASD) when they are suffering from speech issues and social communication issues, which leads to the requirement of an effective emotion recognition approach. Motivated by the alarming facts of autism spectrum disorder globally and locally, this thesis presents a unique dual-branch CNN-based visual transformation model to identify special children with a binary classifier. Since there is not a single dataset available for special children in terms of our country and south Asia, a real-time dataset was created with the high-resolution images that were collected from the two special schools of iv Dhaka, Bangladesh. An original dataset was created for future researchers, which is considered to be one of the key contributions of this research program. The outcome of the developed model is finally evaluated with the other approaches and models, outperforming existing techniques in accuracy and other measures and successful detection of special children. The proposed multiple approaches have been evaluated, considering precision, recall, f-measure, accuracy, specificity, and recognition rate. The proposed models demonstrated superior performance in all measures compared to the other algorithms and successfully detected special children. We also believe the proposed model will benefit our country's underprivileged people who have limitations in the early detection of mental health problems. Thus, the objectives of this Ph.D. research will be worthwhile.