Enhanced Facial Expression Recognition Through a Hybrid Deep Learning Approach Combining ResNet50 and ResNet34 Models
DOI:
https://doi.org/10.30865/klik.v4i6.1874Keywords:
CNN; Attention Mechanism; ResNet50; ResNet34; FERAbstract
Recognizing facial expressions is a critical aspect of computer vision and human-computer interaction. It facilitates the interpretation of human emotions from facial images, aiding in applications such as affective computing, social robotics, and psychological research. In this work, we propose using hybrid deep learning models, ResNet50 and ResNet34, for facial expression classification. These models, pre-trained on large-scale datasets, demonstrate exceptional feature extraction capabilities and have achieved excellent performance in various computer vision tasks. Our approach begins with the collection and preprocessing of a labeled facial expression dataset. The collected data undergoes face detection, alignment, and normalization to ensure consistency and reduce noise. After preprocessing, the dataset is divided into training, validation, and testing sets. We fine-tune the ResNet50 and ResNet34 models on the training set, employing transfer learning to adapt the pre-trained models specifically for the facial expression recognition task. Optimization techniques such as SGDM, ADAM, and RMSprop are used to update the models' parameters and minimize the categorical cross-entropy loss function. The trained models are evaluated on the validation set, achieving an accuracy of 98.19%. Subsequently, the models are tested on unseen facial images to assess their generalization capabilities. This proposed approach aims to deliver accurate and robust facial expression classification, thereby advancing emotion analysis and human-computer interaction systems.
Downloads
References
L. Sels, H. T. Reis, A. K. Randall, and L. Verhofstadt, “Emotion Dynamics in Intimate Relationships: The Roles of Interdependence and Perceived Partner Responsiveness,” in Affect Dynamics, Cham: Springer International Publishing, 2021, pp. 155–179. doi: 10.1007/978-3-030-82965-0_8.
S. C. Leong, Y. M. Tang, C. H. Lai, and C. K. M. Lee, “Facial expression and body gesture emotion recognition: A systematic review on the use of visual data in affective computing,” Comput. Sci. Rev., vol. 48, p. 100545, May 2023, doi: 10.1016/j.cosrev.2023.100545.
T. Kopalidis, V. Solachidis, N. Vretos, and P. Daras, “Advances in Facial Expression Recognition: A Survey of Methods, Benchmarks, Models, and Datasets,” Information, vol. 15, no. 3, p. 135, Feb. 2024, doi: 10.3390/info15030135.
E. G. Krumhuber, L. I. Skora, H. C. H. Hill, and K. Lander, “The role of facial movements in emotion recognition,” Nat. Rev. Psychol., vol. 2, no. 5, pp. 283–296, Mar. 2023, doi: 10.1038/s44159-023-00172-1.
M. A. Solis-Arrazola, R. E. Sanchez-Yañez, C. H. Garcia-Capulin, and H. Rostro-Gonzalez, “Enhancing image-based facial expression recognition through muscle activation-based facial feature extraction,” Comput. Vis. Image Underst., vol. 240, p. 103927, Mar. 2024, doi: 10.1016/j.cviu.2024.103927.
H. Pérez-Espinosa, R. Zatarain-Cabada, and M. L. Barrón-Estrada, “Emotion recognition: from speech and facial expressions,” in Biosignal Processing and Classification Using Computational Learning and Intelligence, Elsevier, 2022, pp. 307–326. doi: 10.1016/B978-0-12-820125-1.00028-2.
A. R. Dores, F. Barbosa, C. Queirós, I. P. Carvalho, and M. D. Griffiths, “Recognizing Emotions through Facial Expressions: A Largescale Experimental Study,” Int. J. Environ. Res. Public Health, vol. 17, no. 20, p. 7420, Oct. 2020, doi: 10.3390/ijerph17207420.
M. A. H. Akhand, S. Roy, N. Siddique, M. A. S. Kamal, and T. Shimamura, “Facial Emotion Recognition Using Transfer Learning in the Deep CNN,” Electronics, vol. 10, no. 9, p. 1036, Apr. 2021, doi: 10.3390/electronics10091036.
K. Sarvakar, R. Senkamalavalli, S. Raghavendra, J. Santosh Kumar, R. Manjunath, and S. Jaiswal, “Facial emotion recognition using convolutional neural networks,” Mater. Today Proc., vol. 80, pp. 3560–3564, 2023, doi: 10.1016/j.matpr.2021.07.297.
H. Ghazouani, “Challenges and Emerging Trends for Machine Reading of the Mind from Facial Expressions,” SN Comput. Sci., vol. 5, no. 1, p. 103, Dec. 2023, doi: 10.1007/s42979-023-02447-z.
Nidhi and B. Verma, “From methods to datasets: a detailed study on facial emotion recognition,” Appl. Intell., vol. 53, no. 24, pp. 30219–30249, Dec. 2023, doi: 10.1007/s10489-023-05052-y.
M. Sajjad et al., “A comprehensive survey on deep facial expression recognition: challenges, applications, and future guidelines,” Alexandria Eng. J., vol. 68, pp. 817–840, Apr. 2023, doi: 10.1016/j.aej.2023.01.017.
S. H. Syed and V. Muralidharan, “Feature extraction using Discrete Wavelet Transform for fault classification of planetary gearbox – A comparative study,” Appl. Acoust., vol. 188, p. 108572, Jan. 2022, doi: 10.1016/j.apacoust.2021.108572.
M. Zuhaib et al., “Faults Feature Extraction Using Discrete Wavelet Transform and Artificial Neural Network for Induction Motor Availability Monitoring—Internet of Things Enabled Environment,” Energies, vol. 15, no. 21, p. 7888, Oct. 2022, doi: 10.3390/en15217888.
A. Topic and M. Russo, “Emotion recognition based on EEG feature maps through deep learning network,” Eng. Sci. Technol. an Int. J., vol. 24, no. 6, pp. 1442–1454, Dec. 2021, doi: 10.1016/j.jestch.2021.03.012.
R. Yuvaraj, P. Thagavel, J. Thomas, J. Fogarty, and F. Ali, “Comprehensive Analysis of Feature Extraction Methods for Emotion Recognition from Multichannel EEG Recordings,” Sensors, vol. 23, no. 2, p. 915, Jan. 2023, doi: 10.3390/s23020915.
J. Zhang, Z. Yin, P. Chen, and S. Nichele, “Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review,” Inf. Fusion, vol. 59, pp. 103–126, Jul. 2020, doi: 10.1016/j.inffus.2020.01.011.
A. Khan, A. Sohail, U. Zahoora, and A. S. Qureshi, “A survey of the recent architectures of deep convolutional neural networks,” Artif. Intell. Rev., vol. 53, no. 8, pp. 5455–5516, Dec. 2020, doi: 10.1007/s10462-020-09825-6.
I. Lobato, T. Friedrich, and S. Van Aert, “Deep convolutional neural networks to restore single-shot electron microscopy images,” npj Comput. Mater., vol. 10, no. 1, p. 10, Jan. 2024, doi: 10.1038/s41524-023-01188-0.
X. Zhao, L. Wang, Y. Zhang, X. Han, M. Deveci, and M. Parmar, “A review of convolutional neural networks in computer vision,” Artif. Intell. Rev., vol. 57, no. 4, p. 99, Mar. 2024, doi: 10.1007/s10462-024-10721-6.
H. Li, W. Wang, M. Wang, L. Li, and V. Vimlund, “A review of deep learning methods for pixel-level crack detection,” J. Traffic Transp. Eng. (English Ed., vol. 9, no. 6, pp. 945–968, Dec. 2022, doi: 10.1016/j.jtte.2022.11.003.
S. Sony, K. Dunphy, A. Sadhu, and M. Capretz, “A systematic review of convolutional neural network-based structural condition assessment techniques,” Eng. Struct., vol. 226, p. 111347, Jan. 2021, doi: 10.1016/j.engstruct.2020.111347.
D. Zhao, Y. Qian, J. Liu, and M. Yang, “The facial expression recognition technology under image processing and neural network,” J. Supercomput., vol. 78, no. 4, pp. 4681–4708, Mar. 2022, doi: 10.1007/s11227-021-04058-y.
B. Ko, “A Brief Review of Facial Emotion Recognition Based on Visual Information,” Sensors, vol. 18, no. 2, p. 401, Jan. 2018, doi: 10.3390/s18020401.
S. Park et al., “Differences in Facial Expressions between Spontaneous and Posed Smiles: Automated Method by Action Units and Three-Dimensional Facial Landmarks,” Sensors, vol. 20, no. 4, p. 1199, Feb. 2020, doi: 10.3390/s20041199.
M. T. Mustapha, I. Ozsahin, and D. U. Ozsahin, “Convolution neural network and deep learning,” in Artificial Intelligence and Image Processing in Medical Imaging, Elsevier, 2024, pp. 21–50. doi: 10.1016/B978-0-323-95462-4.00002-9.
S. R. Shah, S. Qadri, H. Bibi, S. M. W. Shah, M. I. Sharif, and F. Marinello, “Comparing Inception V3, VGG 16, VGG 19, CNN, and ResNet 50: A Case Study on Early Detection of a Rice Disease,” Agronomy, vol. 13, no. 6, p. 1633, Jun. 2023, doi: 10.3390/agronomy13061633.
K. K. Bressem, L. C. Adams, C. Erxleben, B. Hamm, S. M. Niehues, and J. L. Vahldiek, “Comparing different deep learning architectures for classification of chest radiographs,” Sci. Rep., vol. 10, no. 1, p. 13590, Aug. 2020, doi: 10.1038/s41598-020-70479-z.
A. Victor Ikechukwu, S. Murali, R. Deepu, and R. C. Shivamurthy, “ResNet-50 vs VGG-19 vs training from scratch: A comparative analysis of the segmentation and classification of Pneumonia from chest X-ray images,” Glob. Transitions Proc., vol. 2, no. 2, pp. 375–381, Nov. 2021, doi: 10.1016/j.gltp.2021.08.027.
S. K. Khare, V. Blanes-Vidal, E. S. Nadimi, and U. R. Acharya, “Emotion recognition and artificial intelligence: A systematic review (2014–2023) and research recommendations,” Inf. Fusion, vol. 102, p. 102019, Feb. 2024, doi: 10.1016/j.inffus.2023.102019.
E. A. Clark et al., “The Facial Action Coding System for Characterization of Human Affective Response to Consumer Product-Based Stimuli: A Systematic Review,” Front. Psychol., vol. 11, May 2020, doi: 10.3389/fpsyg.2020.00920.
R. Yelchuri, J. K. Dash, P. Singh, A. Mahapatro, and S. Panigrahi, “Exploiting deep and hand-crafted features for texture image retrieval using class membership,” Pattern Recognit. Lett., vol. 160, pp. 163–171, Aug. 2022, doi: 10.1016/j.patrec.2022.06.017.
L. Alzubaidi et al., “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions,” J. Big Data, vol. 8, no. 1, p. 53, Mar. 2021, doi: 10.1186/s40537-021-00444-8.
J. Chai, H. Zeng, A. Li, and E. W. T. Ngai, “Deep learning in computer vision: A critical review of emerging techniques and application scenarios,” Mach. Learn. with Appl., vol. 6, p. 100134, Dec. 2021, doi: 10.1016/j.mlwa.2021.100134.
I. H. Sarker, “Deep Learning: A Comprehensive Overview on Techniques, Taxonomy, Applications and Research Directions,” SN Comput. Sci., vol. 2, no. 6, p. 420, Nov. 2021, doi: 10.1007/s42979-021-00815-1.
X. Zhao, X. Shi, and S. Zhang, “Facial Expression Recognition via Deep Learning,” IETE Tech. Rev., vol. 32, no. 5, pp. 347–355, Sep. 2015, doi: 10.1080/02564602.2015.1017542.
J. Li, K. Jin, D. Zhou, N. Kubota, and Z. Ju, “Attention mechanism-based CNN for facial expression recognition,” Neurocomputing, vol. 411, pp. 340–350, Oct. 2020, doi: 10.1016/j.neucom.2020.06.014.
R. Bendjillali, M. Beladgham, K. Merit, and A. Taleb-Ahmed, “Improved Facial Expression Recognition Based on DWT Feature for Deep CNN,” Electronics, vol. 8, no. 3, p. 324, Mar. 2019, doi: 10.3390/electronics8030324.
Q. T. Ngoc, S. Lee, and B. C. Song, “Facial Landmark-Based Emotion Recognition via Directed Graph Neural Network,” Electronics, vol. 9, no. 5, p. 764, May 2020, doi: 10.3390/electronics9050764.
M. Reyad, A. M. Sarhan, and M. Arafa, “A modified Adam algorithm for deep neural network optimization,” Neural Comput. Appl., vol. 35, no. 23, pp. 17095–17112, Aug. 2023, doi: 10.1007/s00521-023-08568-z.
E. Hassan, M. Y. Shams, N. A. Hikal, and S. Elmougy, “The effect of choosing optimizer algorithms to improve computer vision tasks: a comparative study,” Multimed. Tools Appl., vol. 82, no. 11, pp. 16591–16633, May 2023, doi: 10.1007/s11042-022-13820-0.
R. I. Bendjillali, M. Beladgham, K. Merit, and A. Taleb-Ahmed, “Illumination-robust face recognition based on deep convolutional neural networks architectures,” Indones. J. Electr. Eng. Comput. Sci., vol. 18, no. 2, p. 1015, May 2020, doi: 10.11591/ijeecs.v18.i2.pp1015-1027.
N. M. Dipu, S. Alam Shohan, and K. M. . Salam, “Ocular Disease Detection Using Advanced Neural Network Based Classification Algorithms,” ASIAN J. Converg. Technol., vol. 7, no. 2, pp. 91–99, Aug. 2021, doi: 10.33130/AJCT.2021v07i02.019.
R. R*, S. Lal P, A. M. Philip, and V. V, “A Compact Deep Learning Model for Robust Facial Expression Recognition.,” Int. J. Eng. Adv. Technol., vol. 8, no. 6, pp. 2956–2960, Aug. 2019, doi: 10.35940/ijeat.F8724.08861.
Y. Gong, G. Liu, Y. Xue, R. Li, and L. Meng, “A survey on dataset quality in machine learning,” Inf. Softw. Technol., vol. 162, p. 107268, Oct. 2023, doi: 10.1016/j.infsof.2023.107268.
S. S. Hiremath, J. Hiremath, V. V. Kulkarni, B. C. Harshit, S. Kumar, and M. S. Hiremath, “Facial Expression Recognition Using Transfer Learning with ResNet50,” 2023, pp. 281–300. doi: 10.1007/978-981-99-1624-5_21.
A. Soliman, S. Shaheen, and M. Hadhoud, “Leveraging pre-trained language models for code generation,” Complex Intell. Syst., vol. 10, no. 3, pp. 3955–3980, Jun. 2024, doi: 10.1007/s40747-024-01373-8.
N. Mehendale, “Facial emotion recognition using convolutional neural networks (FERC),” SN Appl. Sci., vol. 2, no. 3, p. 446, Mar. 2020, doi: 10.1007/s42452-020-2234-1.
V. R. Joseph and A. Vakayil, “SPlit: An Optimal Method for Data Splitting,” Technometrics, vol. 64, no. 2, pp. 166–176, Apr. 2022, doi: 10.1080/00401706.2021.1921037.
S. F. Ahmed et al., “Deep learning modelling techniques: current progress, applications, advantages, and challenges,” Artif. Intell. Rev., vol. 56, no. 11, pp. 13521–13617, Nov. 2023, doi: 10.1007/s10462-023-10466-8.
M. Wojciuk, Z. Swiderska-Chadaj, K. Siwek, and A. Gertych, “Improving classification accuracy of fine-tuned CNN models: Impact of hyperparameter optimization,” Heliyon, vol. 10, no. 5, p. e26586, Mar. 2024, doi: 10.1016/j.heliyon.2024.e26586.
S. Rajan, P. Chenniappan, S. Devaraj, and N. Madian, “Facial expression recognition techniques: a comprehensive survey,” IET Image Process., vol. 13, no. 7, pp. 1031–1040, May 2019, doi: 10.1049/iet-ipr.2018.6647.
M. J. A. Dujaili, “Survey on facial expressions recognition: databases, features and classification schemes,” Multimed. Tools Appl., vol. 83, no. 3, pp. 7457–7478, Jan. 2024, doi: 10.1007/s11042-023-15139-w.
M. Gao, D. Qi, H. Mu, and J. Chen, “A Transfer Residual Neural Network Based on ResNet-34 for Detection of Wood Knot Defects,” Forests, vol. 12, no. 2, p. 212, Feb. 2021, doi: 10.3390/f12020212.
J. Yang et al., “Prediction of HER2-positive breast cancer recurrence and metastasis risk from histopathological images and clinical information via multimodal deep learning,” Comput. Struct. Biotechnol. J., vol. 20, pp. 333–342, 2022, doi: 10.1016/j.csbj.2021.12.028.
M. Aamir et al., “A deep learning approach for brain tumor classification using MRI images,” Comput. Electr. Eng., vol. 101, p. 108105, Jul. 2022, doi: 10.1016/j.compeleceng.2022.108105.
X. Wang, X. Wang, and Y. Ni, “Unsupervised Domain Adaptation for Facial Expression Recognition Using Generative Adversarial Networks,” Comput. Intell. Neurosci., vol. 2018, pp. 1–10, Jul. 2018, doi: 10.1155/2018/7208794.
M.-I. Georgescu, R. T. Ionescu, and M. Popescu, “Local Learning With Deep and Handcrafted Features for Facial Expression Recognition,” IEEE Access, vol. 7, pp. 64827–64836, 2019, doi: 10.1109/ACCESS.2019.2917266.
P. Giannopoulos, I. Perikos, and I. Hatzilygeroudis, “Deep Learning Approaches for Facial Emotion Recognition: A Case Study on FER-2013,” 2018, pp. 1–16. doi: 10.1007/978-3-319-66790-4_1.
P. T. Vinh and T. Quang Vinh, “Facial Expression Recognition System on SoC FPGA,” in 2019 International Symposium on Electrical and Electronics Engineering (ISEE), Oct. 2019, pp. 1–4. doi: 10.1109/ISEE2.2019.8921140.
D. Kollias and S. Zafeiriou, “Expression, affect, action unit recognition: Aff-wild2, multi-task learning and arcface,” 30th Br. Mach. Vis. Conf. 2019, BMVC 2019, no. October, 2020.
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Enhanced Facial Expression Recognition Through a Hybrid Deep Learning Approach Combining ResNet50 and ResNet34 Models
ARTICLE HISTORY
Issue
Section
Copyright (c) 2024 Sigit Auliana, Siti Mahrojah, Gagah Dwiki Putra Aryono
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
