Convolutional Recurrent Neural Network with Bi-LSTM for Speech Emotion Recognition
Article Sidebar
Main Article Content
Abstract
Understanding the emotion in a human voice is a complex but crucial part of improving human machine interaction. This study explores the potential of a Convolutional Recurrent Neural Network (CRNN), an architecture that combines the spatial feature-finding ability of a CNN with the temporal awareness of a BiLSTM. This model was trained and tested using the CREMA-D dataset, a collection of audio clips covering six basic emotions including anger, happiness, and sadness. To prepare the audio for the model, the signals were converted into log-Mel spectrograms, a format chosen for its relevance to human auditory perception. In an effort to improve the model's ability to generalize, data augmentation techniques like pitch shifting and time stretching were also implemented. The results suggest that this approach was beneficial. Adding data augmentation appeared to lift the validation accuracy to 68.23%, a noticeable improvement from the 66.08% achieved without it. The model showed particularly promising performance when it came to classifying 'anger' and 'neutral' emotions. Ultimately, this work indicates that the CRNN architecture with BiLSTM could be a capable approach for speech emotion recognition, and it strongly underscores the vital role that data augmentation seems to play in achieving better performance. Looking ahead, further gains might be possible by investigating more sophisticated augmentation methods or by incorporating a richer set of acoustic features for the model to learn from.
Article Details
Section
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
How to Cite
References
M. A. Qamhan, A. H. Meftah, S. Selouani, Y. A. Alotaibi, and Y. M. Seddiq, "Speech emotion recognition using convolutional recurrent neural networks and spectrograms," in Proc. IEEE Canadian Conf. Electr. Comput. Eng., 2020, pp. 1-6.
M. Gokilavani, "Ravdness, Crema-D, Tess based algorithm for emotional recognition using speech," in Proc. 4th Int. Conf. Smart Syst. Inventive Technol., 2023, pp. 1658-1664.
K. A. Mukarram, M. A. Mukhlas, and A. Zahra, "Enhancing speech emotion recognition with deep learning using multi-feature stacking and data augmentation," Bull. Electr. Eng. Informatics, vol. 13, no. 3, pp. 1920-1926, 2024.
S. Premjeet, Md. Sahidullah, and S. Goutam, "Modulation spectral features for speech emotion recognition using deep neural networks," Speech Commun., pp. 53-59, 2022.
S. H. Hawley, "Fourier transforms, audio engineering, and the quantum nature of reality," in Proc. AES 135th Convention, 2013, pp. 1-4.
D. A. Sauter, F. Eisner, A. J. Calder, and S. K. Scott, "Perceptual cues in non-verbal vocal expressions of emotion," Q. J. Exp. Psychol., vol. 63, no. 11, pp. 2251–2272, 2010.
K. Markou, L. Tsochatzidis, K. Zagoris, A. Papazoglou, X. Karagiannis, S. Symeonidis, and I. Pratikakis, "A convolutional recurrent neural network for the handwritten text recognition of historical Greek manuscripts," Lect. Notes Comput. Sci., pp. 227-239, 2021.
M. Zielonka, A. Piastowski, A. Czyzewski, P. Nadachowski, M. Operlejn, and K. Kaczor, "Recognition of emotions in speech using convolutional neural networks on different datasets," Electronics, vol. 11, no. 3831, pp. 1-12, 2022.
K. S. Nugroho, I. Akbar, A. N. Suksmawati, and Istiadi, "Deteksi depresi dan kecemasan pengguna Twitter menggunakan bidirectional LSTM," in Proc. 4th Conf. Innov. Appl. Sci. Technol. (CIASTECH 2021), 2021, pp. 287-296.
N. T. Pham, D. N. M. Dang, N. D. Nguyen, T. T. Nguyen, H. Nguyen, B. Manavalan, C. P. Lim, S. D. Nguyen, "Hybrid data augmentation and deep attention-based dilated convolutional-recurrent neural networks for speech emotion recognition," Expert Syst. Appl., vol. 230, pp. 1-13, 2023.
H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, "CREMA-D: Crowd-sourced emotional multimodal actors dataset," IEEE Trans. Affect. Comput., vol. 5, no. 4, pp. 377-390, 2014.
S. S. Mantrala, "Python's pivotal role in AI and data science," Int. J. Inf. Technol. Manag. Inf. Syst., vol. 16, no. 2, pp. 1676-1686, 2025.
A. Pavithra and S. M. Prakash, "TensorFlow in deep learning," Int. J. Innov. Res. Technol., vol. 5, no. 9, pp. 142-147, 2019.
T. G. Patil and A. V. Zade, "To design and develop advanced speech emotion recognition using MLP classifier with evolutionary LIBROSA library," Int. J. Multidiscip. Res., vol. 5, no. 3, pp. 1-6, 2023.
J. M. E. Diana and A. Mathivanan, "Exploring the paradigm shift: Harnessing data analytics for real-world applications," Int. J. Sci. Res., vol. 12, no. 6, pp. 1467-1480, 2023.
F. Reiss, B. Cutler, and Z. Eichenberger, "Natural language processing with Pandas DataFrames," in Proc. 20th Python Sci. Conf., 2021, pp. 49-57.
F. Makhmudov, A. Kutlimuratov, and Y. -I. Cho, "Hybrid LSTM–attention and CNN model for enhanced speech emotion recognition," Appl. Sci., vol. 14, no. 23, pp. 1-19, 2024.
D. Vazhenina and K. Markov, "End-to-end noisy speech recognition using fourier and hilbert spectrum features," Electronics, vol. 9, no. 1157, pp. 1-18, 2020.
O. P. Yadav, L. P. Bastola, and J. Sharma, "Speech emotion recognition using convolutional recurrent neural network," in Proc. 10th IOE Graduate Conf., 2021, pp. 1164-1172.
D. Singh and B. Singh, "Investigating the impact of data normalization on classification performance," Appl. Soft Comput. J., pp. 1-23, 2019.
Y. Lee, "The CNN: The architecture behind artificial intelligence development," J. Stud. Res., vol. 12, no. 4, 2023.
T. Bezdan and N. Bacanin, "Convolutional neural network layers and architectures," in Proc. Sinteza 2019 - Int. Sci. Conf. Inf. Technol. Data Relat. Res., vol. 4, 2019, pp. 445-451.
P. Meyer, Z. Xu, and T. Fingscheidt, "Improving convolutional recurrent neural networks for speech emotion recognition," in Proc. IEEE Spoken Lang. Technol. Workshop, 2021, pp. 1-9.
B. J. Hutagaol and T. Mauritsius, "Risk level prediction of life insurance applicant using machine learning," Int. J. Adv. Trends Comput. Sci. Eng., vol. 9, no. 2, pp. 2213-2220, 2020.
S. Pothuganti, "Review on over-fitting and under-fitting problems in machine learning and solutions," Int. J. Adv. Res. Electr. Electron. Instrum. Eng., vol. 7, no. 9, pp. 3692-3695, 2018.