REVIEW OF AUTOMATIC RECOGNITION METHODS OF HUMAN EMOTIONAL STATE USING IMAGE
DOI:
https://doi.org/10.20998/2079-0023.2020.01.15Keywords:
emotion recognition, classification, reference point method, convolutional neural network, recurrent neural network, training setAbstract
The problem of recognizing a person’s emotional state from an image is considered. A review of the main ways of describing human emotions is given: the division into a finite number of classes and the use of vector format. Existing developments in the field of emotions recognition by image are presented, as well as a general algorithm for the operation of such systems is provided. The main steps in solving the problem of recognizing emotions are the search for a face in the image and the emotions classification. Information technology for the recognition of emotions is presented in the graphic notation. The principles of the Viola-Jones algorithm, which is used to determine the person’s face in the image, are described. The approaches that are used to solve the classification problem are described: the Viola-Jones algorithm, reference points method, various neural network architectures that are used to classify images. The advantages and disadvantages of the reference point method based on the facial action coding system are analyzed, as well as the way the Viola-Jones algorithm is used to classify emotions. A method for recognizing a person’s emotional state based on visual information using convolutional neural networks is considered. The principles of the action of convolutional, sub-sampling and fully connected layers of the neural network are described. Based on the analysis of published works, the results of recognition accuracy under various conditions are presented. Also presented works in which combination of convolutional and recurrent neural networks is used to analyze the emotional state, where in addition to visual information used an audio stream, which gives more efficient classification of emotions in a video stream. The most popular training data sets for solving the considered problem are presented.References
Tao J., Tan. T., Picard R. W. Affective Computing and Intelligent Interaction. First International Conference. Berlin, Springer-Verlag Publ., 2005, pp. 981–995.
Ekman P. Universals and cultural differences in the judgments of facial expressions of emotion. Journal of personality and social psychology. 1987, vol. 53, no. 4, pp. 712–714.
Russell J., Mehrabian A. Evidence for a three-factor theory of emotions. Journal of research in Personality. 1977, vol. 11, no. 3, pp. 273–294.
Khorrami P., Le Paine T., Brady K., Dagli C., Huang T. S. How Deep Neural Networks Can Improve Emotion Recognition on Video Data. 2016 IEEE International Conference on Image Processing (ICIP). Piscataway, IEEE Publ., 2016, pp. 619–623.
Vision AI. Available at: https://cloud.google.com/vision (accessed 10.03.2020).
Amazon Rekognition. Available at: https://aws.amazon.com/rekognition/ (accessed 11.03.2020).
Face API. Available at: https://azure.microsoft.com/enus/services/cognitive-services/face/ (accessed 09.03.2020).
Watson Tone Analyzer. Available at: https://www.ibm.com/watson/services/tone-analyzer/ (accessed 10.03.2020).
Haar A. Zur Theorie der orthogonalen. Funktionensysteme. Mathematische Annalen. 1910, b. 69, s. 331–371.
Viola P., Jones M. Robust Real-Time Face Detection. International Journal of Computer Vision. 2004, no. 57, pp. 137–154.
Lanitis A., Taylor C. J., Ahmed T., Cootes T. F. Classifying variable objects using a flexible shape model. Fifth International Conference on Image Processing and its Applications, Edinburgh, IET Publ., 1995, pp. 70–74.
LeCun Y., Bottou L., Bengio Y., Haffner P. Gradient-Based Learning Applied to Document Recognition Proceedings of the IEEE. 1998, vol. 86, no. 11, pp. 2278–2324.
Kahou S, Bouthillier X., Lamblin P. EmoNets: Multimodal deep learning approaches for emotion recognition in video. Journal on Multimodal User Interfaces. 2015, no. 10, pp. 99–111.
Challenges in Representation Learning: Facial Expression Recognition Challenge. Available at: https://www.kaggle.com/c/challenges-in-representation-learningfacial-expression-recognition-challenge/data (accessed 12.03.2020).
Downloads
How to Cite
Issue
Section
License
Copyright (c) 2020 Bulletin of National Technical University "KhPI". Series: System Analysis, Control and Information TechnologiesAuthors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).