REVIEW OF AUTOMATIC RECOGNITION METHODS OF HUMAN EMOTIONAL STATE USING IMAGE

Abstract

The problem of recognizing a person’s emotional state from an image is considered. A review of the main ways of describing human emotions is given: the division into a finite number of classes and the use of vector format. Existing developments in the field of emotions recognition by image are presented, as well as a general algorithm for the operation of such systems is provided. The main steps in solving the problem of recognizing emotions are the search for a face in the image and the emotions classification. Information technology for the recognition of emotions is presented in the graphic notation. The principles of the Viola-Jones algorithm, which is used to determine the person’s face in the image, are described. The approaches that are used to solve the classification problem are described: the Viola-Jones algorithm, reference points method, various neural network architectures that are used to classify images. The advantages and disadvantages of the reference point method based on the facial action coding system are analyzed, as well as the way the Viola-Jones algorithm is used to classify emotions. A method for recognizing a person’s emotional state based on visual information using convolutional neural networks is considered. The principles of the action of convolutional, sub-sampling and fully connected layers of the neural network are described. Based on the analysis of published works, the results of recognition accuracy under various conditions are presented. Also presented works in which combination of convolutional and recurrent neural networks is used to analyze the emotional state, where in addition to visual information used an audio stream, which gives more efficient classification of emotions in a video stream. The most popular training data sets for solving the considered problem are presented.