A Music System Based on Emotion Recognition from Facial Expressions

For most people, music is an important part of their lives. In this context, an emotion recognition system that would complement music streaming is a thing of interest. This paper proposes an application that streams music from the available playlists based on user emotion inferred from facial expressions. The emotion recognition system comprises of face detection, feature extraction, and emotion classification. We have used Viola-Jones algorithm for face detection, and the detected face is the input to the Constrained Local Model (CLM) algorithm which handles feature extraction to give feature points. We have defined rules on feature points to classify them into basic emotions which are happy, sad, angry, surprise and neutral. According to the detected emotion, the response song from the predefined playlist is played. Emotion of the user is continuously assessed so as to change music as the user mood changes.

Face Detection

Many algorithms have been proposed for Face Detection until now. Viola-Jones Face Detector is a widely used method for object detection. This algorithm performs a quick scan of the image for faces with a Viola-Jones face detector. Once the Face Detection is performed on the frontal face image, the part of the image that contains the face is forwarded to the next stage.

Facial Feature Extraction

We use constrained local model(CLM) for feature extraction. Before performing close scrutiny with CLM, the first priority is to have a face shape model, and a model of the appearance of individual patch, i.e., a patch model.
1. When the CLM algorithm does the close scrutiny, it searches for patches around its most probable position, and use shape constraints to constrain your search.
2. The process of building a shape and patch model can be termed as the model-building process, and CLM search as search process.

To build shape model, Procrustes analysis is used to remove rotation, scale and translation variation, and then PCA to find a basis of variation and the amount of variation each basis represents (the eigenvectors and eigenvalues).
1. To build patch model for each feature point, train linear SVM with negative and positive samples, so that it can identify whether a given patch of the image is the right one.
2. CLM search processing goes like this: starting from an initial guess, using linear SVM to search local region around its feature point, and obtain a response image. Next, fitting a quadratic function to the response image. Then CLM solves for new feature point positions by optimizing a function of quadratic functions and shape constraints. All the steps are repeated until all points reach a stable position.

Emotion Classification

In this paper, we selected six basic emotions that proposed by Ekman and Friesen(1978), as basic emotion classes. These emotions are Angry, Disgust, Fear, Happiness, Sadness, and Surprise. In this section, we define a set of rules for emotion recognition using facial feature points obtained in the previous step. Two examples of these rules are shown in the following:

Rule 1: If (Eye-Opening is High) And (Eyebrow-Constriction is Low) And (Mouth-Opening isVeryHigh) And (Mouth-Corner-Displacement isLow) Then Surprise

Rule 2: If (Eye-Opening isMedium) And (Eyebrow-Constriction is Very High) And (Mouth-Opening isVeryLow) And (Mouth- Corner-Displacement isNegativeLow) Then Angry

We devised these kind rules for emotion recognition by checking upon various image databases and finally refined them to obtain rules, similar to the human inference(common knowledge) for emotion recognition. So, we defined rules for emotion recognition completely based on human inference manner and tested integrity and correctness of these rules by experimenting with many facial images that included different emotions and checked whether these rules could recognize true emotions (that presented in the tested images) or no.

Figure 1. Basic emotions detected(a) Happy (b) Sad (c) Angry (d) Surprise

Music Selection

Music always has a profound effect on the human psyche. It is undeniably important in shaping moods, and, likewise, certain frames of mind require certain kinds of songs.

There have been lot of studies on how the dynamic aspect of music emotion is related to the cognition of music structure. People report that their primary motivation in listening to music is its emotional effect. Music can be reliably described at the level of basic emotions. The sad excerpts have slow tempos, minor harmonies, and fairly constant ranges of pitch and dynamics. The happy excerpts have relatively rapid tempos, dancelike rhythms, major harmonies and relatively constant ranges of pitch and dynamics.

So whether one is feeling down and need a pick-me-up; one is down and like to stay that way for a bit whether one is in an ”Empire State of Mind,” or its just another ”Manic Monday” pick perfect songs to suits ones mood.

Full Report

report.pdf