Speech-Driven Cartoon Animation with Emotions

Yan Li, Feng Yu* Ying-Qing Xu, Eric Chang, Heung-Yeung ShumMicrosoft Research China: 3F Sigma Center, No.49 Zhichun Road, Beijing 100080, China

{yli,yqxu,echang,hshum)@microsoft.com

ABSTRACTIn this paper, we present a cartoon face animation systemfor multimedia HCI applications. We animate face cartoonsnot only from input speech, but also based on emotions de-rived from speech signal. Using a corpus of over 700 utter-ances from different speakers, we have trained SVMs (sup-port vector machines) to recognize four categories of emo-tions: neutral, happiness, anger and sadness. Given eachinput speech phrase, we identify its emotion content as amixture of all four emotions, rather than classifying it into asingle emotion. Then, facial expressions are generated fromthe recovered emotion for each phrase, by morphing differentcartoon templates that correspond to various emotions. Toensure smooth transitions in the animation, we apply low-pass filtering to the recovered (and possibly jumpy) emotionsequence. Moreover, lip-syncing is applied to produce thelip movement from speech, by recovering a statistical audio-visual mapping. Experimental results demonstrate that car-toon animation sequences generated by our system are ofgood and convincing quality.

KeywordsMultimedia IICI, cartoon animation, speech emotion recog-nition, lip-syncing

1. Introduct ionFacial animation has been an active research topic for com-puter graphics and multimedia. In particular, facial anima-tion can be used as an effective communication channel forhuman-computer interface (I-ICI). For instance, MPEG4 hasa face and body animation committee that defines how thehuman face and body should be modeled and transmitted.Llore and more commercial products have been developedrecently in the forms of talking heads, virtual friends, face

*Visiting from Computer Science Department, TsinghuaUniversity, China.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage and thatcopies bear this notice and the full citation on the first page. To copyotherwise, or republish, to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.MA4’01, Sept. 30-Oct. 5,2001, Ottawa, Canada.Copyright2001 ACM I-58113-394-4/01/0009...$5.00

email, talking shows, virtual announcers, and so on [l, 2, 3,41. By animating human faces in cartoon forms, we can addsome artistic styles [lo, 241.

Directly animating human faces is challenging because thereare so many parameters to be controlled for realistic fa-cial expressions. To alleviate such difficulties for animators,speech-driven animation techniques (e.g., [8, 91) have beenproposed to learn the mapping between the voice and thefacial motion, and then to drive the facial animation fromthe speech signals. However, most previous face animationsystems, to the best of our knowledge, have not consideredexplicitly the emotions existing in the speech signal. Whilespeaking the same content, people may have significantlydifferent facial expressions depending on whether they arehappy or sad. Emotions must be considered in facial ani-mation systems [ll, 221.

Although much work [15, 19, 201 has been done for emo-tion analysis in the field of speech recognition, there aresome difficulties in analyzing emotions from speech. It isnot clear how many categories of emotions should be COP

sidered for facial expressions and facial animation. Recogni-tion of emotions has been shown to be much more difficultthan recognition of phonemes or words. For example, recentwork reports that human subjects can categorize five differ-ent emotions (normal, happy, angry, sad, and afraid) withthe average accuracy of 63.5% [21]. In comparison, in thespeech recognition field, character accuracy rates for largevocabulary continuous speech recognition tasks can be wellover 90% [12].

In our work, we simplify emotion analysis in two ways. First,because our purpose of analyzing emotions is to animatecartoons, we can afford to classify the speech into only fourdifferent but representative emotions: neutral, sadness, hap-piness, and anger. Second, instead of classifying the utter-ance into one of these four emotional categories, we modelthe emotion in the utterance as a mixture of all emotions.

For each of the emotions, we generate a series of cartoontemplates corresponding to the intensity of the emotion.For example, five cartoon templates are used to representthe emotion from neutral to happy. Given the recoveredemotions from speech signal, facial expressions can be gen-erated by morphing between these cartoon templates. Areal-time lip-syncing algorithm is also developed in our sys-tem to make the cartoon animation more lively and believ-

365

Figure 1: System ovcrvicw. The system consists of two parts: emotion-driven cartoon animation and speech-driven cartoon animation

Figure 2 : Four difkrrnt levels (25%: 50%, 75% and 100%) of the cartoon face templatm for three emotions(anger. happiness and sadness). The first levels of these three emotions (or 0% intensity) correspond toneutml rmotion.

lhppiness

/

NeutralAnger

Sadness

Figure 3: An emotion space where the origin is theneutral expression. Any motion is considered as amixture of three emotions (sadness, anger and hap-piness).

feature vector is that different languages (e.g., Chinese andEnglish) do not require training different models. In thetraining stage, all the lip configurations (training data) areclustered into a set of templates automatically. For eachtemplate, a generic model is trained to map from acousticfeature vector to visual feature vector in real time. In thesynthesis stage, we employ Bayesian estimation to obtainthe most probable lip configuration for a given acoustic vec-tor.

Finally, the cartoon animation sequence is synthesized bycombining morphed cartoon templates with the synchro-nized lip configurations.

3. Emotion AnalysisAs an important human behavior for conveying psycholog-ical information, emotion has been studied for centuries.Many aspects of emotions have been studied for emotion-antecedent appraisal, emotion induction, physiological reac-tion and expression of emotion (facial and vocal), and emo-tional behavior on autonomous agents [5].

3.1 Emotion Modeling and EvaluationFrom the psychological point of view, human emotions areoften described as some subjective perceptions such as hap-piness, sadness, surprise, etc. From the experimental andcomputational perspective, however, we require emotion tobe classified explicitly by some parameters. More specif-ically, the intensity of an emotion can be measured by avalue, which can be further used to drive the cartoon facemodel.

Four emotions are recognized in our system: neutral, hap-piness, sadness and anger. Although they cannot encorn-pass all emotions present in speech, it has been shown thathumans can recognize them from utterances with less ambiguity. In addition, these four emotions are representativeenough for a cartoon animation system, especially for appli-cations such as talking shows, avatars and t,eleconferencing.

However, it is difficult to accurately recognize these fouremotions. For the purpose of cartoon animation, rather thanassigning a single emotion to an input utterance, we assumethat the input utterance is a mixture of several emotionswith different intensities. As shown in Figure 3, we model

Table 1: Number of training samples for each emo-tion

Neutral 1 Anger 1 Happiness 1 Sadness 1/ 2 4 2 1 2 1 5 1 1 3 6 I 1 2 8 I

human emotion as a point in an emotion space. Neutralemotion is placed at the origin of this space because it can beused to describe the relative lack of other emotions. Threeaxes represent the intensity of sadness, happiness and anger,respectively. Note that the three axes do not need to beorthogonal in the emotion space. Modeling human emotionin such a three-dimensional continuous space is meant to bea simplification for recognition and animation, as we latershow.

3.2 Training DataWe have collected more than 1000 movie clips and extractedthe acoustic data from the clips as training data, where theaverage clip length is about 10 seconds. The utterances arecarefully selected from speakers with different genders andages. A trained experimenter classified the utterances intofour categories: neutral, happiness, sadness and anger.

In a separate step, we had five adult subjects listen to eachof the utterances and determine its emotion category. Anutterance is considered to be valid as training data onlywhen all the subjects agree unanimously with the initialclassification. Table 1 shows the number of training samplesfor each category in our experiment.

3.3 Feature ExtractionWe extract the acoustic features as used in [15]. For each ut-terance, a 16 dimensional vector with the following elementsis calculated:

.

.

.

.

.

3.4

Statistics related to rhythm: Speaking rate, Averagelength between voiced regions, Number of maxima /number of (minima + maxima), Number of up slopes/ number of slopes.

Statistics on the smoothed pitch signal: Min, Max,Median, Standard deviation.

Statistics on the derivative of the smoothed pitch: Min,Max, Median, Standard deviation.

Statistics over the individual voiced parts: Mean min,Mean max.

Statistics over the individual slopes: Mean positivederivative, Mean negative derivative.

Continuous Speech Emotion Recog-nition

In our system, we use continuous input speech to drive a car-toon face model. We build multidimensional discriminatorsto classify each utterance into its proper category. Becausethe emotion recognizer is trained from a set of utterancesand the emotion can only be stable within a short period,we need to segment the continuous speech into phrases. For

3 6 7

each phrase, we apply the recognition algorithm that pro-duces the proportions of each emotion in that phrase. Thesegmentation algorithm is based on the following assump-tions:

. The intensity of an emotion is unchanging within agiven phrase.

. The emotion is neutral when the phrase is silent orthe classifier cannot determine the emotion categoryof that phrase.

Because the acoustic features are extracted from pitch, thephrases can be separated at regions where the pitch valueappears to be zero for a period T. T can be used as a param-eter to tune the smoothness of the emotion. For instance,T should be small when speech emotion changes drastically.On the other hand, a large T is used to generate smoothemotion transitions. In our experiments, we set T=40ms.

For analysis, we process each phrase as a whole. Because dif-ferent features extracted from the audio data are correlated,nonlinear classifiers need to be designed. Existing emotionrecognition algorithms are mostly based on K-nearest neigh-bor (KNN) or neural networks. In our system, we use sup-port vector machines (SVMs) that can be trained quicklywithout considering the correlations between different fea-tures. Moreover, training data are separated from the recog-nition process after we obtain the classifiers. We have im-plemented SVMs using Gaussian kernels:

Four l-v-r (one-versus-rest) SVMs are trained to distinguishone class from the three others After the training process,we obtain four two-class classifiers S,(,u) (i = 1 .4), where1) is the feature vector. The recognition process can be rep-resented by

1

u E class (if S,(v) > 0)u is rqjected by classifier i (if S,(v) 5 0) (2)

Because the discriminative plane tends to approach the classwit,h more training samples, the performance of SVM will beinfluenced by the numbers of positive and negative trainingsamples. For each discriminator, we select training samplesrandomly from data sets and choose the numbers of positiveand negative training samples that are nearly equal. Forexample, in order to train an SVM that recognizes the emo-Con “anger”, we select 150 anger-labeled utterances as posi-tive samples. The negative samples meanwhile consist of 50happiness-labeled, 50 neutral-labeled and 50 sadness-labeledsamples. Table 2 shows the performance of the SVMs andtheir corresponding training samples. The performance ofthe classifier is evaluated by judging w’hether the automaticclassifications yield the same label as the subjective ones.In our experiment, the most clearly recognizable emotion isneutral, and the recognition performance of happiness andanger is relatively poor. However, what we have found isthat even for human listeners, it is very easy to confusebetween happiness and anger. Our results are comparablewith the state-of-art results recently obtained by other re-searchers [15, 191.

Table 2: Performance of SVMs and training samplenumbers

Number of training samplesA H N S Accuracy

Anger 162 46 49 52 77.16%Happiness 31 102 31 32 65.64%

Neutral 61 68 194 64 83.73%Sadness 31 34 31 96 70.59%

; wh

Figure 4: Filtering the emotion curve. The solidlines are the emotion recognition results. The dashline is the smoothed emotion curve.

One benefit of an SVM discriminator is that it will providea recognition confidence, which can be considered as theproportion of a specific emotion in that phrase. We willdescribe in Section 4.2 that the proportion can be used tomorph between different cartoon templates.

4. Animating Facial ExpressionsThe emotion recognition results are then used to generatecartoon animation, using cartoon templates drawn for allfour emotions.

4.1 Cartoon Face TemplatesAs shown in Figure 2, the cartoon face model begins witha set of hand-drawn images. For each emotion, we have anartist draw four cartoons that correspond to different emo-tion levels or intensities ranging evenly from 25% to 100%.Since neutral emotion is the origin in the emotion space,we need only one template for it. In other words, the firstlevel (or 0% intensity) of three emotions (sadness, anger andhappiness) always corresponds to the neutral emotion.

4.2 Facial Expression AnimationBecause the emotion discriminators produce a determinis-tic metric for each emotion in a phrase, the emotion willchange abruptly between neighboring phrases. Directly animating the facial expression from those outputs will resultin frequent face jitter. Therefore, we apply Gaussian filtersto the emotion curves to generate a smooth emotion tran-sition. For example, Figure 4 shows the emotion curve ofhappiness before and after low-pass filtering.

Similarly, we can obtain smooth emotion curves for sadnessand anger. At any time instant, we have three values thatmeasure the intensity of emotional happiness, sadness, andanger in a speech phrase, represented by oh, ws and wa. Weonly use three emotions here because we assume the emo-tion is neutral when the phrase is silent or the discriminatorcannot determine the emotion category of that phrase.

368

3 6 9

l All the lip configurations of a cartoon character can dimension of the acoustic feature vector distributes indepen-be clustered into several classes called, proto-lips. dently, the likelihood p(a]pi) can be represented by:

l Any given lip configuration can be represented by alinear combination of these proto-lips.

From the above assumptions, the first step in the trainingprocess is to obtain the key lip templates and their weights(or relative proportions). Although the training data canbe obtained by some vision-based techniques such as eigenpoints [14], creating more artistic work for cartoon face an-imation is more important than depicting the character re-alistically; so training data can also be derived from manuallabeling results by an artist.

After we have obtained the training data, an unsupervisedalgorithm is adopted to model the Gaussian mixtures [7].The training data is finally clustered into n classes (n=15 inour case). Figure 5 shows the mean (proto-lip) and relativeproportion for each class. Note that (~1, ~2,. . , pLn) is notthe linear decomposition in vector space because v may notlie in the span of (yi, ~2,. , hn). Since these templatesrepresent the lip configuration well, any new configurationcan be approximated by a linear combination of the proto-lips.

Given the proto-lips, the next step is to classify each lipconfiguration in the training data into different classes. Herewe use the Muhalanobis distance as the similarity measure,i.e.,

vz E class

if

E = argm&(v, - p~~lc)~R~~l(v~ - pk) ( I ; = 1.. n). (4)

Because each lip configuration corresponds to an 18-dimen-sional acoustic feature vector a = (ai, as,. , a~s)~, all thesamples in the audio vector space are also classified into nclasses, each of which is associated with a proto-lip template.We further assume that the random vector a, for class i hasa Gaussian distribution and each dimension in the vectordistributes independently. By regression, we can computethe mean tii, and covariance crt3 for each Gaussian model(for class i, dimension j)~ After the training process, wehave the following model parameters:

l Proto-lip templates pi and their relative proportion 7rTT2(i = 1 n)

l Mean iii, and covariance g2? for the j-th dimension ofi-th class for the acoustic feature vector (i = 1. n,j = 1.. .18).

5.3 Audio to Visual MappingGiven a new audio clip, we first segment the audio signalinto frames of 40ms each. Then the acoustic feature vectora for each frame is calculated as the system input. Sincewe do not have any information about the lip configurationwhen the speaker is silent, we assume that the mouth isclosed when the speech energy is below a predefined thresh-old. Otherwise, we approximate the lip configuration by alinear combination of the proto-lips. Since we assume each

According to Bayesian estimation, the posterior probabilityP(P&) is

p(pl,a) = nP(+4z(P~)(6)

~pM4Pw

where p(pL2) = 7r2 is the prior. Then the mapping resultbecomes

nv = ~w(k4a).

2=1Due to mapping error and the existence of noise, the syn-chronized sequence will appear to flutter open and closedinappropriately. Thus, we apply a Gaussian filter to thesynthesized sequence to achieve a smooth transition betweenneighboring frames.

6. Animation ResultsAfter we obtain the facial expression and lip-syncing results,the two components are combined to generate the final an-imation. Figure 6 shows three animation sequences gen-erated by our system: emotion-driven cartoon animation(A), speech-driven cartoon animation (B) and combinationof both (C). It can be seen that convincing facial expressioncan be generated according to the emotion recognition re-sults. In addition, lip configuration is synchronized with thecharacter’s speech.

7. Conclusions and Future WorkIn this paper, we have proposed a system to animate car-toon faces from speech with emotions. Our system consistsof two components: emotion-driven cartoon animation, andspeech-driven cartoon animation. First, a speech emotionrecognition system is developed. Using a corpus of over 1000utterances from different speakers, we have trained SVM’sto recognize four categories of emotions: neutral, happiness,anger and sadness. Given an input speech, the emotionrecognizer can generate a smooth transition curve for ev-ery emotion, which is further used to drive a cartoon facemodel. The cartoon face model consists of a small num-ber of hand-drawn templates. Facial expressions are thenanimated by morphing between these templates. Moreover,the lip shape of each cartoon frame is synthesized from theinput audio. Our lip-syncing algorithm uses acoustic sig-nals rather than conventional phoneme-viseme mapping andtherefore is language-independent and can run in real-time.The lip shapes are then composed with the morphed facialexpression images to form the final cartoon animation. Webelieve our emotion and speech-driven cartoon animationsystem will be very useful for HCI applications on desktopPCs and on the Internet.

While our cartoon results are encouraging, there remain anumber of areas to be further explored. First, the emotionrecognition can be improved. And more emotions such as

370

Figure 6: An example animation sequence. A. Facial expressions without lip-sync+ (emotion-driven cartoonanimation). B. Lipsyncing without facial expressions (speech-driven cartoon animation). C. Combination of