SSML Extension for Expressive Mandarin TTS

Shuang LiHongwu YangLianhong Cai

Tsinghua University

Outline

MotivationMotivation

Expression of SpeechExpression of Speech

Proposed SSML extension Proposed SSML extension

ConclusionConclusion

Motivation(1/3)

• Sentences with the same text can be expressed with different styles, emotions and moods

• Current tts system lacks variability

Motivation(2/3)• Current SSML cannot define speaking style, em

otion and mood– Good news: 生日快乐 “ Happy birthday”

expressed in happiness (emotion)

– Bad news: 张总去世了 “ Director Zhang passed away” expressed in sadness (emotion)

– Information provider: 飞往纽约的飞机将要起飞 “Flight for New York is going to take off”:

Expressed in a mild mood

– Dialog: 是中国队赢了吗？“Did Chinese team win?”: Emphasize “Chinese”, with interrogative mood

• Current SSML hard to show the difference between the expressions above

Motivation(3/3)

emotion

Positive, neutral, negative

style

news

Sports comment

dialog

Info providing

……

characteristic

Expressive speech

Emotion, style and characteristic are relatively independent but cannot be separatedCharacteristic and style: relatively stable and global featuresEmotion: short-time, local feature

Expressing pattern

No tag

Phisiological/social characteristics

Voice tag

Phisiological reactations

No tag

With different speaking stylesRepresenting speaker’s attitude, purpose and emotionMore harmonious with the circumstance

Outline

MotivationMotivation

Expression of SpeechExpression of Speech

Proposed SSML extension Proposed SSML extension

ConclusionConclusion

Expression of SpeechStyle ： speaking style( dialog, news, information providing…)Mood ： mood( request, acquisition, affirmation, apology…) Emotion ： emotional activities( neutral, negative, positive)

Mood Emotion

Intonation Emphasis

Speaking RateBreak

Spectral Features

Duration Energy Pitch

Style

Hierarchical framework of Prosody

• Break level– B0: no break– B1: Syllable – B2: Prosodic word– B3: Prosodic Phrase– B4: Breath Group– B5: Prosodic Group

• Chiu-yu Tseng,et al. Fluent speech prosody: Framework and modeling. Speech Communication, 46(2005) 284-399

我永远忘不了 <B3/25ms> 一张对日抗战时的新闻照片， <B3/507ms> 轰炸后的废墟焦土上，<B3/272ms> 一个衣不蔽体、 <B3/384ms> 满身尘土灰烟的幼儿 <B3/100ms> 坐在地上 <B3/75ms> 无助的大哭着。 <B5/1110ms> 那是一再令我热泪盈眶的镜头。 <B3/507ms> 新闻摄影中的战争传真 <B3/276ms> 已不能只称是照片了。 <B5/802ms>

• From Chiu-yu Tseng, report in Beijing University, Oct 11, 2005

Outline

IntroductionIntroduction

Expression of SpeechExpression of Speech

Proposed SSML extension Proposed SSML extension

ConclusionConclusion

Proposed tag （ 1/2 ）• Utterance: prosodic group, expressing a complete meaning

– Attributes:Style ： speaking style

Value ：News, Reading, Information provider, dialog, etc

Emotion: speaking emotion

Value ：Happy 、 Sad 、 Angry 、 Calm 、 Despair, etc

+1 for positive,0 for neutral, -1 for negative

mood ： speaking mood

Value ：given, request, acquisition, affirmation,apology, etc

Proposed tag （ 2/2 ）• BG: breath group

– attributes:intonation ：

Value ： indicative, interrogative, imperative

• PPh: prosodic phrase

• PW: prosodic word

• Syl: Syllable

Some examples(1/3)• <?xml version="1.0"?>• <speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis"• xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"• xsi:schemaLocation="http://www.w3.org/2001/10/synthesis• http://www.w3.org/TR/speech-synthesis/synthesis.xsd"• xml:lang=“zh-CN">• <utterence style=”information provide” emotion=”-1” mood=”apology”>• <bg intonation=” indicative”>• <pph>1121 次航班 (Flight 1121)</pph>• <pph> 延误 (has been delayed )• <pw><emphasis level=”strong”>1 小时 (for an hour )</emphasis></pw></pph>• <break strength=”medium”, time=”215ms”/>• <pph> 请旅客们到 (Please go to )</pph>• <pw><emphasis=”moderate”>G6</emphasis=”moderate”></pw>• <pph> 候机厅等候 (the waiting room)</pph>• </bg>• </utterence>• </speak>

Some examples(2/3)• <?xml version="1.0"?>• <speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis"• xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"• xsi:schemaLocation="http://www.w3.org/2001/10/synthesis• http://www.w3.org/TR/speech-synthesis/synthesis.xsd"• xml:lang=“zh-CN">• <utterence style=”dialog” emotion=”neutral” mood=”acquisition”>• <bg intonation=”interrogative”>• <pph><pw>• <emphasis level=”strong”> 张威 (Zhang Wei )</emphasis>• </pw></pph>• <break strength=medium time=75ms/>• <pph> 担心肖荫开车发晕 (is afraid of Xiao Yin being dizzy when driving

)</pph>• </bg>• </utterence>• </speak>

Some examples(3/3)• <?xml version="1.0"?>• <speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis"• xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"• xsi:schemaLocation="http://www.w3.org/2001/10/synthesis• http://www.w3.org/TR/speech-synthesis/synthesis.xsd"• xml:lang=“zh-CN">• <utterence style=”dialog” emotion=”angery”>• <bg intonation=”interrogative”>• <prosody rate=”x-fast”> 难道不是你的错吗？ (Isn’t it your fault? )• <break strength=”medium” time=”520ms”/>• </bg>• <bg intonation=”imperative”>• 以后你小心一点 (Be careful next time)• </bg>• </utterence>• </speak>

Outline

MotivationMotivation

Expression of SpeechExpression of Speech

Proposed SSML extension Proposed SSML extension

ConclusionConclusion

Conclusion & question?

• 5 elements for hierarchic prosodic structure– utterance, bg, pph, pw, syl

• 3 expressive attributes for utterance– style– emotion– mood

• 1 intonation attributes for bg– intonation