Towards Conversational Speech Synthesis;

“Lessons Learned from the Expressive Speech Processing Project”

Nick Campbell

NiCT / ATR-SLCNational Institute of Information and

Communications Technology&

ATR Spoken Language Communication Research LabsKeihanna Science City, Kyoto 619-0288, Japan

nick@nict.go.jp, nick@atr.jp

The JST/CREST ‘ESP’ corpus

• The ATR “Expressive Speech Processing” project (JST/CREST) lasted from 4/’00 to 3/’05 and resulted in a corpus of 1,500 hours of natural conversational speech

• All recordings were transcribed, and about 10% are annotated for speaking-style, etc.

• The corpus is divided into 3 sections : i: esp_f, ii: esp_c, and iii: esp_m

Transcription example

匿名One “utterance” per line

Sections of the ESP corpus• esp_f

– one female speaker, head-mounted mic, 600 hours of daily spoken interactions, emotion/speech-act/etc …

• esp_c– 10 adult speakers, 5m 5f, 2 Chinese, 2 English, – 30-minute telephone conversations x 10 weeks– all conversations in Japanese, free content

• esp_m– multi-speaker, head-mounted microphones, variety of

interaction settings (like esp_f but many more voices)

finding #1: the Function of Conversational Speech

• To establish a rapport with the listener• To show interest and attention• To convey propositional content

• Contrast “broadcast mode” (one-way) with “interactive mode” (two-way) speech

• Speech Synthesis can do broadcast mode but Conversational Speech is two-way!

Backchannels & affect bursts

the hundred most common utterances

Non-Verbal Speech Sounds• Short, simple, repetitive noises• How they are spoken is usually more

important than what is being said

• Some examples:– The word ほんま– Means “really”– Used a lot in Osaka conversations …

Synthesis of Non-Verbal Speech Sounds

• The challenge now is how to synthesise these non-lexical speech sounds

• the same speaker says the same word in many consistently different ways …

• How should they best be (a) described (b) realised?

Tap-to-talk

http:feast.atr.jp/imode

Characteristics of Non-Verbal Utterances

• Better described by icons?

• Short, expressive sounds• Phonetically ambiguous• Prosodically marked

• Not well specified by text input!– But frequent and textually ‘transpar

ent’

‘Wrappers’ and ‘Fillings’ - Interaction Devices

• Often used as “edge-markers”– At beginning and end of utterance chunks

• Add expressivity to propositional content– Not just “fillers” –they ‘wrap’ the utterance

e.g., “erm, it’s very simple, you know”

The Acoustic features of Wrappers (and Fillers)

• Prosodically very variable in more than just pitch & duration …

• pca dimension reduction shows– 3 components account for more than 50% – 7 components account for more than 80% – Voice-quality comes up in the 1st component!

Voice Quality in Synthesis

• Chakai – affect-based unit selection

• using “whole-phrase” units• that vary according to expressivity• selected by their acoustics (princomps)

• They show affective relationships• and serve a pragmatic (phatic) function

Chakai

KeyTalk

Touch-sensitive Selection• One big advantage of using a midi keyboard is

touch-sensitivity – controlled sustain & attack

i.e., (perfect for the natural input of prosody)– with pitch-blend as well …

• Another is that keys can be intuitively grouped into related sets of utterances

Greetings replies opinion calling etc …

Octave or sub-octave clusters …. 5 & 7 black & white keys

Grouping Related Utterances• It remains as future work to group related

utterance types and plan a full keyboard for non-verbal speech sound synthesis

• Demo software is provided on the cd-rom proceedings – please let me know if you have any helpful ideas or suggestions

:-)

Summary

• Non-verbal sounds offer a challenge for the synthesis of interactive speech

• They are frequent and carry important affective and discourse-flow information

• Segments can be selected and reused from a conversational speech corpus

Conclusion • This paper has presented some examples

of non-linguistic uses of speech prosody

• Synthesis of expressive sounds is easy!– ‘Units’ can be whole phrases

• But unit selection is difficult!– They carry subtle differences of meaning– That can be very hard to specify in text

Listen:• Some examples of conversational speech

• (a) taken from the corpus (natural)• (b) synthesised using current technology• (c) concatenated from a very-large corpus

• Listen to the non-linguistic prosody!

NATRnext-generation

advanced text rendering

– The original dialogue – ditto - synthesised– CHATR (& original ）– NATR – large-corpus– NATR – more lively

Morning もしもしMorning もしもしHello こんにちはhi_there_ まいどHaha ハハハbeen_a_long_time 久しぶりですねー came_staight_to_the_eighth_floor もう直接、八階の方に、はいReally あ、そうなんReally あ、ほんまseventh_floor_today 七階すか、今日yeah_yeah うーん、そうそうHahaha ワーハハハー what_time_did_you_come 何時頃来たんすかjust_now さっきabout_now さっきぐらいウアハハハハハハ、まじで bit_late ちょっと遅なっ＜てんやん、アハjust_in_time ぎりぎりー not_really いや、そういうわけじゃないねんけどyeah_yeah_yeah はーいあいはいはい Umm うんYeah そっかそっかSo そう came_by_bike あたし自転車やからfrom_Kyooubashi 京橋のほうじゃなかったですっけReally そうやYeah でしょうUmm うんfrom_Kyoubashi 京橋からby_bike チャリンコですぐですか、あーそんなもんで来れるんやー Yeah そうYeah うん、だいたいReally あそうなーんすかYeah うん

7. Acknowledgements• This work is supported by the National Institute o

f Information and Communications Technology (NiCT), and includes contributions from the Japan Science & Technology Corporation 　 (JST), and the Ministry of Public Management, Home Affairs, 　 Posts and Telecommunications, Japan (SCOPE).

• The author is especially grateful to the management of ATR Spoken Language Communication Research Labs for their continuing encouragement and support.

Thank you

coming next:

Thank you