Pen ResearchPen Research

Jay PittmanDevelopment LeadTablet PC Handwriting RecognitionMicrosoft Corporation

Jay PittmanDevelopment LeadTablet PC Handwriting RecognitionMicrosoft Corporation

TDNNTDNN

Time-Delayed Neural Network

Ink is cut into segments via simple algorithmFor Latin script, we just cut at bottoms

For Arabic or Hindi, we may need a different algorithm

Features are computed per segmentChebychev coefficients

Variety of size metrics: width, height, offsets to neighbors, etc.

Each supported letter or character has an output nodeMany have 2 outputs, for beginning of character and continuation of character

Very large training sets are collected from tens of thousands of native writers

Time-Delayed Neural Network

Ink is cut into segments via simple algorithmFor Latin script, we just cut at bottoms

For Arabic or Hindi, we may need a different algorithm

Features are computed per segmentChebychev coefficients

Variety of size metrics: width, height, offsets to neighbors, etc.

Each supported letter or character has an output nodeMany have 2 outputs, for beginning of character and continuation of character

Very large training sets are collected from tens of thousands of native writers

Beam SearchBeam Search

Very similar to speech recognition systems

System lexiconSimple list of words (like a spellchecker)

Stored as a trie

Plus regular expressions for numbers, dates, times, currency, phone numbers, etc.

Low-scoring sequences are “culled” from the trie as we go

Converts character recognizer into word recognizerWe can recognize a word even if you mangle one of the letters

Supports sloppier (and therefore faster) handwriting

Very similar to speech recognition systems

System lexiconSimple list of words (like a spellchecker)

Stored as a trie

Plus regular expressions for numbers, dates, times, currency, phone numbers, etc.

Low-scoring sequences are “culled” from the trie as we go

Converts character recognizer into word recognizerWe can recognize a word even if you mangle one of the letters

Supports sloppier (and therefore faster) handwriting

88 8 682263574 4461575723

923151 9 4720

711252 8 7913

5318792857 6

……

…

1381 8 2 14 3

1717 5 7 4390

7 1657914415

Output Matrix

dog 68

clog 57

dug 51

doom 42

divvy 37

ooze 35

cloy 34

doxy 29

client 22

dozy 13

Ink Segments

Top 10 List

d 92

a 88

b 14

c 86

o 77

a 73

l 76

t 5

g 68

t 8

b 6

o 65

g 57

t 12

TDNN

SpaceTDNN

a

b

do

g

ab

t

tc

l

og

t

Lexicon

e

a

…

…

…

…

… Beam Search

a*a

d*d

g*g

o*o

TDNN + Beam SearchTDNN + Beam Search

PersonalizationPersonalization

New In Vista

Shape adaptationCollect samples from you

Simple idea: continue same training we do at Microsoft, but only on your samples

Implicit

Explicit

Text adaptationCollect your personal words from Word and outgoing emails

New In Vista

Shape adaptationCollect samples from you

Simple idea: continue same training we do at Microsoft, but only on your samples

Implicit

Explicit

Text adaptationCollect your personal words from Word and outgoing emails

Capital I?

dev RTM KKOMO dogfooding

Qi trie featurize Herry

Open SystemOpen SystemRecognizer API is published

Any recognizer may support this API

A non-Microsoft recognizer can be installed, and it will be invoked by the inking platform

Non-Microsoft recognizers available now in Japan, China, and Russia

Compete with my group

Or cover languages we don’t cover

Or cover non-text (music, math, chemistry)Sorry: recognition result API is very text oriented

Strings in Unicode

Perhaps your text might be XML

Recognizer API is publishedAny recognizer may support this API

A non-Microsoft recognizer can be installed, and it will be invoked by the inking platform

Non-Microsoft recognizers available now in Japan, China, and Russia

Compete with my group

Or cover languages we don’t cover

Or cover non-text (music, math, chemistry)Sorry: recognition result API is very text oriented

Strings in Unicode

Perhaps your text might be XML

http://www.microsoft.com/downloads/details.aspx?FamilyID=69640b5c-0ee9-421e-8d5c-d40debee36c2&displaylang=en

Microsoft Tablet PC / Tablet PC Platform / Programming the Tablet PC / Creating a RecognizerStart / All Programs / Microsoft Tablet PC Platform SDK / Microsoft Tablet PC Platform SDK Documentation

Latin OrthographyLatin Orthography

XP has U.S. English, U.K. English, German, French, Spanish, and Italian

Vista adds Dutch and Brazilian Portuguese

Currently working on Swedish, Danish, Norwegian (Bokmal), Finnish, Polish, Czech, Portugal Portuguese, Catalan, Romanian, Croatian, and Serbian

No ship estimates are available

Serbian is written in both Latin and CyrillicMaking plans for the next batch: Probably Bahasa Indonese (Indonesian), Hungarian, Turkish, Slovak, Slovene, Lithuanian, Estonian, Latvian, Vietnamese, Tagalog (Filipino), others TBD

World’s largest orthographyLargest count of languages

Largest combined count of literates

XP has U.S. English, U.K. English, German, French, Spanish, and Italian

Vista adds Dutch and Brazilian Portuguese

Currently working on Swedish, Danish, Norwegian (Bokmal), Finnish, Polish, Czech, Portugal Portuguese, Catalan, Romanian, Croatian, and Serbian

No ship estimates are available

Serbian is written in both Latin and CyrillicMaking plans for the next batch: Probably Bahasa Indonese (Indonesian), Hungarian, Turkish, Slovak, Slovene, Lithuanian, Estonian, Latvian, Vietnamese, Tagalog (Filipino), others TBD

World’s largest orthographyLargest count of languages

Largest combined count of literates

East Asian OrthographiesIdeographic OrthographiesEast Asian OrthographiesIdeographic Orthographies

Completely different code baseFocus on large count of characters

XP has Japanese, Chinese (Simplified), Chinese (Traditional), and Korean

Vista adds personalization, and improves cursive recognition

Completely different code baseFocus on large count of characters

XP has Japanese, Chinese (Simplified), Chinese (Traditional), and Korean

Vista adds personalization, and improves cursive recognition

Cyrillic and Greek OrthographiesCyrillic and Greek Orthographies

Same TDNN + beam search technology works equally well in Greek and Cyrillic

But we must collect new training sets

Cyrillic has more upper/lowercase confusion

Russian Collection completedCurrently working on recognizer

No estimate on shipping date

Some work also underway in SerbianCyrillic: Russian, Ukrainian, Bulgarian, Serbian, Byelorussian, Macedonian, Kazakh

Other former-Soviet Turkic republics are in a state of transition back to Latin scripts

Serbian is written in both Latin and Cyrillic

Same TDNN + beam search technology works equally well in Greek and Cyrillic

But we must collect new training sets

Cyrillic has more upper/lowercase confusion

Russian Collection completedCurrently working on recognizer

No estimate on shipping date

Some work also underway in SerbianCyrillic: Russian, Ukrainian, Bulgarian, Serbian, Byelorussian, Macedonian, Kazakh

Other former-Soviet Turkic republics are in a state of transition back to Latin scripts

Serbian is written in both Latin and Cyrillic

GreekGreek

ΕλληνικήέδωσανμουγλώσσαΤη

Ink by Dr. John Drakopoulos

I was given a language that is Hellenic

RussianRussian

Ink by Vladimir Smirnov

Кремль. Кремль, Ото слышал всех я про

Kremlin, Kremlin. I’ve heard from everyone about [it] Kremlin, Kremlin. I’ve heard from everyone about [it]

Bi-Directional OrthographiesArabic and HebrewBi-Directional OrthographiesArabic and Hebrew

Text written right-to-left, but numbers written left-to-rightArabic has its own digits

Hebrew uses “western” digits

No uppercase / lowercase distinction

Arabic is cursive-onlyUp to 4 forms per letter (initial, medial, final, isolated)

Hebrew is print-only5 letters have a separate final form

Abjabs (consonant alphabets)

We have collections underwaySome initial experimentation in both languages

No estimate on shipping dateArabic script variations used for Farsi (Persian), Urdu, Kurdish, Azeri (in Iran), Pashto, Darwi, Baluchi, Sindhi (in Pakistan), Kashmiri (in Pakistan)

Text written right-to-left, but numbers written left-to-rightArabic has its own digits

Hebrew uses “western” digits

No uppercase / lowercase distinction

Arabic is cursive-onlyUp to 4 forms per letter (initial, medial, final, isolated)

Hebrew is print-only5 letters have a separate final form

Abjabs (consonant alphabets)

We have collections underwaySome initial experimentation in both languages

No estimate on shipping dateArabic script variations used for Farsi (Persian), Urdu, Kurdish, Azeri (in Iran), Pashto, Darwi, Baluchi, Sindhi (in Pakistan), Kashmiri (in Pakistan)

Bi-Directional ExampleArabicBi-Directional ExampleArabic

١٢٣123

عمره[his] age

� عًامًاYear[s]

Ink by Ahmed Kamal

Bi-Directional ExampleHebrewBi-Directional ExampleHebrew

.מבבל

Plimpton 322 [is] from Babylonia.

322 פלימפטון

Ink by Ethan Zoller

Brahmic OrthographiesBrahmic OrthographiesLeft to right, no uppercase / lowercase

Abugidas (syllabic alphabets)Default vowel (short “Ə” or “uh” sound) is not written

All other vowels require a vowel sign added to the consonantThis includes a “no-vowel sign”

Called halant (“choked”) in Hindi, virama in other Indic languages

Vowel sound follows consonant sound

There are also independent vowels

Hindi Devanagari collection in progressSome initial experimentation

No estimate on shipping date

Brahmic scripts: Devanagari (Hindi, Marathi), Thai, Bengali, Gujarati, Gurmukhi (Punjabi), Tamil, Telugu, Kannada, Malayalam, Oriya, Sinhala, Khmer, Lao, Tibetan, Myanmar, Sindhi (in India), Kashmiri (in India)

Left to right, no uppercase / lowercase

Abugidas (syllabic alphabets)Default vowel (short “Ə” or “uh” sound) is not written

All other vowels require a vowel sign added to the consonantThis includes a “no-vowel sign”

Called halant (“choked”) in Hindi, virama in other Indic languages

Vowel sound follows consonant sound

There are also independent vowels

Hindi Devanagari collection in progressSome initial experimentation

No estimate on shipping date

Brahmic scripts: Devanagari (Hindi, Marathi), Thai, Bengali, Gujarati, Gurmukhi (Punjabi), Tamil, Telugu, Kannada, Malayalam, Oriya, Sinhala, Khmer, Lao, Tibetan, Myanmar, Sindhi (in India), Kashmiri (in India)

Vowel SignsVowel Signs

“KAA” “KU” “KUU”“KO” “KOW”

“KE” “KEE”“KA” “KAI”“KAY”

“KR”

“K”

Consonant ClustersConsonant Clusters

S T halant R I N Ghalant

Unicode buffer:

A transliteration of the English word “string”

Brahmic ComparisonBrahmic Comparison

K Kh G Gh C Ch

Devanagari: क ख ग घ च छBengali: ক খ গ ঘ চ ছGurmukhi: ਕ ਖ ਗ ਘ ਚ ਛGujarati: ક ખ ગ ઘ ચ છTamil: க சTelugu: క ఖ గ ఘ చ ఛKannada: ಕ ಖ ಗ ಘ ಚ ಛ

K Kh G Gh C Ch

Devanagari: क ख ग घ च छBengali: ক খ গ ঘ চ ছGurmukhi: ਕ ਖ ਗ ਘ ਚ ਛGujarati: ક ખ ગ ઘ ચ છTamil: க சTelugu: క ఖ గ ఘ చ ఛKannada: ಕ ಖ ಗ ಘ ಚ ಛ

Demo TitleDemo Title

NameTitleGroup

NameTitleGroup

Early FeedbackEarly Feedback

As for Vista, I am in love with the B2 Tablet implementation! I can see where the effort is going! The TIP is awesome and the handwriting recognition is vastly improved. There is hardly a time where I have to correct my input - no matter how sloppy I am.

Unsolicited feedback from a Vista Beta II user whose identity I do not know

© 2006 Microsoft Corporation. All rights reserved.Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation.Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft,and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Backup SlidesBackup Slides

Devanaguri ExampleDevanaguri Example

प“PA”

[R sign]period[abbreviation]

headlineheadline

ल“LA”

period[abbreviation]

headline

म“MA”

ि�[vowel sign]

“I”

� [vowel sign]

“AA”

ि�[vowel sign]

“I”

Pr. Ltd. [Private Limited]Pr. Ltd. [Private Limited]