Localization and Internationalization

5 — Localization

From Code to Product gidgreen.com/course

Getting it wrong

From Code to Product Lecture 5 — Localization— Slide 2 gidgreen.com/course

Something we should know?


Lecture 5

•  Countries and languages •  Character sets •  Unicode •  Text localization •  Outsourcing translation •  Other localization


Population


China 1,347 M 19.3%

India 1,210 M 17.3%

USA 313 M 4.5%

Indonesia 238 M 3.4%

Brazil 192 M 2.8%

Pakistan 179 M 2.6%

Nigeria 162 M 2.3%

Russia 143 M 2.0%

Bangladesh 142 M 2.0%

Japan 128 M 1.8%

Mandarin 845 M 12.1%

Spanish 329 M 4.7%

English 328 M 4.7%

Hindi-Urdu 240 M 3.4%

Arabic 221 M 3.2%

Bengali 181 M 2.6%

Portuguese 178 M 2.5%

Russian 144 M 2.1%

Japanese 122 M 1.7%

Punjabi 109 M 1.6%

2011-2012 from Wikipedia

Economic weight (nominal)


USA $14.4 T 23.7%

Japan $4.9 T 8.1%

China $4.3 T 7.1%

Germany $3.7 T 6.0%

France $2.9 T 4.7%

UK $2.7 T 4.4%

Italy $2.3 T 3.8%

Russia $1.7 T 2.8%

Spain $1.6 T 2.6%

Brazil $1.6 T 2.6%

English $21.3 T 34.9%

Chinese $5.2 T 8.4%

Japanese $4.9 T 8.1%

German $4.4 T 7.2%

Spanish $4.2 T 6.8%

French $4.0 T 6.5%

Italian $2.5 T 4.1%

Russian $2.2 T 3.7%

Portuguese $1.9 T 3.1%

Arabic $1.9 T 3.1%

2008 from globalization-group.com, IMF

Internet users


China 485 M 36%

USA 245 M 78%

India 100 M 8%

Japan 99 M 78%

Brazil 76 M 37%

Germany 65 M 80%

Russia 60 M 43%

UK 51 M 82%

France 45 M 70%

Nigeria 44 M 28%

English 565 M 43%

Chinese 510 M 37%

Spanish 165 M 39%

Japanese 99 M 78%

Portuguese 83 M 32%

German 75 M 80%

Arabic 65 M 19%

French 60 M 17%

Russian 60 M 43%

Korean 39 M 55%

2011 from internetworldstats.com

Internet penetration


E-commerce volumes

$135B

$51B

$37B

$36B $28B $28B $19B

$16B

$15B

$13B

$123B

USA

Japan

China

Germany

France

UK

Italy

Canada

Spain

South Korea

Other


2009 from Everis

Multilingual countries


English 21M

French 8M

Canada

German

5.0M

French 1.6M

Italian 0.5M

Switzerland

Language variations

•  US vs UK English – color | colour – vacation | holiday – Where are you (at)?

•  European vs Brazilian Portuguese •  French •  Spanish


Language codes (ISO-639-1)


ar Arabic

fr French

nl Dutch

de German

he Hebrew

it Italian

ja Japanese

pl Polish

ru Russian

es Spanish

zh-CN Chinese (simplified)

zh-TW Chinese (traditional)

en-GB English (UK)

en-US English (US)

pt-BR Portuguese (Brazilian)

pt-PT Portuguese (Portugal)

es-AR Spanish (Argentina)

es-CL Spanish (Chile)

es-MX Spanish (Mexico)

es-ES Spanish (Spain)

Lecture 5



Computer representation

From Code to Product Lecture X — SUBJECT— Slide 14 gidgreen.com/course

0 1 0 0 0 0 0 1

0 … 65 … 255 .,/?;:’!%abcdefghijklmnopqrstuvwxyz… A …BCDEFGHIJKMNOPQRSTUVWXYZ0123456789

00 … 41 … FF

US-ASCII


Image from czyborra.com

ISO-8859-1


Windows-1252


ISO-8859-5


ISO-8859-8


Problems with character sets

•  Extra metadata •  Potential for misdisplay •  Mutually exclusive •  Little space to grow - e.g. € •  Ideographic languages – 70,000+ Chinese characters – Multibyte encoding


Lecture 5



The Unicode solution

•  One global character set – Over 110,000 characters – Over 100 alphabets

•  1,114,112 code points – 0…255 compatible with ISO-8859-1 – U+0041 = A

•  Multiple encodings


U+0000 … U+007F


U+0080 … U+00FF


U+0400 … U+047F


U+0590 … U+060F


U+4E00 … U+4E7F


U+2190 … U+220F


U+2800 … U+267F


UTF-16 encoding

•  2 or 4 bytes per code point •  Simple for U+0000…D7FF and E000…FFFF – “Basic Multilingual Pane”

•  Higher code points use 4 bytes •  U+FEFF = byte-order mark – No well-followed default

•  Windows APIs since Windows 2000 – Also .NET, Android, iOS, Mac OS X


UTF-8 encoding

•  1 to 6 bytes per code point •  1 byte for U+0000…007F – Perfect compatibility with ASCII

•  2 bytes for U+0080…07FF – etc…

•  Byte order mark allowed – But unnecessary, causes problems

•  Dominant on web, email


UTF-8 encoding


UTF-8 advantages

•  Natural compression for English •  English works in old tools/APIs – HTML tags unaffected

•  No shared values between byte types – Easy to synchronize mid-stream – Easy to search by byte value

•  No zero bytes (good for C) •  Byte-sorting = codepoint-sorting


Unicode on the web


Sour

ce:

goog

lebl

og.b

logs

pot.

com

Lecture 5



The original source code


function Check_Username(username) … if Username_Taken(username)… error="username is taken." … return error end function

And now in Spanish…

function Check_Username(username) … if Username_Taken(username)… error="username se toma." … return error end function


Internationalized

function Check_Username(username) … if Username_Taken(username)… error=Get_String("un-taken") … return error end function


Internationalized

function Check_Username(username) … if Username_Taken(username)… error=Translate("username is taken") … return error end function From Code to Product Lecture 5 — Localization— Slide 39 gidgreen.com/course

IDs vs English strings


IDs English strings

More compact code More explicit code

English can be changed Enforces sync between languages

Less error-prone Easier for third parties

Concatenation is evil

print Translate("You will travel from ") + from_city + Translate(" to ") + to_city


You will travel from London to Paris

Usted viajará de London a Paris

Sie wird von London nach Paris reisen

Substitutions


raw=Translate("You will travel from %from% to %to%") raw=replace(raw, "%from%", from_city) print replace(raw, "%to%", to_city)

You will travel from %from% to %to% Usted viajará de %from% a %to% Sie wird von %from% nach %to% reisen

Singular/plural

if (credits is 1) c_string=translate("1 credit")

else c_string=replace(translate("%#% credits",

"%#%", credits) raw=translate("You have %credits% left”) print replace(raw, "%credits", c_string)


You have 3 credits left You have 1 credit left

Text in images


Width in layouts أشكركم على الدفع. 感谢您的付款。 Gracias por su pago.

אנו מודים לך על התשלום. Спасибо за ваш платеж. Thank you for your payment. Vielen Dank für Ihre Bezahlung. Σας ευχαριστούµε για την πληρωµή σας. Nous vous remercions de votre paiement. お支払いしていただきありがとうございます。


+57%!

LTR / RTL


Lecture 5



Outsourcing translation

•  Preparing code •  Collecting (English) assets •  Choosing a provider •  Costs and quotes •  Glossary •  Translation memory •  Independent review


Collecting assets

•  Text files – Simple arrays or resource files – Standard formats, e.g. gettext, XLIFF

•  HTML files – Risk of accidental markup changes

•  Graphics files – Originals, not rendered

•  Think about text expansion


Choosing a provider

•  Problem: you can’t assess quality •  Go by reputation and clients – Examples of previous work

•  Ask who will actually do it – Native speaker of target language – Subject-specific experience

•  Consider future language needs


Cost and quotes


Ibidem-translations.com

•  Add 15-50% for specialized areas •  Clarify how words are counted •  Check for extra costs

Glossary

•  Fixed translation for specific terms – Control over branding – Domain-specific terminology – Consistency

•  Not-to-be-translated terms •  Requires thorough review of product


Glossary


Image from Google Translator Toolkit Help

Translation memory

•  Lots of translation is repetitive – Same text in many places – Small changes between versions

•  Same sentence = same translation – Save time and money – Help ensure consistency – But manual confirmation required

•  Should be owned by you


Translation memory


Imag

e fr

om k

ilgra

y.co

m s

cree

nsho

ts

Machine translation


Lecture 5



Numbers

1,234,567.89 — Japan, UK, USA 1 234 567,89 — France, Central Europe 1.234.567,89 — Germany, Scandinavia 1’234’567.89 — Switzerland 123,4567.89 — China 1’234,567.89 — Mexico 12,34,567.89 — India


Date and Times

7/21/2012 21/7/2012 21.7.2012 2012-07-12 7. 21. 2012 7-12-2012


15:45 3.45 PM 3:45 pm

Time zones


Map from wikipedia.org

Displaying times online

•  Store times independent of zone •  Options for display – Ask the user for their time zone – Show an explicit time zone – Use “ago” notation

•  Javascript to get from browser


Currencies

•  Biggest traded currencies: $ € ¥ £ – But there are almost 200

•  How to display – Number formatting – Symbols: ₪ ₩ ฿ $ – Currency codes: USD EUR JPY GBP CAD AUD

•  Also: currency conversion – Live feed, e.g. from ECB


Names

•  Surname can come first – China, Japan, Korea, Hungary

•  Multiple surnames – José Santos Tavares Melo Silva

•  Middle names/initials •  Double-barrelled names – Sarah-Jane Darlington-Whit

•  No spaces in CJK


Names


Full Name:

What should we call you?

Family name:

Other/given names:

•  Or localize based on language •  Do you need names at all? – Username or email can be enough

Addresses


John Doe Acme, Inc Suite 3B-3824 294 W Ronson Dallas TX 75211 USA

John Smith Acme, Ltd Flat 384 33 Walton Road Birmingham B26 3QJ UK

〒100-8994 東京都中央区八重洲一丁目5番3号東京中央郵便局 Tokyo Central Post Office 1-5-3 Yaesu, Chuo-ku Tokyo 100-8994 Japan

C/Pescadoro, 13, 2°, 3ª 28331 – Madrid Spain

Addresses

•  Single multi-line field •  Change in response to country •  Generic format


Phone numbers

UK: +44 (0) 123-456-7890 France: +33 1-23-45-67-89 China: +86 10-2345-6789 USA: +1 (123) 456-7890 x123


•  Country selector •  Change in response to country •  Generic format

Indexing, sorting, searching

•  Capitalization and accents – Øyvind matches oyvind?

•  Collation (sort order) – Swedish: a b c … x y z å ä ö – French: cote côte coté côté

•  CJK (ideographic languages) – No spaces between words – Sort based on stroke count


Paper sizes


US Legal 356 x 216 mm

US Letter 279 x 216 mm

A4 297 x 210 mm

Domain names

•  Country-code top-level domains – .fr .de .uk .in .br .jp .cn

•  Need separate registrar for many •  Some countries have restrictions – .com.au requires registered company – .ca requires nationality/residence – Also restricted: .fr .br .cn .ie .jp …

•  Internationalized domain names


And there’s more…

•  Units of measurement •  Colors •  Images of people •  Calendars •  Holidays •  Border disputes •  Culture •  Law From Code to Product Lecture 5 — Localization— Slide 71 gidgreen.com/course

Google in China

•  2005: Chinese language google.com •  2006: google.cn under censorship •  2009: China blocks YouTube •  2010: Google claims hacking attack – Redirects google.cn to google.com.hk – China blocks it for a day

•  Today: Baidu 79%, Google 17% – Baidu links to MP3/movie downloads


Getting real

•  It’s time consuming and costly •  Cheap wins in version 1.0 – Parameterize + functionize – Use Unicode throughout – Flexible layouts

•  See where there is demand •  Identify most important locales


Getting real

•  Don’t skimp the details – Needs to look native

•  Use serious service providers •  Prepare for tech support – Machine translation an option?

•  It will slow development – So wait for product maturity


Localization and Internationalization

Technology

Transcript of Localization and Internationalization