1

Unicode 101

Bouvet søkekollokvie, 2011-10-26Lars Marius Garshol, <larsga@bouvet.no>http://twitter.com/larsga

2

Agenda

• Scripts and languages– background for what follows

• Character encodings– basic concepts

• Unicode– it’s bigger than you think

• Programming– some practical lessons

3

A quick intro to grammatology

Scripts and languages

4

The beginning of writing

4000 BCE 3000 BCE 2000 BCE 1000 BCE

Sumerian cuneiform

Egyptian hieroglyphics

Chinese writing

5

Logographic scripts

• All of these first scripts are logographic– sometimes called “ideographic”, too

• One character – one word– 山 = mountain– 口 = mouth / door– 門 = gate

• Compounds for complicated concepts– 問 = question (mouth inside gate)– 酒店 = hotel (literally: alcohol shop)– 日本 = Japan (literally: sun root)

6

Simplified Chinese

• In the 1950s and 60s the Communist Chinese government simplified the shapes of many characters– these modified characters are used in

mainland China– Japan and Taiwan still use the original shapes

Character Traditional Simplified

Gate 門 门Country 國 国Vehicle 車 车East 東 东

7

Logographic scripts (2)

Good• Iconic and striking• Compact• Language-independent

(to some degree)

Bad• Hard to learn• Hard to write on

computers• Works poorly for

inflected languages• Sorting is hard

8

The next step: abjads

• The Egyptians later developed a script called “hieratic”– meaning “priestly writing”– oldest known example from 1600 BCE, but

must be older– actual origin kind of obscure

• It is an abjad– that is, an alphabet with only consonant signs– (and some logographic elements)

• A precursor to our own alphabet

9

Why only consonants?

• In Semitic languages everything revolves around “word roots”– these consist of three consonants– many, many words can be derived from the same root in

a systematic way– (the same applies to Egyptian)

• Example– s l m = peace– salaam = peace (related to Hebrew shalom)– islam = to have peace– moslem = one who has peace

• Abjads therefore work well for Semites– and not quite as well for others

10

Abjad family tree

Hieratic

Proto-Sinaitic

Ethiopic Phoenician

Greek Aramaic

Arabic HebrewKharoshthi

Much simplified. This familyhas many, many more scripts.

11

The invention of the alphabet

• The Greeks didn’t much like not having signs for vowels– so they invented them, thus giving us the

alphabet

• Salaam– in Arabic: سالم (m l s)– in Greek: σαλαμ (s a l a m)

• Grammatologists consider only scripts with both consonant and vowel signs alphabets– thus, there is no ”Arabic alphabet” or

”Chinese alphabet”

12

Alphabet family tree

Greek

Etruscan

Latin Armenian

Again much simplified.

Cyrillic Georgian

13

Alphabetic order

14

ABG?

• The Etruscans had no g sound, so they pronounced the third character k– the Romans inherited this

• Thus, in Latin “C” was used for both k and g– this left Spurius Carvilius Ruga with a bit of a

problem– he had to write his surname as “Ruca”– (Latin “ruca”: to fart)– so he invented “G” (C with a stroke)

15

Backtracking

Hieratic

Proto-Sinaitic

Ethiopic Phoenician

Greek Aramaic

Arabic HebrewKharoshthi

16

Kharoshthi

• A script developed for writing Sanskrit in India– 4th century BCE, or thereabouts

• Sanskrit is Indo-European, like Greek– hence, the absence of vowels is a major

problem– however, the Indians came up with a different

solution

• This is an abugida– Ethiopic, too

17

The descendants of Kharoshthi

• The Indian subcontinent has a large number of scripts– all are descended from Kharoshthi and follow

the same basic model

• These scripts also spread into Indo-China and beyond to Indonesia– Thai, Khmer, Javanese, Tibetan, etc etc

• Total number of users must approach 2 billion

18

Back to China

• Japanese is an inflected language– therefore developed extra characters for writing

grammatical endings

• The Vietnamese later moved on to Latin characters under French influence

• The Koreans did something completely different...• Leaving out a number of minor ethnic scripts here

Chinese

Japanese Korean Vietnamese

19

Hangul

• On October 9 1446, king Sejong announced the introduction of a new script: Hangul– it was designed to be easier to learn than

Chinese characters– it was also designed to be a better fit to the

Korean language

• Hangul is like an alphabet, but better– wikipedia: Numerous linguists have praised

Hangul for its featural design, describing it as "remarkable", "the most perfect phonetic system devised", and "brilliant, so deliberately does it fit the language like a glove.”

20

Hangul design

• Vowels have different shapes from consonants– these follow the shape of the vocal cords when

speaking the vowel

• Consonants have a different system

Follows Chinese convention of equal-sizedboxes for all characters. Each box containshangul letters for one syllable.

21

Syllabaries

• Like abugidas, but unsystematic– that is, every letter combination must be

learned by rote

22

1800 – present

• Cover this?

23

Kinds of scripts

wordfeature

what basic shapes correspond to

letter syllable

featural alphabet syllabary logographic

abjad

abugida

24

Text directions

• LTR top-down– Latin, Greek, Cyrillic, ...

• RTL top-down– Arabic, Hebrew,

Syriac, ...

• Top to bottom, left to right– Monglian, Uighur,

Buryat, ...

• Top to bottom, right to left– Nushu, Rong

• Bottom up, left to right– Some minor Indonesian

abugidas

• Bottom up, right to left– one minor Moslem

Chinese abjad

• Upwards boustrophedon– Ogham (ancient Irish

runes)

25

Combining characters

• In Arabic, the same letter can have up to four different shapes– depending on its position in the word

26

Basics

Character encoding

27

Two key concepts

• Character set– a function number -> character– usually with a limited, fixed number of characters

• Character encoding– a mapping from a bit stream to a sequence of

numbers– the numbers, of course, refer to characters in

some character set

• Example– UTF-8 is an encoding for Unicode– UTF-16 is another

28

Kinds of encodings

• Single-byte– each byte is a number 0-255. end of story– ISO 8859-x

• Double-byte– each word (2 bytes) is a number 0-65536. end of story– UCS-2

• Variable length– more complex rules (UTF-8)

• Escape code-based– uses escape codes to change between different modes– ISO 2022

29

ASCII

• The mother of all character sets– 7-bit

• Nearly all character sets today are ASCII subsets

• The exception is EBCDIC– mostly used by IBM mainframes– also a terrible design

30

Code pages

• Primitive character set solution on IBM PCs

• Basically, changing the code page would change the system font– 65 would always be ‘A’– 216 could be ‘Ø’, Cyrillic, Greek, ... depending

on code page

• Essentially, swapping code page would change the contents of all text files...– awful for text processing software

31

ISO 8859-x series

• Lower 128 characters is ASCII• Higher 128 characters are language-

specific• Now obsolete, thanks to Unicode• Microsoft have their own extensions

– Windows-12xx, add extra characters where 8859 have obsolete control codes

1 Western Europe 5 Cyrillic 9 Turkish 13 Baltic

2 Central Europe 6 Arabic 10 Nordic (Sami) 14 Celtic

3 South Europe 7 Greek 11 Thai 15 Latin-1 ++

4 North Europe 8 Hebrew 12 Doesn’t exist 16 Latin-3 ++

32

The Far East

• Generally, one character set per country– JapanJIS X 0208– Korea KS X 1001 (and 1003)– China GB 2312– Taiwan ???

• Combined with different character encodings– ISO 2022 (-JP, -KR, -CN)– EUC (-JP, -KR, -TW)

• Additional variants– Shift-JIS (Japanese, from Microsoft)– Big5 (Taiwanese)– ...

33

China

• Doesn’t want to use Unicode• Instead introduced GB 18030

– takes GB 2312, then adds Unicode after the GB 2312 part

– requires a mapping table for lower part– higher part can be mapped to Unicode

algorithmically

34

VISCII

• Vietnamese character set– tries hard to maintain ASCII compatibility, but

there are just too many Vietnamese characters...

35

Unicode

36

Unicode

• The character set to end all character sets

• Before, there was at least one character set for each script– generally, it would have Latin plus one more

script– software therefore had to support many

different internal representations of text

• Now, Unicode supports every character that’s ever appeared in a character set anywhere– therefore, it’s the only character set you need

37

Origin• 1987

– Engineers at Xerox and Apple discuss the possibility of a universal character set

– they investigate, and decide it’s feasible

• 1988– tentative proposal for a 16-bit character set

• 1989– Unicode Working Group set up– all of ISO 8859 added

• 1990– many more people join– Chinese characters added

• 1991– Unicode Consortium founded– Unicode 1.1 released

• 1992– ISO 10646 killed off, and replaced by Unicode

38

Design goals

• Universal– should be the only character set ever needed

• Semantics– characters should have well-defined semantics– Ø ≠ ∅

• Dynamic composition– characters can be composed dynamically

• Convertibility– every character in an existing character set,

must have a single corresponding character in Unicode

39

Structure

• Originally intended to be 16-bit– 0x000 – 0xFFFF– explicit rationale: enough to encode all

characters in daily use– implicit: not excessive use of space

• Unfortunately, this is not nearly enough– decided to expand it in 1996– keep the 16-bit structure– original range becomes Basic Multilingual Plane– each stretch of 0xFFFF characters is a plane– 17 planes (0-16) in all

40

Contents

• As of Unicode 6.0– more than 109,000 characters– 93 different scripts

41

The planes

• Plane 0: BMP – Nearly the only one needed

• Plane 1: Supplementary Multilingual Plane– mostly historical scripts and weird symbols

• Plane 2: Supplementary Ideographic Plane– historical Chinese characters

• Panes 3: Tertiary Ideographic Plane– not in use, reserved for ancient Chinese characters

• Planes 4-13: Unused• Plane 14: Supplementary Special-purpose Plane• Planes 15-16: Private Use Area

42

Basic Multilingual Plane

0xxx 1xxx 2xxx 3xxx 4xxx 5xxx 6xxx 7xxx 8xxx 9xxx Axxx Bxxx Cxxx Dxxx Exxx Fxxx

x1FF

x3FF

x5FF

x7FF

x9FF

xBFF

xDFF

xFFF

Latin

CJK

CJK CJK CJK CJK CJK CJK

Han

gul

Han

gul

Han

gul

Han

gul

CJKSyll.

Yi

Surro

gates

Priv

ate use

Priv

ate use

Stuff

Scripts

Scripts

Symbols

Sym

Latin

Braille

Misc

43

Plane 10xxx 1xxx 2xxx 3xxx 4xxx 5xxx 6xxx 7xxx 8xxx 9xxx Axxx Bxxx Cxxx Dxxx Exxx Fxxx

x1FF

x3FF

x5FF

x7FF

x9FF

xBFF

xDFF

xFFF

LTR

RTL

Indic

African

Conlang

Neareast

Hieroglyphics

Hieroglyphics

Hieroglyphics

Undeciph

Undeciph

NorthAm.

Sumerian

Notational

Notational

Notational

LargeAsian

LargeAsian

LargeAsian

LargeAsian

LargeAsian

44

Encodings

• UCS-2– from ISO 10646– two bytes per character– can only encode the BMP

• UCS-4– also from ISO 10646– four bytes per character– can encode the whole thing

• UTF-32– same as UCS-4

45

UTF-16

• Like UCS-2– but extended with a trick to cover the full set

• Surrogates– a block of code points set aside specifically for

UTF-16

• Each non-BMP character is written as two surrogates, one low and one high– first 10 bits (0-03FF) added to D800 = first two

bytes– next 10 bits added to DC00 = next two bytes

• So, the characters become:– 1101 10xx xxxx xxxx 1101 11xx xxxx xxxx

46

UTF-8

• Cleverly designed variable-length encoding

• ASCII is encoded as ASCII• Can encode all of Unicode as 4 bytes

– whether more or less compact than UTF-16 depends on the text being coded

– for files UTF-8 is usually far more compact

47

UTF-8

• Far and away most used Unicode encoding– because of compatibility with ASCII

• Easy to recognize bit patterns– Lett Ã¥ kjenne igjen = UTF-8 interpreted as

8859-1– æøå = æøå– ÆØÅ = ÆØÅ

48

Reading UTF-8 as UTF-16

• Two bytes get treated as a single character– effectively turns it into a random character

As UTF-8 As UTF-16

49

Han unification

• Not only are there many Chinese characters– there are different variants of each character– differences between China (traditional &

simplified), Japan, and Korea (also Vietnam)

• Unicode has decided to encode these only once– different renderings are considered visual

differences only

• This is quite unpopular in the Far East– particularly in Japan

50

Too many characters

• Latin characters have a nearly infinite number of variants:

• New ones pop up all the time– these can’t all be encoded

a á à â

ã ä å ā

ą ă ậ ẩ

ạ aː ȧ ...

51

Solution: combining characters

• What if I needed Z with stroke, cedilla, and umlaut?

• Simple, encode as– U+01B5 LATIN CAPITAL LETTER Z WITH

STROKE– U+0327 COMBINING CEDILLA– U+0308 COMBINING DIAERESIS

• Norwegian å should actually be written– a + combining ring

52

Many ways to write a character

• Unicode has inherited precomposed characters (like å) from older character sets– these are all included, for ease of

roundtripping

• Unicode normalization provides ways of streamlining this– unfortunately, it’s complex, with numerous

different normalization forms– won’t go further into it

53

Unicode Character Database

• Unicode contains more than just the characters– there is a whole database of characters with many

fields

• It contains things like– names for each character– decomposition mappings– deprecation mappings– case mappings– breakdown into blocks– category for each character– numeric value– what script the character belongs to– ...

54

Uses for UCD

• Matching in regular expressions– by character category– by script– ...

• Upper- and lower-casing of strings– beware, this is complicated...

• Stripping accents– use decomposition mappings in UCD

• ...

55

More stuff in the Unicode standard

• Guidance on upper/lower-casing– tricky, because there are national variations

• Unicode Normalization• Sorting algorithm• Regular expression guidelines• Line breaking algorithm• Bidirectional text display (Arabic)

56

Dealing with characters

Programming

57

The key principle

• One internal representation for text– used everywhere, with no deviations– text from outside must always be converted

• Modern programming languages enforce this– char/String vs byte (in Java)– Stream vs Reader/Writer (also Java)

• Older languages do not– C char has no defined representation

58

In an ideal world

• The internal encoding of strings should not be visible– they should just be sequences of Unicode

characters

• In practice, this turns out to be difficult– string.charAt(): what should this return?– string.length(): what should this return?– etc

59

Internal encodings

• C strings are byte arrays• C++ bytes, UTF-16, or UTF-32• Java UTF-16• .NET UTF-16• Python UTF-16• Ruby ???• JavaScript UTF-16 or UCS-2 (poorly

defined)• PHP strings are byte arrays1)

• Perl UTF-81) http://kore-nordmann.de/blog/php_charset_encoding_FAQ.html#which-charset-encoding-do-strings-have-in-php

60

Java

• String.charAt(int ix)– returns the UTF-16 code unit (word) at that

index

• String.codePointAt(int ix)– like charAt, but if it’s a high surrogate,

returns code point by combining with charAt(ix + 1)

• String.length()– returns the number of UTF-16 code units in

the string representationFew programming languages document thebehaviour of the String class well enough forthis to be clear...

61

How long is a string?

String str = "\u01B5\u0327\u0308";System.out.println(str.length());

Output is 3, even though there is just a single,combined character.

62

Find the bug

import java.io.*;

public class Cat {

public static void main(String[] argv) throws IOException { BufferedReader in = new BufferedReader(new FileReader(argv[0])); String line = in.readLine(); while (line != null) { System.out.println(line); line = in.readLine(); } }}

Q: What encoding are we reading?A: We have no idea.

63

How to solve

• Either find some way to auto-detect encoding– requires you to know the syntax of the file– and that syntax to have auto-detect rules

• Or find some way for the user to specify the encoding– for example a command-line parameter

64

Find the bug, 2

public HttpResponse get(String request) throws IOException { InputStream responseContent = null; HttpGet httpGet = new HttpGet(request); HttpResponse response = new DefaultHttpClient().execute(httpGet); responseContent = response.getEntity().getContent(); return new HttpResponse(response.getStatusCode(), response.getStatusLine().getReasonPhrase(), read(new InputStreamReader(responseContent, "UTF-8"))); }

Q: How do we know this is UTF-8?A: We don’t.

65

How to solve

• Need to look at the Content-type– “Content-type: text/plain; charset=iso-8859-1”

• Occasionally, it can be even harder– MIME-type specific rules for deciding charset

if not specified in request

• Best solution (for a generic Get class) is to return the stream (not the reader)– and provide enough info for clients to figure

out the encoding

66

URI vs IRI

• Originally, the character encoding of URIs was not defined– characters must be ASCII, or %-escaped– however, character set of %-escapes not defined – this was RFC 2396

• Then, RFC 3986 defined %-escapes as being UTF-8– explicit characters must still be ASCII only

• RFC 3987 introduced IRIs– here, everything is UTF-8– non-ASCII characters do not need to be escaped

67

How to distinguish IRIs and URIs

• Well, uh, you can’t, really...– if there are non-ASCII characters it’s probably

an IRI

• Specifications can decide to support IRIs– for example, in XTM it’s all IRIs

• So the context can tell you, in some cases

If it sounds like a mess, that’s because it is...

68

XML and Unicode

• One of the good things about XML is that it gets Unicode right– text coming out of an XML parser is always

Unicode– unless the author has made a stupid mistake,

there will be no character encoding problems

• Does this via– syntax for declaring encoding in the document,– careful, explicit rules for detecting encoding,– escape syntax for Unicode characters, and– well-designed rules for Unicode characters in

identifiers and elsewhere

69

Representing characters in XML

• Don’t ever use entity references– &aring; and similar are the spawn of the devil– they require the DOCTYPE to be downloaded

• Use UTF-8 as the character encoding– and declare it in the <?xml, or omit entirely– this way all characters can be expressed

directly

• If you absolutely must use stupid tricks, use &#XXXX; character references– it’s better to avoid these, but in special cases

(human authoring) they can be useful