Neural Electroencephalography (EEG) - Oxford …teh/teaching/smldmHT2014/W2slides.pdf ·...
62
Neural Electroencephalography (EEG) http://newton.umsl.edu/tsytsarev_files/Lecture02.htm 70
-
Upload
nguyencong -
Category
Documents
-
view
224 -
download
0
Transcript of Neural Electroencephalography (EEG) - Oxford …teh/teaching/smldmHT2014/W2slides.pdf ·...
Neural Electroencephalography (EEG)
http://newton.umsl.edu/tsytsarev_files/Lecture02.htm 70
Neural Spike Waveforms
71
Pairs Plot of Principal Components
!5
0
5
!20
!10
0
10
!10
!5
0
5
!5 0 5!5
0
5
!5 0 5
!5 0 5
!20 !10 0 10
!20 !10 0 10
!10 !5 0 5
!10 !5 0 5
!5 0 5
!5
0
5
!5
0
5
!20
!10
0
10
!10
!5
0
5
72
Clustering
� Many datasets consist of multiple heterogeneous subsets. Clusteranalysis is a range of methods that reveal this heterogeneity bydiscovering clusters of similar points.
� Model-based clustering:� Each cluster is described using a probability model.
� Model-free clustering:� Defined by similarity among points within clusters (dissimilarity among points
between clusters).� Partition-based clustering methods:
� Allocate points into K clusters.� The number of cluster is usually fixed beforehand or investigated for various
values of K as part of the analysis.� Hierarchy-based clustering methods:
� Allocate points into clusters and clusters into super-clusters forming ahierarchy.
� Typically the hierarchy forms a binary tree (a dendrogram) where eachcluster has two “children”.
73
Hierarchical Clustering
� Hierarchically structured data can be found everywhere (measurementsof different species and different individuals within species), hierarchicalmethods attempt to understand data by looking for clusters.
� There are two general strategies for generating hierarchical clusters.Both proceed by seeking to minimize some measure of dissimilarity.
� Agglomerative / Bottom-Up / Merging� Divisive / Top-Down / Splitting
Hierarchical clusters are generated where at each level, clusters arecreated by merging clusters at lower levels. This process can easily beviewed by a dendogram/tree.
74
Measuring DissimilarityTo find hierarchical clusters, we need some way to measure the dissimilaritybetween clusters
� Given two points xi and xj, it is straightforward to measure theirdissimilarity, say d(xi, xj) = �xi − xj�2.
� It is unclear however how to extend this to measure dissimilarity betweenclusters, D(Ci,Cj) for clusters Ci and Cj.
Many such proposals though no concensus as to which is best.(a) Single Linkage
D(Ci,Cj) = minx,y
(d(x, y)|x ∈ Ci, y ∈ Cj)
(b) Complete Linkage
D(Ci,Cj) = maxx,y
(d(x, y)|x ∈ Ci, y ∈ Cj)
(c) Average Linkage
D(Ci,Cj) = avgx,y (d(x, y)|x ∈ Ci, y ∈ Cj)
75
Measuring Dissimilarity
76
Hierarchical Clustering on Artificial Dataset
−20 0 20 40 60 80 100
−40
−20
020
4060
80
V1
V2
1
2
34
5
6
78
910
11
12
13
14
15
16
1718
19
20
21
22
23
24
25
26
2728
29
30
31
3233 34
3536 3738
3940
4142
43
44 4546
47
4849
50
51
52
535455
56
5758
59
60
6162
63
64
65
66 67
68
6970
71
7273
747576777879
80
8182
8384
85
86
87
888990
91
92
93
94
95
96
97
9899
100101
102103
104
105106
107
108109
110
111112
113114
115
116117
118
119
120
121122
123124
125
126127
128129
130
131132
133
134135
136137
138 139
140141142
143144 145
146
147
148149
150151
152
153
154
155156
157
158
159160
161162163
164
165
166
167 168
169
170171172
173
174
175176 177
178179180
181 182183
184
185
186
187188
189190191
192193 194195
196 197198
199
200201
202
203204
205
206
207208
209210 211
212
213
214
215
216217
218
219 220
221 222223
224225
226
227228229
230231
232233
234
235
236237238
239
240241242
243
244
245246247
248
249
250251
252
253 254 255256257
258259
260261
262
263
264 265
266
267268
269270
271272
273274
275
276277
278
279
280
281282
283284285286
287
288
289 290
291
292
293
294
295296
297
298299300 301
302
303
304305
306
307308
309
310
311
312313314
315
316
317
318319
320
321
322
323
324
325
326
327
328
329330
331
332 333334 335 336337338339
340
341
342 343344345
346347
348349
350351
352
353
354
355
356357
358
359
360
361
362
363364
365
366
367
368369
370371372
373
374375
376377 378
379380
381382
383384385
386
387
388
389
390391
392393
394
395
396 397398399
400
401402
403
404
405406
407
408409
410
411412413
414
415
416
417 418419420
421
422
423424
425426
427
428
429
430
431432433
434
435
436437
438
439440441 442
443
444445
446447 448
449
450
451
452453
454
455
456
457458
459460 461
462
463464465466 467
468
469
470
471
472
473
474
475476
477478 479480
481
482
483
484
485
486
487488
489
490491 492
493
494
495496
497
498499
500501
502
503 504
505
506507
508509
510
511
512
513
514
515
516
517 518519
520521 522
523
524
525
526
527
528
529530
531
532
533534
535
536
537
538539 540
541542
543544
545546
547
548549
550
551552
553
554
555556557
558
559
560
561
562
563
564
565
566567
568
569570
571 572573
574
575576577578
579
580581
582 583
584585586587
588
589590
591
592593 594
595596 597
598
599600
601
602603604605
606
607
608
609
610611
612
613
614615
616617
618619
620
621622
623 624
625
626627
628629
630
631632
633
634 635636637
638
639
640641642643 644
645
646647
648
649650
651
652653
654
655
656657
658659660
661
662663
664 665666
667
668669
670
671672
673674
675 676
677
678
679
680
681
682683
684
685
686
687688
689
690691
692
693
694
695696
697698699700701
702
703 704
705
706707
708
709
710
711
712
713
714
715
716717
718
719
720
721722
723 724725
726
727728729
730731
732
733
734
735
736
737738
739
740
741
742743
744
745
746
747748
749
750751 752753
754755756757
758
759
760
761762
763764
765
766767768769
770771772 773
774
775
776777
778
779780 781
782
783784
785786787
788
789
790791
792793794 795
796797
798
799
800801
802803 804
805
806
807
808
809810811
812
813814
815
816
817
818819
820821
822823
824825
826 827828
829
830
831
832 833834
835
836
837838839 840
841
842
843844
845
846
847848
849
850851
852
853854
855856
857 858
859860861
862
863
864
865866 867
868
869
870
871872 873874
875876
877878879880
881882
883
884885886 887
888889
890
891
892893
894895
896897
898
899
900
901
902
903904905
906
907 908
909
910
911912
913
914
915916
917
918919920
921922
923
924
925926
927
928929930 931
932
933
934
935
936937
938
939
940
941942
943944
945
946
947
948
949
950951952
953
954955
956957
958959960
961
962
963
964965
966967 968969
970
971972
973974
975976977
978979
980
981
982983984
985
986
987
988
989990
991
992
993
994995
996997998
9991000
1001
1002
1003
1004
100510061007
1008 1009
1010
101110121013
1014
1015
10161017
1018
1019
1020
10211022
10231024
1025102610271028
1029 103010311032
10331034 1035
1036
1037
1038
1039
1040
10411042
10431044
1045
10461047
10481049
1050
1051
1052
1053
10541055
1056
1057
10581059106010611062
1063
1064
1065 1066
1067
1068
1069
1070
1071
1072
1073
1074
1075 1076
1077
1078
1079
1080
1081
1082
1083
10841085
1086
1087
1088
1089
109010911092
1093
10941095
1096 10971098
109911001101
1102
1103 110411051106
1107
11081109
1110
1111
111211131114
1115
1116
1117 11181119
1120
11211122
11231124
11251126
1127
1128
11291130
1131
11321133
1134
1135 1136
1137
11381139
1140
11411142 11431144
1145
1146
1147
11481149
1150
11511152
1153
1154
11551156
1157
1158
11591160
11611162
1163
1164
116511661167
11681169
1170
1171117211731174
1175
117611771178
1179
11801181
118211831184
11851186
1187
11881189
1190
1191
1192
1193
1194
1195
1196
119711981199
1200
1201
12021203
12041205
12061207
1208
12091210
1211
12121213
1214
1215
1216
12171218
1219
1220
122112221223
1224
1225
12261227
12281229
1230
1231
12321233
1234
1235
1236
1237
1238
1239124012411242
1243
1244
124512461247
12481249
1250
1251
1252
1253
125412551256
1257
12581259
1260126112621263
1264
12651266 1267
12681269
12701271
12721273
1274
1275
12761277
12781279 12801281
1282
1283
12841285
1286
1287
1288
1289
1290
1291
1292
12931294
12951296
1297
1298
129913001301
13021303
13041305 13061307
1308
13091310
13111312
131313141315
131613171318
1319
1320
13211322
1323
1324
1325
13261327
1328 132913301331
13321333
1334
1335
1336
1337
1338
1339134013411342
13431344 1345
1346
1347
134813491350
1351
1352
1353
1354
13551356
13571358 1359
1360
13611362
1363
1364
1365
13661367
136813691370 1371
1372
1373
1374
137513761377
1378
1379
1380
13811382
1383
13841385
1386
1387
13881389
139013911392
13931394
1395
13961397
13981399
14001401
14021403
1404
1405140614071408
1409
1410
1411
14121413
1414
14151416
1417
1418
14191420
14211422
14231424
1425
1426
142714281429
1430
1431
1432
14331434
14351436
1437
1438
143914401441
1442 1443
1444
1445
1446
1447
14481449
1450
1451
1452
145314541455
1456
1457
14581459
146014611462
14631464146514661467
1468
1469
1470
1471
14721473
1474 1475
1476
1477
14781479
14801481
14821483
148414851486
1487 1488
1489
1490
1491
14921493
14941495
1496
1497
14981499
1500
15011502
1503
1504
150515061507
1508
1509
1510
15111512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
15241525
15261527
15281529
15301531
15321533
15341535
1536
1537
1538
1539 1540
15411542
1543154415451546
154715481549 1550
15511552 1553
1554
1555
1556
1557
1558
15591560
15611562
15631564
1565
1566
1567
1568
1569
1570 15711572
1573
1574
1575
1576
15771578
1579
15801581
1582
1583
1584
158515861587
1588
15891590
1591
15921593
159415951596
1597
159815991600
1601
1602 1603
16041605
1606
1607
1608
1609
1610
1611
161216131614
16151616
16171618
1619 16201621
1622
1623
16241625
16261627
162816291630
16311632
1633
1634
16351636 1637
16381639
1640
1641
1642
1643
1644
1645
1646
16471648
1649165016511652
1653
1654
1655
1656
1657
1658
16591660
1661
166216631664
16651666
16671668
1669 1670
1671
1672
16731674
1675
167616771678
1679
1680
1681 1682
1683 1684
1685
16861687
1688 16891690
1691
1692
16931694
1695
1696
1697 169816991700
17011702
17031704
17051706
1707
1708
1709
17101711
17121713
1714
17151716
1717
1718
17191720
1721
1722
1723
1724 17251726
1727
1728
17291730
17311732
1733
1734
1735
1736 17371738
1739 1740
174117421743
1744
174517461747
1748
1749
1750
17511752
1753
1754
1755
17561757
1758
1759176017611762
1763
1764
1765
1766
1767
1768
1769
1770
1771
17721773
17741775
1776 1777177817791780
1781
17821783 17841785
17861787
1788
178917901791
179217931794
17951796
1797
1798
1799
1800
1801
18021803
18041805 1806
1807
1808
18091810
18111812
1813181418151816
1817
18181819
1820
1821
1822
1823
182418251826
1827
1828
1829
1830
183118321833
1834
183518361837
183818391840
1841
1842 1843
1844
1845
1846
18471848
1849
1850
1851
18521853
1854185518561857 1858
1859
1860
18611862
18631864
1865
18661867
186818691870
1871
1872
1873
1874
18751876
18771878
1879
188018811882
1883
18841885
18861887
18881889
18901891
18921893
1894
18951896
18971898
18991900
19011902
1903 1904
1905
1906
190719081909
1910
19111912 19131914
1915
1916
19171918
1919
1920
1921
1922
1923
1924
1925
19261927
1928
1929
1930 1931
1932
1933
19341935
1936
19371938 193919401941
1942
1943 19441945
19461947
19481949
1950
19511952
1953
1954
1955
195619571958
1959
19601961 1962
1963
1964
19651966
1967
196819691970
1971 1972
1973
19741975
1976
1977
19781979
1980 19811982
19831984
1985
19861987
1988
19891990
19911992
1993 1994 1995
19961997
199819992000 2001
20022003
2004
20052006
2007
20082009 2010
2011
2012
2013
2014
2015
2016
201720182019
2020 20212022
2023 20242025
20262027
2028
2029
2030
2031
2032
20332034
2035
20362037
2038
20392040
2041
2042 20432044
204520462047
20482049
2050
2051
20522053
2054
2055
2056
2057
20582059 2060
2061
206220632064
206520662067
2068
20692070
2071
20722073
2074
207520762077
2078
2079
2080
2081
2082
2083
208420852086
2087
2088
20892090
20912092
2093
209420952096
20972098 2099
2100
2101
2102
2103 2104
2105
2106
2107
21082109
2110
211121122113
21142115
2116
21172118
2119
2120
2121
212221232124
2125
2126
2127
2128
2129
21302131
2132
2133
2134
2135
2136
2137
21382139
2140
2141
2142
2143
21442145
2146
2147
21482149
2150 21512152
2153
2154
2155
2156
2157
2158
2159
2160
2161
21622163
2164
21652166
2167
2168
21692170
2171
2172
21732174
2175
2176
2177 2178
21792180
21812182
2183
2184
21852186
21872188
2189
2190
2191
21922193
2194
21952196
2197
2198
2199
2200
2201
2202
22032204
22052206
2207 2208
2209
2210
2211
2212
2213
22142215
2216
22172218
2219 2220
2221
2222
22232224
2225
2226
2227
2228
22292230
2231
2232
22332234
22352236
2237
22382239
2240
2241
2242
2243
2244
22452246
22472248
2249
22502251
2252
2253
2254
2255 22562257
2258
22592260
2261
22622263
2264
2265 22662267
2268
22692270
227122722273
2274
22752276
2277
2278
2279
22802281
2282
2283
2284
2285
2286
2287
2288
2289
22902291
229222932294
22952296
22972298
2299
2300
2301
23022303
23042305
23062307
2308
230923102311
2312
23132314
23152316
2317
2318 2319
2320
2321
23222323
2324
23252326
232723282329
2330
2331
2332
2333
23342335
2336
2337
2338
2339
2340
2341
2342
2343
23442345
2346
2347
23482349
2350
23512352
2353 2354
23552356
23572358
2359
2360
23612362
2363
23642365
2366
2367
2368
23692370
2371
2372
2373
2374
23752376
2377
23782379
23802381
23822383
2384
23852386
2387
2388
2389
23902391
23922393
23942395
2396
2397
23982399
24002401
2402 2403240424052406
24072408
24092410
2411
2412
2413241424152416
2417
24182419
2420
2421 2422242324242425
2426
242724282429
2430
2431
2432
24332434
2435
24362437
2438
2439
244024412442
2443 2444
2445
24462447
2448
2449
2450
2451
24522453
2454 2455
2456
24572458 2459
2460
2461
24622463
246424652466
246724682469
2470
24712472
2473
2474
2475
2476
2477
24782479
2480
2481
2482
24832484
2485
2486
2487
24882489
2490
2491249224932494
2495
2496
2497
24982499
2500
2501
2502
2503
250425052506
25072508
2509
2510
2511
2512
251325142515
2516
2517
2518 2519
2520
2521
2522
2523
252425252526 2527
2528
25292530
2531
2532
2533
2534
25352536
2537
25382539 2540
2541
2542
2543
2544
25452546
25472548
254925502551 2552
25532554
2555
25562557
2558255925602561
2562
2563
2564
2565
256625672568
2569
2570
2571
2572
2573
257425752576
2577
2578
2579
2580
2581
25822583
25842585
2586
2587
25882589
2590
25912592
25932594
2595
25962597
25982599
2600
2601
2602
2603
2604
2605
2606
260726082609
26102611
2612
261326142615
2616
26172618
2619
26202621
2622
262326242625
2626
2627
2628
2629
2630
2631
2632263326342635
2636
263726382639
26402641
26422643
26442645
264626472648
2649
2650
26512652
26532654
2655
2656
2657
26582659 2660
266126622663
2664
2665
2666
2667
26682669
2670
2671
2672
2673
2674
2675
2676
26772678
2679
26802681
2682268326842685 26862687
2688
2689
2690
26912692
26932694
2695
2696
2697
2698
2699
27002701
2702
27032704
270527062707
2708
2709
2710
2711
2712
2713 27142715
2716
27172718
2719
27202721
2722
2723
2724
2725
2726
2727
2728
2729
27302731
2732
2733
2734
27352736
2737
27382739
2740
274127422743
27442745
2746
2747
2748
27492750
27512752
2753
2754
27552756
2757
2758
2759
2760
2761
2762
276327642765 2766
2767
2768 2769
2770
2771
2772
2773
2774
27752776
2777
2778
277927802781
2782
27832784
2785
2786
27872788
27892790
27912792
27932794
2795
2796
2797 2798
2799
2800
2801
2802
2803
28042805
2806
28072808
2809281028112812
281328142815
28162817
2818
28192820
2821
2822
2823
28242825 2826
282728282829
2830
2831283228332834
283528362837
2838
2839
28402841
284228432844
28452846
2847
2848
28492850
2851
28522853
2854
285528562857
285828592860
2861
2862
28632864
2865286628672868
28692870
2871
28722873
2874
28752876
2877
2878
28792880
28812882
2883
2884
288528862887
2888
2889
28902891
2892
2893
2894
2895
2896
2897
2898
2899
2900 2901
2902
2903
2904
29052906
2907
2908
29092910
2911
29122913 2914
2915
2916
29172918
2919
2920
2921
2922
2923
292429252926 2927
292829292930
29312932
2933
29342935
2936
293729382939
2940
2941
2942
2943
2944
2945
29462947
29482949
2950
29512952
2953
2954
29552956
2957
2958295929602961
2962
296329642965
2966
2967
2968
2969
2970
29712972
2973
2974
29752976
2977
297829792980
2981
2982
29832984
29852986 29872988
2989
29902991 2992
2993
2994
2995
29962997
2998 2999
3000
77
Hierarchical Clustering on Artificial Dataset
#start afreshdat=xclara #3000 x 2library(cluster)
#plot the dataplot(dat,type="n")text(dat,labels=row.names(dat))
plot(agnes(dat,method="single"))plot(agnes(dat,method="complete"))plot(agnes(dat,method="average"))
78
Hierarchical Clustering on Artificial Dataset
110
392
694
505
547 39
5 70 24 760 54
121
680
349
036
515
389
0 899
113
294 3
30 709 24
669
122
451
530
747 62
080
424
051
632 502 49
713
361
5 158 6
1217
945
432
621
262
751
235
289
180
271
4 112
631
781 64
825
476
4 5536
648
3 422
737 40 231 4
88 386 398 82
8 201
621
54 68 338 5
09 838
31 167
299
349
656 25
660
0 237
703
396
769
670
700
63 706
588 95 17
568
673
040
533
913
038 632 56
7 4726
277
3 264
744
852
7378
481
7 357
875 7
51 825 557
474
517
559
746
813
718 750
521
531
450
549 44
76 152
354
872
885
796 79
455
552
221
78 881 34
610
615
431
189
782 184 19
645
714
437
2 693
253
323
28 893 58
5 126
124
634
1729
663
6 424
436
797
397
845 7
83 823
699
88 193
233
310
777 38
244
851
459
162
8 260
284
213
495 52
0 880 61
675
465
938
387
8 682 83
358
778
5 874
575
735 51
159
2 75 234
291
861
683
879 64
273
1 457
079
0 285
77 335
674
755
343
741
188
720 589 23
845
329
555
160
2 814
738
801 42
930
548
1 52
102
617
444
778 202
848
576 37
036
42 657
163
538
321
780
603
652
859 2
9322
660
470
571
7 282
333
434
460
810 3
80 844
103
314
171
776
855
208
445
394
584
223
159
432
877
594 176
206 540
580
81 269 84
149
653
400
716 143
418 65
830
470
245
299 209
347
471
161
162
147
619
172
236
142
378
433 89
222
253
3 556
724
862
864 27
416
838
5 609
766
277
629 40
337
676
8 887 27
935
279
517
842
619
479
3 313
329 3
6724
780
6 626
812
637
527
743
865
808
542
512 26
129
230
812 334
677
771 723 28
649
980
519
237
783
7 849 59
071
013
552
4 856
281
563
548
359 2
4211 351
207
297
368 23
989
816 500
155
847 26
337
944
341
2 834
5621
564
126 503 25
9 258 49
419
778
7 273
74 701 88
6 504
668 5
890 341 10
867
567
8 458
889
882
651
842
742
748 25
231
269
7 198
218 3
3237
146
5 466 288
417
786
13 300
664
826 6
551
060
773
967
269
877
4 29
322
719
388
545
46 722
210
391 84
0 165
136
721 276
579
469 375
736 11
782
286
612
858
669
676
3 513 80
734
575
764
414
058
259
968
039
314
171
389 195
477
37 336
467
605
230
431
442
830
486
873
518
752 50
134
145
767
820 57 779
759
2347
653
757
8 104 11
985
419
148
7 858
428 227
871
596
618
268
410 1
3959
861
180
081
689
116
057
175
680
979
2 219
271
870 55
364
0 660 7
211 298
301
894
5327
064
659
7 728 53
466
5 888
479
635
35 704
214
265
66 225
464
643 4
8555
590
051
567
385
356
479
818
757
779
182
162
578
818
536
711 36
017
371
512
335
6 508 287
320
835
430
647 30
617
020
520 373
49 278
492
550 40
269
241 690
478
839 423
794
762 11
480 200
543
844
658
368
4 732 7
6173
4 484
67 369
645
654 53
550
712
125
136
3 811 66
930
366
738
142
1 729
480
827 190
449
183
613
315
109
661
249
257
440
447
896
156
860
302
733 344
355
472
836
87 189
970
786
7 48
473
802
290
43 132
272
122
851
758
129
624
863 3
28 415 6
0131
742
039
940
665
045 407 14
892
526
528
461
843
501 1
51 506
695
765 69
401
671 8
6912
540
884
654
648
915 164
745
782
841
622
681
712
3364 199
97 525 131
389
409 48
221
250
475
544
83 94 127
868 68
522
441
1 358
419
459 1
81 825
220
364
404 79
9 630
496 532
266
468 19
248
523 33
134
081
556
059 185
6211
855
4 228
362 39 319
177 98
116
120
283
522 49
337
457
293
614
655
414
633
649
427
595 89
518
657
388
472 244
539 441
384
342
593 34
843
983
2 753
157
666 498
361 3
3711
156
1 772 5
4313
858
151
910
020
478
915
046
356
885
7 267
676
275
413
689 4
6252
963
982
422
945
668
7 688
101
169
166
679 1
4674
025
555
877
581
843
732
487
649
156
510
530
7 325
566
203
9123
231
657
472
581
988
313
724
339
031
856
927 638
217
662
60 726 3
5010
717
438
660
814
85 850
110
606
831
245
727
96 435
470
1451 62
328
071 182
630
1683
901
1748
919
1390
1767
1931 1
078
1094
1489 1151
1400
1877
2037 15
9219
99 1991
1494
1123
1825
1583
1870 93
398
320
01 936
1827
1006
1791 17
9896
397
516
5110
98 1935
1725
1949
1172
1621 1367 11
4411
7019
4611
8316
06 1509
1581
1522
1298
1669
1897
1095
1561 1382
1907
1315
1831 10
5015
8620
50 1840
1984
1925
1329
1109
1311 12
6718
5812
7297
617
11 1226 1468
1247
1512
1782 18
6419
4013
0313
3015
3015
73 170
011
2912
8714
6216
2514
7218
93 1309
1620
1832
1527
1737 1
002
1655
1944
1731
1546
1173
1696 14
9819
78 1917
926
1042
1958 14
5918
7913
1313
6218
02 1936
1007
1579 18
1312
8515
5219
9413
8817
34 1076
1369
2029
1708
2020 10
7011
0113
4611
7618
6716
3520
40 143
614
2513
4210
4716
9412
41 109
693
713
6820
49 944
989
981
1764
1564
1759
1420
1225
1036
1107 1
299
1029
1482
1569
1762 1604
959
1636
1125
1189
1483
1710 1
691
1932
1122
1359
1559
1649
1892 1355
1456
1828 1976
974
984
1919
1422 16
1510
0812
93 1223
1143
1909 18
7415
5016
5219
72 146
314
7420
23 1577
1580
1632
1809
1061
1230
1562 1262
1054
1848 18
3912
3919
6316
0317
8017
7712
8118
3516
6416
6318
0618
6019
3711
0515
8712
0818
16 934
1769
1923
1055
1578
1529 19
6710
5912
6818
9016
3013
4015
8513
83 1855
1012
1916
1372
1904 1847 16
3410
6613
3915
6714
0519
65 1548
950
1822 11
7412
8018
68 1682
1453
1740
1624
2007 15
75 1533
1104
1974 2031
1058
1826
1908
1464 99
617
8414
6516
5019
5719
4110
8813
4199
792
018
33 1389 1126
1057
2035 13
84 1493
1024
1661
1074
1263
1374 13
0217
51 1572 11
8011
0218
11 1541
1234
1505 1
157
1757
928
1376
1902
1086
1760
2022
1945
1203
1207
962
1014
1531
1326 13
9913
0619
69 1478
971
1049
1133
1458
1637
1100
1275
2010 14
7620
3312
6414
4913
1691
311
62 1260
1707
1778
932
949 1
015
1765
1998
941
1438
1779 1515
1804
1878
1149
1205 12
1917
73 907
1331
1544 1818
1198
1983
1165
1783
1519
1990
1862
1232
1360
1416
1674
2003 9
7716
2311
9715
5518
7313
3216
5917
1713
8517
1515
1411
5315
0816
7611
8115
3215
4511
7117
36 1766
1662
1911 1
523
938
1333 10
2512
8418
5213
4417
24 131
818
9619
2917
7620
1514
1314
1999
510
3719
81 904
1871
988
1046
1128
1253 16
4710
8514
24 994
1747 17
1311
5916
97 1716
1912
1397
1407 19
8218
5610
1111
4213
1212
7911
5214
0618
98 982
1114
1597 1432
1211
1654
1084
1118
1411
1113
1454 10
2811
0318
6614
67 1528
1222
1681 16
4211
3814
15 1256
1314
1501
1885 15
58 1704
916
1457
1466
1108
1216
1979
1345
1952 11
4111
47 181
591
517
20 1005
1845
2025 13
5319
96 924
1511 16
0219
5913
2817
1211
3619
3819
5614
8497
010
93 1985 1206
1214
1876
1026
1677
1300 16
7911
7813
7011
8612
3713
81 1379
1255
1612
1837
1295
1499 12
82 190
110
6812
5912
7417
58 930
985
1884
967
2011
1087 17
4619
1311
4012
3112
4513
7514
03 1743
1961 14
8812
1214
4116
4318
88 1626
2032 1
882
991
1616
1921 10
3911
5611
66 1795
1947
1185
1242 16
33 1486
1485
1537
1638 98
620
02 172
611
6712
6618
6919
8916
9918
05 1323
1391 11
7912
4091
219
60 1132 94
817
8114
4416
87 1927 19
0615
0320
08 1551
1308
1479
1448
1510
987
1439
2045
1215
1598 1
304 1673
956
978
1335 1513 1824
1729
1060
1971 1730
1204 10
0312
5717
9319
8711
0611
8216
9017
89 2030 15
18 1023
1092
1702
1445
1471
1524 9
6110
6413
2119
70 154
014
9020
1293
115
6317
8893
514
3711
9513
3619
95 1030
1742
1875 13
5210
0411
4613
6192
212
5412
33 1955 13
4814
34 1591
1013
1261 16
2819
39 150
613
2219
1813
7820
2411
3013
9818
3418
6318
59 1455
940
1135
1365
1402
911
929
1846 17
5412
1013
5817
85 1534
1808 19
7312
4317
27 2047
1027
1619 19
4813
9319
97 1075
1814
1079
1943 15
47 1905
1112
1953 1
252
1350
1750 14
9618
19 1301
1396
1394
1617
1738
1521
1703 15
9614
73 1188
1660
2009 10
6519
03 1680
1044
1830 1
270
1296
1613
927
1090
1966 1
701
1786
1883
2036 1891
1192
1841 9
6917
2117
63 966
1276 1
770
903
1117
1688
1796 95
215
4310
4110
8217
56 162
918
36 1549 15
3518
0318
5710
7211
3913
1710
6216
3911
1511
55 1881
1228
1343
1611
2013
1942
1986 96
015
6097
214
87 161
411
7796
411
5011
68 1040 1334 13
5717
61 1283
1872 20
1612
9290
813
4912
4613
07 158
418
9519
1494
717
5214
2318
1017
3218
29 1670
1269
1421 1
934
951
1790
1689 97
916
0915
2616
98 1097
1131
1443
1258
1291 2
048
1861
1160
1553 20
2191
711
6310
9914
9710
2210
7710
3518
51 1111 1
954
1470
1653
990
1213
1706
1217
1641 1656
1018
1605
1395
2043 1854 90
920
46 1305
1417
1412
1693
1993
1539
1608
1193
1354
992
1071
1286
1290
1794
1817
2044
1739
1705
1031
1073
2039
1191
1461 14
2714
4616
8611
6116
9211
1011
8414
52 1347 9
7310
5217
9720
42 1556 12
9712
6515
4216
4020
0519
3320
2616
2712
7713
92 177
112
4410
8013
5618
1214
9218
38 1695
1570
1657
1418
1977
1091
1236
1568 9
8011
1916
2218
80 178
712
9414
35 1631
1043
1271
1718 13
8010
8919
2611
2112
50 958
1887
1409
1594
1429
1440
1962 1
428
1221
1502 1
000
1218
1672
1016
1600 1
538
1842
1968 19
2016
8494
618
8610
1717
3391
813
6317
9217
9919
5019
8010
63 1507 14
69 1722
1475
1536
1574
1671
2017
902
957
1196
1337 14
9115
8218
94 1714 1928
1988
2027
1319
1667 2
019
993
1051
1566
2038 12
3810
1915
20 1554
1900
1158
1164
1447
1709
1410
2006
1404
1450 92
317
35 2018 11
4593
912
0920
41 1251
1288 9
4311
54 1665 99
910
3317
41 1460
1310
1377 2
004 16
66 1889 11
6914
1410
3414
3318
6519
2411
8715
04 1351
954
1124
1593
942
1190
1278
1229
1495
1148
1431
1992 17
49 1202
1745
1853 1
588
1387
1009 94
592
119
1590
610
01 157
613
8614
7710
5614
8010
6990
516
45 127
317
7518
2014
81 1408
1010 96
815
6515
7117
1996
516
4610
2114
2619
6420
14 1910
1517
955
1500
2028
1320
1324
914
1768 20
3410
6715
99 1045
1755
1116
1048
1843
1744
1823 1
821
1235
1200
1224
910
1053
1801
1844
1083
1849
1675
1199
1595
1678
1327
1220
1668 1774 13
2518
99 1194
1516
1607
1364
1175
1032
1371
1020
1366
1590
1753
1807
1850
1975
1137
1589
1648 11
3495
318
0017
2310
8116
4492
516
01 122
7 1557
1401
2000 1951 16
1814
4217
7213
3832
714
30 1120
1289
1922
708
1038
1610
1201
1373
1249
1728
1525
1930
241
425
34 663 6
182
911
599
816
8520
5125
0729
4820
5521
2029
3828
25 2860
2169
2925
2149
2407
2213
2066
2703 22
8126
2924
6520
7822
8421
1526
9721
3926
0225
2129
3526
4924
1429
6427
88 2832
2614
2737 24
0929
5420
9726
58 2669
2123
2769
2267
2400 2570
2786
2305
2308 29
30 2529 21
3324
8926
72 2209
2536
2896 2
464
2706
2806
2122
2440
2720
2778
2592
2885 26
1527
2128
1827
5128
7829
73 2709
2620
2888 2
837
2869
2071
2383
2617 28
1523
2823
48 2513
2965
2998 24
7625
9721
4227
8120
8420
9825
46 2870 20
9023
2924
85 2311
2351
2096
2300
2811 28
5622
0123
34 2245
2471 2809
2235
2277
2487
2753 25
66 2408
2413
2976 2836
2928 24
2920
9525
49 2265
2677
2898
2349
2374 28
6428
55 2410 22
1922
6228
7628
7323
1528
12 268
129
9224
1523
8223
6925
3526
5325
7928
5228
83 2427
2401
2498
2755 2462
2639
2491
2594
2343
2635 23
6526
22 2686
2493
2969
2276
2929 24
28 215
823
6127
9320
8021
3725
62 2531
2130
2322 24
2522
3222
9026
8327
60 2293
2458
2984 2
178
2621
2902
2205
2475 2
460
2278
2403
2532
2704
2582
2404
2656
2438 25
1722
4627
6129
5120
6021
4626
0725
4329
11 2738
2564 2730
2805 27
5020
6424
7922
5222
30 2188
2673
2075
2168
2337
2559 27
9528
3529
00 2416
2700
2695 28
7522
8624
2126
09 212
822
7023
52 296
123
9024
7421
9123
9125
0423
6424
5926
3323
1723
60 2558
2424
2914 2
611
2651
2820
2439
2872
2393
2584 2436
2514
2576 20
6222
9723
41 2207
2103
2814
2829 23
9225
7525
6928
6529
5328
1328
8128
8421
2122
2221
1222
2925
4826
75 281
625
1128
2321
7428
1021
3126
5226
9229
0821
4323
7820
5720
8922
6827
0721
75 2148
2499
2346
2624
2782
2826
2067
2765
2224
2848
2394
2680 26
4421
1822
2825
2725
73 2405 22
3623
7027
4224
4528
43 219
022
3822
1122
5423
9825
6827
46 220
425
67 274
328
4221
1026
16 2295
2114
2135
2312 2
645
2822
2170
2868
2273 2239
2266
2335
2824 25
0229
3122
2323
5725
5626
8926
2727
14 2117
2323
2797
2124
2515 22
4925
61 2492
2217
2406
2525
2638 28
4428
0224
5122
6321
3223
5827
05 274
128
6323
8820
8529
8823
7629
9322
5628
49 2647
2478 25
06 2113
2151
2827
2423
2373
2968 23
8125
05 2719
2309
2985
2368
2324
2978 28
4026
9129
87 2747
2648
2995
2444
2661
2173
2780
2257
2306
2828 25
0025
5328
6128
91 298
627
17 2272
2088
2657 22
8220
9121
6224
5029
56 2109
2269
2271
2728 24
1120
9229
42 2430 29
6022
5122
6028
38 2604
2682 2
632
2225
2758 23
0224
3423
7124
96 2587 2
244
2915
2554
2698
2420
2461
2516
2522
2684
2913
2608
2107
2326
2385
2975 2
395
2618
2668
2218
2939
2715
2186
2231
2193
2776
2291
2526 23
8629
7224
1829
47 2530 29
09 2678
2052
2431 22
9627
1127
1320
7721
3421
5026
2829
50 2166
2181
2214
2237
2980
2303
2733
2770 22
9420
6322
55 2545 2748
2779 20
5325
9620
7620
9928
6723
3125
5726
9421
8723
9922
03 2466
2996
2642
2857
2333
2362
2789
2094
2756
2422
2552
2457
2890
2894 2
767
2634
2905 26
4326
6424
9424
1926
3020
7429
8128
4625
8524
3729
3621
8227
77 222
722
9228
80 2375 23
0725
1927
22 2944
2785 2
959
2798
2967 21
4522
4027
75 2102
2655
2783 2
482 3000
2136
2195
2248 27
36 276
220
5829
9020
8621
0826
25 2316
2946
2958
2454
2606
2768 26
7421
5623
5321
6723
1829
34 2957 21
7724
7225
63 2907
2544
2659 23
8927
2328
8221
0528
33 2912
2886
2275
2921
2497
2560 26
8728
5927
08 2850 2739
2518
2685
2804 2
731
2261
2061
2359 24
4124
8425
77 2725
2927
2654
2997
2336
2766
2065
2593
2192 21
59 2910
2887
2179
2220
2455
2068
2974
2247
2384
2417 22
5928
99 2470
2533
2314
2473 27
2621
7229
70 2547
2877
2989 2
800 24
8320
7921
9927
4523
8722
9826
46 2841
2773
2082
2469
2901 29
71 2701 28
6623
1929
4129
6328
3125
7428
0823
5426
93 2463
2379
2637
2509
2600 27
8728
9225
41 2550
2718
2999
2083
2242
2141
2920
2093
2641
2126
2397
2206
2481 22
7424
3524
49 2233
2456
2752
2164
2740
2138
2834 2
185
2234
2338
2501
2763 2
583
2572
2304
2660
2539 2363
2542
2937
2924 24
8824
3220
8121
1122
7924
6825
28 2879
2208
2667
2790
2087
2226
2327
2161
2350
2922 2
650
2994 27
12 2665
2906
2280
2153
2503
2520 23
4225
9920
5627
8426
0128
4529
4925
1222
8323
5520
6925
89 217
127
2929
1625
3422
1229
4322
0223
2128
7421
9426
36 221
6 279
922
8729
52 2977
2571
2160
2955
2127
2918
2180
2749 21
63 2807
2565
2839
2250
2754
2771
2688
2871
2332
2299
2452
2101
2119
2759
2072
2285
2598
2372
2367
2258
2613
2330
2140
2716
2734
2155
2603
2523
2962
2581
2591
2198
2917 24
4829
2629
3223
0124
8624
4226
4023
4526
7928
9320
7326
9626
7622
8923
2525
88 2510 29
82 2495
2586
2610
2772
2991 2
623
2059
2819
2221
2310
2106
2670
2895
2578
2215
2288
2744
2443
2144
2366 2
524
2402
2699
2426
2904
2264
2116
2189
2196
2889
2347
2671
2480
2796
2356
2396
2508
2979
2847
2176
2100
2619
2537
2612
2147
2538
2129
2241 25
9527
9425
8028
3027
9121
5224
9024
3327
35 2626
2727
2757
2732
2344
2710
2919
2702
2897
2054
2165
2339
2104
2858
2253
2724 24
46 234
025
4024
5326
9029
3321
8426
0529
6621
8328
53 2903
2125
2555
2854 2
447
2200
2320
2774
2666
2243
2631
2983 2
590
2801
2154
2157
2313
2851
2412
2862
2477
2197
2764
2551
2817
2070
2821
2803
2945
2940 30
9 1127
1658
2792
2923
610
1248
387
562
2662
2380
2467
353
770
2377
2210 41
674
926
63
02
46
810
Dendrogram of agnes(x = dat, method = "single")
Agglomerative Coefficient = 0.93dat
Hei
ght
79
Hierarchical Clustering on Artificial Dataset
124 760
392
694
395 10 541 70
505
547
153
890
899 22 365
212
627 13 300
826
664
698
774
713
140
582
599
680
268
410
393
667 65
510
607
672
739
822
866
399
406
650 490
29 322
388
545
719
276
579
320
835 86 398
828
242
136
721
469
375
736
165
210
391
840 18 536
711
360
173
715
287
51 235
289
180
246
691
123
356
508
430
647 73 784
817
170
205
80 200 306 3 54 68 33
8 838
201
621
405
63 706
175
588 95
686
730 38 632
262
773
264
357
875
509
567 47 744
852
751
135
524
160
571
756
809
792
192
377
837
849
590
710 8
474
517
521
531 25 557
559
746
813
718
750 44 79 455
552 31 167
299
349
656
256
600
396
769 74 701
886
196
46 722
117
237
703
670
700 76 152
354
796
872
885 82 184
130
339 41 690
478
839
109
661
249
257
440
447
896
111
561
157
666
498
337
1433 64 199 97 525
131 59 185 62
118
554
389
409
482
113
294
330
709
216
803 20 373 49 278
402
423
794
762
492
550
692
186
573
884
228
362 114
427
595
895
414
633
649
229
462
491
565
27 638
318
569 43
772 244
441
361
348
384
539
439
832
753 772
138
581
519
342
593 543
60 726
350
107
241
425 3
87 416
749 6
87 189
472
836 681 20
312
786
868
535
841
945
949
653
218
182
541
368
922
079
936
440
463
0 11 351
207
297 898
119
854
239
368
564
23 476
104
537
578
191
487
428
858 21
250
475
544
224
411
190
449
302
733 67 369
645
654
183
613
315
507
535
121
669
251
363
811
344
355 83 94 272
504
668 84 658
143
418
149
653
400
716 19
248
523
331
340
815 560
85 850
110
606 83
196 435 470 14
5112 334
677
771
723
198
218
856
332
466
371
465
417
786 853
156
860
798
288
800
464
643
485
555
900 28 893
585
126 81 269
383
878
874
587
785
682
833
144
372
693
253
323
154
311
897
450
549
291
861
683
879
642
731 75 234
308
871
281
563
359
548
286
499
805 16 500
155
847
263
379
443
412
834
219
553
640
271
870
660 66 225
328
415
26 503
259
258 56 215
641
213
495
520
880
575
735
616
754
659
577
791
821
788
100
204
789
558
775
818
819
883
122
851
601
758
129
624
863
317
420
489
101
169 146
166
679 74
015
046
356
885
7 187
515
673 82
4 623
353
271
411
263
178
164
8 40 231
488
139
598
611
816
891 37 336 50
467
605 55
366
483
422
737
254
764
134
145
767
820
304
702
452
159
540
176
206
432
877
594 45 407 92
526
528
148
712
151
506
695
765
461
843
501 4
570
790
285 77 335
674
755
343
741
102
617
444
778
223
36 42 657
163
538
321
780
603
293
652
859 78 881
221
106
346
124
634
457
260
284
103
314
305
481
429
171
776
855
238
453
580
295
814
551
602
738
801
197
787
273
222
533
724
556
862
864
759
52 370
202
848
576
188
720
589
460
810
208
445
394
584
226
604
705
717
282
333
380
434
844
128
513
345
757
644
511
592
586
696
763
807 5
438
484
446
583
684
732
761 53 270
646
597
728
888
534
665 7
211
298
301
894 89 195
477
480
827
99 209
479
635
194
793
313
329
261
292 35 704
214
265 54
623
044
283
043
148
687
351
862
514
237
843
389
216
838
560
976
637
676
814
761
917
223
616
116
234
747
127
762
988
727
940
330
375
217 424
436
296
636
699
596
618
274
397
845
783
797
823
88 193
233
310
777
448
514
382
591
628 58 90 341
141
227
252
312
697
458
889
882
651
842 43 132 745
69 401
671
125
408
846
869
494
527
743
865
542
57 779
352
795
108
675
678
742
748
178
426
367
247
806
637
626
812
512
808 39 319
177 255
9811
612
028
352
249
337
457
210
530
732
556
613
724
3 390
245
727
267
676 72
938
142
173
462
232
487
657
472
5 71 182 27
528
061
012
48 970
786
7 48 290
473
802 15 164
782
841
266
468 91
232
316
34 663 61 829 11
5 309
32 502
497
240
620
804
133
615
158
516
451
530
747 93
614
655
179
454
326
612
456
687
688
174
386
608
217
662
529
639 30
1683 1081
1644
943
1154 99
916
6510
3411
6912
5112
8814
3318
65 1924
708
1038
1610
1120
1289 19
2232
714
30 965
1646 1021
1069
1187
1504
1351
953
1800 1200
1235
960
1560
1357
1761 992
1071
1286 964
1334
1040
1150
1168
1283
1872
917
1163
1099
1497
1022
1077
1217
1641
1656
1018
1705
1035
1851
1111
1954
1470
1653
1722
944
989
1834
1863 981
1764
1564
1759
1420
1029
1299
1604
1482
1569
1762
1036
1107
1225
1180
1359 98
714
3920
4512
1515
9813
0499
012
1317
0695
916
3614
8317
1011
2511
2211
8916
9111
3013
9818
5914
5618
2819
76 908
1349
1246
1307
1584
1895
1914
947
1752
1670
1423
1810
1732
1829 951
1790
1689 97
916
0915
2616
98 1009
1161
1692
1269
1421
1934
950
1822
1174
1533
1575
1624
2007
1013
1261
1628
1939
1104
1974
1493
1882
1280
1868
1682
1453
1740
1097
1258
1291
1131
1443
2048
1221
1502
1160
1553
1861
2021 92
119
1595
818
8714
2814
0915
9414
2914
4019
6292
212
5412
3319
55 931
1563
1195
1336
1995
1788
1030
1352
1742
1875 93
514
3713
6194
618
8610
1710
0411
4610
4418
3012
7012
9211
9915
9516
78 1327
1744
1823 1821
925
1601 9
9810
3213
71 1227 16
8591
417
6820
3410
4818
4310
6715
9910
4517
55 1116
1000
1218
1672 1684
1016
1600
1538
1842
1968
1920
1043
1380
1271
1718
1089
1926 11
7515
6511
9411
3413
3813
6415
1616
07 1249
1728
1320
1324 15
5790
117
4815
2717
3711
2919
1712
8714
6216
2514
7218
9313
0916
2018
3210
0216
5519
4414
2511
7316
9619
7814
9812
1214
41 985
1488
1546
1731
1743
1961
2016
1064
1540
1321
1970
1490
2012 928
1086
1203
1207
1264
1449
1476
2033
1760
2022 93
011
0012
7520
1013
1696
210
1415
3113
9913
2613
7619
0219
4516
4318
8813
0619
6914
7813
8817
3416
4918
92 912
1960
1132
1503
2008
1906
1308
1479
1448
1510
1551 948
1781
1444
1687
1927
1348
1434
1591
971
1049
1133
1458
1637
1140
1231
1245
1375
1403 91
614
5714
6611
0812
1619
7913
4519
5211
4111
4711
3814
1515
2819
0110
6812
5918
1511
5311
8115
3215
45 982
1114
1597
1211
1432
1654
1103
1866
1467
1222
1681
1642
1256
1314
1501
1885
1558
1028
1084
1118
1411
1214
1876
1113
1454
1704
1136
1938
1956
1484 92
610
4219
5813
1314
5918
7910
9613
6218
0219
3611
8612
9514
9912
5516
1218
3712
8212
3713
8113
79 970
1093
1985
1206
1026
1677
1300
1679
1178
1370
1047
1694
1241
1413
1419
1007
1579
1813
1708
2020
1285
1552
1994
1076
1369
2029
1070
1436
1101
1346
1176
1867
1635
2040 92
018
3313
8911
2693
398
320
0113
4293
618
2710
0617
9117
9897
617
1112
2614
6816
6918
9718
8412
4715
1217
8218
6419
4013
0313
3017
0015
3015
73 996
1784
1650
1957
1941 99
714
6510
5720
3513
8413
2219
1810
8813
4112
7417
58 963
1725
1949 975
1651
1098
1935
1144
1298
1522
1170
1946
1095
1561
1382
1315
1831
1907
1109
1311
1050
1925
1984
1172
1621
1367
1329
1586
2050
1840
1061
1262
1230
1562
1580
1632
1809 90
418
71 994
1747
1397
1407
1647
1982 988
1128
1253
1046
1085
1424
1159
1697
1713
1912
1716
1856
915
1720
1005
1845
2025
1353
1996 924
1511
1602
1328
1959
1011
1142
1152
1279
1312
1406
1898
1062
1639
1115
1155
1228
1881
1549
1712
1343
1611
2013
1673
1942
1986 94
211
9012
7812
2914
9510
2314
4514
7115
2412
9017
9418
1720
44 956
978
1335
1513
1003
1092
1702
1060
1971
1204
1730
1729
1824
1257
1793
1987 919
1390
1767
1931
1078
1183
1606
1509
1581
1094
1489
1991
1151
1400
1877
2037
1494
1825
1583
1870
1592
1999
1024
1661
1074
1263
1374
1751
1302
1572
1267
1858
1272 96
110
6519
0316
8011
2313
5511
0615
1811
8220
3016
9017
89 913
1162
1260
1707
1778
1515
1804
1878 97
414
6410
5818
2619
08 984
1919
1463
1474
2023
1577
1008
1293
1223
1422
1615
1143
1909
1874
1550
1652
1972 932
949
1015 941
1438
1779
1149
1205
1787
1102
1811
1757
1541
1157
1234
1505
1765
1998 934
1769
1923
1055
1578
1529
1967
1746
1339
1567
1405
1965
1548
1012
1916
1634
1066
1372
1904
1847
1059
1268
1890
1630
1855
1340
1585
1383
1054
1848
1663
1806
1839
1105
1587
1208
1816
1239
1963
1603
1780
1777
1281
1835
1664
1860
1937
1219
1773
1913
1626
2032 92
796
917
0117
8617
2117
6318
8320
3610
9019
6618
9111
9218
4196
720
1110
87 995
1037
1981 96
612
7617
7014
9115
8218
9417
1419
2819
8820
27 918
1507
1063
1792
1799
1363
1950
1980
1052
1797
2042
1265
1542
1627
1640
2005
1933
2026 98
012
7713
9211
1916
2218
8010
8013
5618
1214
1819
7712
4417
7112
2016
6817
7413
2518
9914
6914
9218
3816
9515
7016
57 945
955
1500
2028 99
310
5115
6620
3812
3810
5614
80 1201
1224
1723
973
1556
1110
1347
1184
1452 1148
1431
1992
1749
1019
1520
1554
1900
1158
1164
1447
1709
1091
1236
1297
1568
1410
2006
903
1117
1796
1688 952
1543
1041
1082
1756
1629
1836
1535
1803
1857
1179
1776
2015 938
1333
1025
1344
1724
1284
1852
1165
1783
1519
1990
1862
1416
1674
2003
1605
907
1232
1360 1072
991
1616
1921
1039
1198
1983
1331
1544
1818
1156
1166
1795
1485
1537
1638
1185
1242
1633
1947
1486
1027
1619
1948
1393
1997
1075
1814
1139
1317 91
111
88 972
1487
1614
1193
1354
929
1846
1754
1660
2009
1296
1613
1210
1358
1785
1243
1727
2047
1534
1808
1973 937
1368
2049
1932
1318
1896
1523
1929
1559
2031 977
1623
1514
1197
1555
1873
1332
1659
1717
1385
1715
1171
1736
1766
1662
1911
1508
1676 90
920
4613
0514
17 939
1209
2041
1191
1461
1427
1446
1686
1412
1693
1993
1539
1608
1031
1073
2039
1394
1473
1112
1953
1252
1396
1177
1301
1350
1750
1496
1819
1033
1741
1460
1889
1310
1377
2004
1666
1414
940
1135
1365
1402
1294
1435
986
2002
1726
1240
1869
1989
1631
1699
1805
1167
1266
1323
1391 17
3910
1011
2112
5010
7919
4315
4719
0515
2117
0315
9616
1717
3812
0215
88 1408
1745
1853
1020
1850
1975 16
4811
3715
8913
6615
9017
5318
0714
0120
0019
5115
2519
3014
4217
72 1618
902
957
1196
1337 91
010
8318
4910
5318
0118
44 906
1001
1576
1404
1450
1386
1477 15
1790
516
4512
73 1481
1775
1820
1319
1667
2019 92
317
3511
4595
411
2415
9314
2619
1019
6420
14 968
1671
2017 13
8714
7515
3615
7417
3315
7117
1913
7820
2414
5515
0613
9520
4318
54 2018
1127
1373
1675 16
5856
226
62 2380 24
6720
5928
19 2578
2215
2288
2744
2150
2628
2950 2571
2221
2310
2497
2560
2739
2327
2443
2106
2670
2895
2356
2396
2508
2979 21
2521
2922
4125
9522
4429
1525
5426
9822
9924
5224
5326
90 2933
2792
2052
2431
2296
2711
2713 2077
2454
2674
2606
2768
2912
2105
2833
2886
2167
2333
2177
2472
2907
2389
2563
2723
2147
2538
2318
2934
2957
2544
2659
2402
2699
2426
2904 20
7028
2121
8328
5326
3129
8325
5128
1722
7529
2125
1827
3126
8728
5928
5027
0822
8028
8729
6324
8326
8528
0424
4829
2629
32 2524
2586
2610
2772
2991
2055
2825
2120
2938
2860
2706
2806
2149
2407
2213
2169
2925
2267
2400
2305
2308
2930
2529
2570
2786
2066
2703
2281
2629
2078
2284
2139
2602
2465
2097
2658
2669
2709
2878
2973
2489
2672
2521
2935
2115
2697
2954
2620
2888
2837
2869
2409
2788
2832
2614
2737
2414
2964
2649
2107
2186
2385
2975
2193
2776
2231
2138
2834
2185
2234
2338 2583
2218
2939
2715
2291
2526
2909
2418
2947
2530
2326
2395
2618
2668
2386
2972
2678
2080
2137
2562
2531
2130
2322
2425
2232
2290
2683
2760
2293
2458
2984
2413
2976
2836
2928
2608
2809
2084
2098
2546
2870
2245
2471
2158
2429
2276
2929
2428
2748
2779
2362
2789
2522
2684
2913
2063
2255
2545
2103
2814
2829
2392
2575
2088
2657
2282
2604
2682
2569
2865
2953
2091
2162
2450
2956
2109
2271
2728
2269
2411
2225
2758
2371
2496
2587
2302
2434
2092
2942
2430
2632
2251
2260
2838
2960
2173
2780
2306
2828
2500
2986
2553
2861
2891
2134
2166
2181
2214
2257
2717
2237
2980
2303
2294
2733
2770
2051
2507
2948
2420
2461
2516
2079
2199
2387
2745
2112
2229
2143
2378
2511
2823
2174
2810
2548
2675
2816
2131
2652
2692
2908
2060
2738
2146
2607
2564
2730
2805
2543
2911
2750
2298
2646
2841
2630
2773
2062
2207
2297
2341
2121
2128
2961
2813
2881
2884
2270
2352
2390
2474
2074
2981
2585
2846
2307
2375
2437
2936
2250
2332
2182
2777
2227
2292
2880
2127
2918
2163
2180
2749
2565
2839
2807
2688
2871
2754
2771
2058
2990
2316
2086
2108
2625 2910
2272
2946
2191
2391
2504
2439
2872
2317
2360
2558
2424
2914
2611
2651
2820
2393
2584
2436
2514
2576
2085
2988
2376
2993
2959
2256
2849
2506
2444
2478
2647
2113
2381
2505
2151
2827
2423
2719
2373
2968
2309
2985
2368
2958
2094
2756
2831
2114
2135
2312
2645
2822
2863
2156
2353
2422
2552
2767
2457
2890
2894
2494
2634
2905
2643
2664
2064
2479
2252
2188
2230
2673
2075
2168
2337
2559
2795
2835
2900 22
3520
9623
0028
1128
5622
0123
3422
8624
2126
0924
1627
0026
9528
7522
1922
7725
6624
0824
8727
5323
4923
7428
5528
6423
8820
7123
8326
1729
6529
9823
2823
4824
7625
1320
9023
2924
8523
1123
5121
4227
8125
9728
1520
9525
4926
7728
9822
6524
1024
9329
6921
2425
1522
4925
6121
3223
5827
0524
9222
1728
4424
5128
0224
0625
2526
3822
6327
4123
6424
5926
3322
6228
7628
7324
6226
3924
9125
9423
1528
1223
8224
0124
9827
5524
1526
8129
9223
4326
3523
6526
8626
2223
6925
7928
5228
8324
2725
3526
5320
5627
8426
0128
4529
49 2512
2283
2355
2155
2603
2345
2367
2102
2655
2783
2482
2136
2736
3000
2195
2248
2762
2140
2716
2734
2198
2917
2523
2962
2679
2893
2412
2862 24
7722
5327
2424
4623
4427
10 2919
2068
2974
2179
2574
2808
2160
2955
2247
2384
2417
2259
2899
2314
2473
2726
2470
2533
2324
2978
2840
2648
2995
2661
2691
2987
2747
2100
2619
2537
2612
2172
2970
2547
2877
2989
2800 21
5721
9628
89 2791
2313
2851
2116
2189
2480
2796
2347
2671 24
4728
3029
0328
0329
4577
029
4020
7222
8525
9823
7221
7625
8021
5323
4224
3225
0325
20 2599
2555
2854
2093
2641
2126
2397
2456
2752
2501
2763 2924
2539
2542
2937
2161
2350
2922 21
6423
0426
6023
6322
0624
8122
7422
3324
3524
4925
7227
4021
2228
1824
4027
2027
7825
9228
8526
1527
2121
4522
4027
7527
5121
2327
6921
3322
4627
6129
5122
2226
6526
5029
9427
1221
0121
19 2759
2264
2727
2757 21
8427
3220
5325
9620
9928
6720
7623
3125
5721
8723
9922
0324
4127
2529
2726
9428
4220
8722
2622
2024
5524
6629
9626
4228
5727
9829
6725
8125
9123
0124
8624
4226
4022
8925
1023
2525
8829
8223
4025
4026
2320
5723
4626
2420
8921
4824
9921
7522
6827
0722
1121
9024
4528
4325
6827
4620
6727
6522
2428
4823
9426
8026
4422
0924
6425
3628
9621
1822
2825
2725
7324
0522
3622
3823
9823
7027
4223
6127
9321
1026
1622
9522
0425
6727
4322
2323
5722
5427
8228
2621
1723
2327
9725
5626
8926
2727
1421
7028
6822
7329
3122
3925
0222
6623
3528
2420
6123
5921
5920
6525
9321
9223
3627
6624
8425
7726
5429
9720
8322
4221
4129
20 2794
2069
2589
2799
2943
2171
2729
2916 2534
2212
2287
2952
2977
2202
2321
2874
2319
2941 2261
2495
2178
2621
2902
2404
2656
2438
2517
2205
2475
2460
2532
2704
2278
2403
2582
2258
2613
2330
2419
2488
2073
2696 2676
2194
2636
2216
2144
2366
2923
2200
2243
2320
2774 2666
2590
2801
2605
2966
2210
2054
2165 2339
2104
2858
2702
2897 21
5428
4720
8121
1122
7921
5224
9024
3327
35 2626
2082
2469
2901
2701
2971
2785
2866
2519
2722
2944
2882
2906
2208
2667
2790
2468
2528
2879
2354
2693
2463
2718
2999
2379
2637
2550
2509
2600
2787
2892
2541
2663
2197
2764
2377
020
4060
8010
012
014
0
Dendrogram of agnes(x = dat, method = "complete")
Agglomerative Coefficient = 0.99dat
Hei
ght
80
Hierarchical Clustering on Artificial Dataset
124 760
216
803 10
392
694
395 70
505
547
153
890
899
5123
528
9 180
113
294
330
709
246
691
1433 64 199
97 525
131
59 185 62
118
554
389
409
482
13 300
826
664
698
774 713 65
510
607
672
739
822
866
268
410 39
366
729 322
388
545
719
136
721
469
375
736
276
579 22
451
530
747
212
627
365
541
399
406
650 49
032 502
497
240
620
804
516
133
615
158
612
179
454 32
6 9361
465
552
963
9 217
662
174
386 608
383
820
162
140
586 398
828
242
54 68 338 73
784
817
357
875 170
205
18 536
711
360
320
835
123
356
508
173
715
287
306
38 632
509
567 47 744
852
262
773
264
135
524
751
192
377
837
849
590
710
160
571
756
809
792
686
730 20 373
114
49 278
402
492
550
692
430
647
41 690
478
839 80 200
423
794
762
109
661
249
257
440
447
896
186
573
884
228
362 12 334
677
771
723
198
218 856
288
417
786 85
333
246
637
146
5 187
515
673
464
643
485
555
900 798
800
871
27 638 318
569 43
772 244
384
539
441
157
666
498
361
772
111
561 337
138
581 51
934
259
334
843
983
275
3 543
229
462
491
565
414
633 64
942
759
589
5 271
411
263
178
164
813
414
576
782
0 37 336 50
467
605 45 407
5536
648
342
273
725
476
4 40 231 488
139
598
611
816
891 92
526
528
461
843 501
151
506
695
765
358
419
459 49
653
239 319
177 98
148
712
116
120
283
522
493
374
572
105
307
325
566 25
55
438
484
446
583
684
732
761 7
211
298
301
894
479
635 53
270
646
597
728
534
665
888
35 704
214
265 54
623
044
283
043
148
687
3 518
625
303
752
17 424
436
783
797
296
636
699
823
596
618
88 193
233
310
777
382
448
514
591
628
142
378
433
892
274
397
845 75
922
253
355
672
486
286
416
838
560
976
627
762
940
327
937
676
888
7 57 779
352
795
147
619
172
236 99 209
161
162
347
471 89 195
477
512
808
178
426
367
194
793
313
329
261
292 58 90 341
252
312
697 141
227
140
582
599
680 69 401
671
869
125
408
846
108
675
678
458
889
882
651
842
742
748
247
806
637
626
812 49
452
774
386
5 542 9
707
867 48 290
473
802
480
827 15 164
782
841 43 132 74
526
646
845
668
768
8 9123
231
64
570
790
285
188
720
589
460
810
238
453 52 370
202
848
576
102
617
444
778 36 42 657
163
538
321
780
603
293
652
859 77 335
674
755
343
741 22
310
331
417
177
685
512
463
4 457
197
787
273
295
814
551
602
738
801
305
481
429
128
586
696
763
807
513
644
208
445
394
584
345
757 84 658
143
418 380
149
653
400
716
304
702 452 27
215
943
287
759
417
620
654
058
022
660
470
571
728
233
343
484
4 847
451
7 25 557
750 44 79
455
552 196
559
746
813
718 74 701
886
76 152
796
221
354
872
885 82 184
130
339
670
78 881
346
106
260
284 31 167
299
349
656 63 706
588
175 95
237
703
700
256
600
396
769 46 722 117
165
210
391
840
28 893
585
126 81 269
154
311
897
450
549
521
531
383
878
682
833
587
785
511
592
874
75 234
308
291
861
683
879
642
731
144
372
693
253
323
281
563
359
548
286
499
805 11 351
207
297
368 898
119
854 564
16 500
155
847
263
239
379
443
412
834
219
553
640
271
870
660
26 503
259
258
56 215
641
213
495
520
880
575
735
616
754
659
156
860
577
791
821 788
66 225
328
415
601
122
851
758
129
624
863
317
420 48
910
020
4 789
558
775
818
819
883
101
169 14
616
667
9 740
150
463
568
857
824 62
319
248
523
331
340
815 56
085 850
110
606 8
3123 476
537
578 104
191
487
428
858 67 369
645
654
183
613 315
507
535
190
449
121
251
363
811
669
344
355
504
668 21
250
475
544 83 94 127
868
685
181
825
220
364
404
630
799 413
689
87 189
681
224
411
302
733
472
836
203
34 663 61 829 1
15 309
137
243 39
024
572
726
767
638
142
173
472
962
232
487
657
472
5 2070
2821
96 435 47
0 1451
353
60 726 350 10
724
142
538
76
708
1038
1610
1120
1289 1
922
327
1430 7
118
2 275 280
610
1248
416
749
3016
8310
3411
6912
5112
8814
3318
65 1924
939
1209
2041 94
311
54 999
1665 10
8116
44 101
010
3317
4114
60 1889
1414
1310
1377
2004
1666
903
1117
1796
1688
1549
1082
1756
1629
1836 952
1543
1041 1072
1535
1803
1857 90
713
3115
4418
1811
9819
8312
3213
6014
1616
7420
03 1605
938
1333
1025
1284
1852
1344
1724
1165
1783
1519
1990
1862 940
1135
1365
1402 986
2002
1726
1699
1805 16
3111
6712
6618
6919
8913
2313
9111
7917
7620
15 1240 17
3990
920
4613
0514
1711
9313
5414
1216
9319
9315
3916
0810
3110
7320
3911
9114
6114
2714
4616
8692
918
4617
5412
9616
1316
6020
0912
1013
5817
8515
3418
0812
4317
2720
4719
7310
2716
1919
4813
9319
9710
7518
1411
3913
1710
7919
4315
4719
0511
2112
5013
9414
7315
2117
0316
1717
38 1596
1112
1953
1252
1396
1301
1350
1750
1496
1819 11
7710
2018
5019
7513
6615
9017
5318
0711
3715
89 164
815
2519
3012
0215
8817
4518
5312
9414
35 1408
1401
2000
1951 1618
1442
1772
901
1748
1527
1737
1129
1917
1287
1462
1625
1472
1893
1309
1620
1832
1002
1655
1944 1731
1546
1173
1696
1978
1498
1425 98
510
6413
2119
7015
4014
9020
1210
9711
3114
4312
5812
91 2048
1160
1553
1861
2021
928
1086
1376
1902
1760
2022
1945
1212
1441
1643
1888 962
1014
1531
1326
1399
1649
1892 93
011
0012
7520
1012
6414
4913
1614
7620
3311
8013
5912
0312
0713
0619
69 1478
971
1049
1133
1458
1637
1140
1231
1245
1375
1403
1488
1743
1961 20
1692
212
5412
3319
5593
514
3711
9513
3619
95 1788
1361
931
1563
1004
1146
1030
1742
1875
1352
946
1886 1017
1044
1830 1270
1292
912
1960
1132
1444
1687
1927
1906
1503
2008
1308
1479
1448
1510
1551 948
1781
1348
1434
1591
1043
1271
1718
1380
1089
1926
1000
1218
1672 16
8410
1616
0015
3818
4219
6819
20 911
1188
960
1560
1357
1761 972
1487
1614
992
1071
1286 964
1334
1040
1150
1168 1283
1872
944
989
981
1764
1564
1759
1420
1036
1107
1225
1029
1299
1604 98
714
3920
4512
1515
9813
0495
916
3611
2511
8914
8317
10 1559
1122
1691
1932
1482
1569
1762
1318
1896
2031
1929
1130
1398
1834
1863 18
5914
5618
2819
76 917
1163
1099
1497
1022
1077 990
1213 1706
1217
1641
1656
1035
1851
1111
1954
1470
1653 10
6911
8715
0413
51 1722
953
1800 12
0012
3596
516
46 1021
1018
1705
904
1871 988
1128
1253
1046
1085
1424 994
1747
1397
1407
1647
1982
1159
1697
1713
1912
1716
1856
915
1720
1005
1845
2025
1353
1996 924
1511
1959
1602
1328
1712
1011
1142
1312
1152
1279
1406
1898
1062
1639
1115
1155
1228
1881
1343
1611
2013
1942
1986 16
7395
697
813
3515
1317
2918
2410
6019
7112
0417
3012
5717
9319
8710
0310
9217
0211
0620
3015
1811
8216
9017
89 961
1065
1903
1680 13
5591
614
5714
6618
1511
0812
1619
7913
4519
5211
4111
47 991
1616
1921
1039
1156
1166
1795
1947
1185
1242
1633
1486
1485
1537
1638 97
716
2315
1415
2311
9715
5518
7313
3216
5917
1713
8517
1511
7117
3617
6616
6219
1115
0816
7610
6812
5911
5311
8115
3215
45 926
1042
1958
1313
1459
1879
1362
1802
1936
937
1368
2049
1007
1579
1813
1708
2020
1285
1552
1994
1388
1734
1076
1369
2029
1096
1070
1101
1346
1436
1176
1867
1635
2040 970
1093
1985
1206
1214
1876 19
0110
2616
7713
0016
7911
7813
7010
4716
9412
4114
1314
1911
8612
5516
1218
3712
8212
3713
8113
7912
9514
99 920
1833
1389
1126
1057
2035
1384 99
713
42 996
1784
1465
1650
1957
1941
1088
1341 933
983
2001
1669
1897
1884
936
1827
1006
1791
1798
1172
1621
1367 976
1711
1226
1468
1247
1512
1782
1864
1940
1303
1330
1700
1530
1573 982
1114
1597
1211
1432
1654
1028
1084
1118
1411
1113
1454
1704
1136
1938
1956
1484
1103
1866
1467
1528
1222
1681
1642
1138
1415
1256
1314
1501
1885
1558
1024
1661
1074
1263
1374
1302
1751
1572
1109
1311
1267
1858
1272
913
1162
1260
1707
1778
1515
1804
1878
967
2011 1087
1583
1870 99
510
3719
81 919
1390
1767
1931
1078
1094
1489
1151
1123
1991
1400
1877
2037
1592
1999
1494
1825 963
1725
1949 975
1651
1098
1935
1095
1561
1382
1315
1831
1907
1144
1298
1522
1170
1946
1183
1606
1509
1581
1050
1925
1984
1329
1586
2050
1840
1061
1262
1230
1562
1580
1632
1809 97
411
5710
5818
2619
0814
6498
419
1914
2210
0812
9312
2316
1514
6314
7420
2315
7711
4319
0918
7415
5016
5219
7211
0515
8712
0818
1612
3919
6316
0317
8017
7712
8118
3516
6416
6318
0618
6019
3712
1917
73 1913
1626
2032
932
949
1015
941
1438
1779
1765
1998
1149
1205 17
8710
1219
1616
3410
6613
7219
0418
4711
0218
1117
5712
3415
0515
41 902
957
1319
1667
2019
905
1645
1273
1775
1820 14
8192
317
3520
1895
411
2415
93 1145
1426
1964
2014
1910
1373
968
1671
2017
1475
1536 1574
1733 13
8790
813
4912
4613
07 1584
1895
1914
1013
1261
1628
1939
950
1822
1174
1533
1575
1624
2007
1221
1502
1274
1758
1322
1918
934
1769
1923
1055
1578
1529
1967 1746
1059
1268
1890
1630
1855
1340
1585
1383
1378
2024
1455
1506
1395
2043
1854
1054
1848
1839
1339
1567
1405
1965
1548
1104
1974
1493
1882
1280
1868
1682
1453
1740
947
1752 16
7014
2318
1017
3218
2912
6914
21 1934 10
0911
6116
9295
117
9016
89 979
1609
1526
1698 15
7117
1990
610
0115
7613
8614
7714
0414
50 151
791
010
8318
49 1053
1801
1844 1
675
927
1192
1841
1090
1966 1891 969
1721
1763
1701
1786
1883
2036 966
1276
1770 9
4519
2819
8820
2711
9613
3714
9115
8218
9417
14 918
1363
1950
1980 1063
1792
1799 97
315
5610
5217
9720
4212
6515
4216
2716
4020
0519
3320
26 980
1277
1392
1119
1622 1880
1080
1356
1812
1418
1977 1244
1771
1469
1507
1492
1838
1695
1570
1657
1220
1668
1774
1325
1899
942
1190
1278
1229
1495
1023
1445
1471
1524
1290
1794
1817
2044
1110
1184
1452
1347 1148
1431
1992
1749
1019
1520
1554
1900
1158
1164
1447
1709 1091
1236
1297
1568
1410
2006 9
5515
0020
28 1224
1201
1723
993
1051
1566
2038
1238
1056
1480 91
417
6820
3410
6715
99 1048
1843
1744
1823 18
2110
4517
55 1116
921
1915
1199
1595
1678 13
2795
818
8714
2814
0915
9414
2914
40 1962
1032
1371 12
2792
516
0199
816
8511
3413
3811
7515
6511
9413
6415
1616
07 1249
1728
1320
1324 15
5711
2716
5856
226
62 2380 2
467
2792
2125
2453
2690 29
3321
2922
4125
9522
4429
1525
5426
9822
9924
5220
5224
3122
9627
1127
13 2077
2105
2833
2886
2454
2674
2606
2768
2912
2167
2333
2193
2776 2231
2107
2186
2385
2975
2218
2939
2715
2326
2395
2618
2668
2291
2526 2909
2386
2972
2678
2748
2779 21
4725
3821
7724
7225
6323
8927
2323
1829
3429
5725
4426
5929
0724
0226
9924
2629
0420
5928
1922
2123
1021
3421
6621
5026
2829
50 2571
2106
2670
2895 25
7822
1522
8827
44 2443
2356
2396
2508
2979
2275
2921
2518
2731
2687
2859
2850
2708 23
2724
9725
60 2739
2280
2887
2963
2483
2685
2804 24
4829
2629
32 2524
2586
2610 2772 2991 21
8328
5326
3129
8325
5128
1720
5125
07 2948
2420
2461
2516
2079
2199
2745
2387
2298
2646
2841
2630
2750 27
7320
7429
8125
8528
4623
7524
3729
36 2307
2182
2777 2227
2292
2880
2574
2808
2127
2918
2163
2180
2749
2565
2839 2807
2250
2754
2771 23
3223
6726
8828
7120
5627
8426
0128
4529
49 2512
2155
2603 23
4521
0226
5527
8324
8222
8323
5521
3627
3630
0021
9522
48 2140
2716
2734
2198
2917
2523
2962
2679
2893
2412
2862
2477
2253
2724
2446
2344
2710 29
1920
6829
74 2179
2247
2384
2417
2160
2955
2259
2899
2470
2533
2314
2473
2726
2324
2978
2840
2648
2995 2661
2691
2987
2747
2100
2619
2537
2612
2172
2970
2547
2877
2989
2800 2
157
2791
2313
2851
2116
2189
2480
2796
2196
2889 2347
2671 24
4728
3029
0328
0329
4520
5829
9023
1620
8621
0826
25 2272
2946
2910
2191
2391
2504
2364
2459
2633 27
4123
1723
6025
5824
2429
1426
1126
5128
2024
3928
7223
9325
8424
3625
1425
7620
6027
3821
4626
0725
6425
4329
1127
3028
0520
6222
9723
4122
0721
2128
1328
8128
8429
6120
6424
7922
5222
3021
8826
7320
7521
6823
3725
5927
9528
3529
0021
2822
7023
5223
9024
7420
8529
8823
7629
9329
5922
5628
4925
0624
7826
4724
4421
1323
8125
0523
0929
8523
6821
5128
2727
1923
7329
6824
23 2958
2063
2255
2545
2103
2814
2829
2392
2575
2569
2865
2953
2112
2229
2143
2378
2511
2823
2174
2810
2548
2675
2816
2131
2652
2692
2908
2088
2657
2282
2604
2682
2091
2162
2450
2956
2109
2269
2411
2271
2728
2225
2758
2302
2434
2371
2496
2587 27
6220
9229
4224
30 2632
2251
2260
2838
2960
2257
2717
2306
2828
2500
2553
2861
2891
2986
2071
2383
2617
2965
2998
2328
2348
2513
2476
2809
2413
2976
2836
2928
2084
2098
2546
2870
2245
2471
2158
2429
2276
2929
2428
2095
2549
2265
2410
2677
2898
2142
2781
2597
2815
2090
2329
2485
2311
2351
2096
2300
2811
2856
2201
2334
2408
2286
2421
2609
2416
2700
2695
2875
2219
2349
2374
2855
2864 2388
2235
2277
2487
2753
2566
2080
2137
2562
2531
2130
2322
2425
2173
2780
2232
2290
2683
2760
2293
2458
2984
2362
2789
2522
2684
2913 26
0821
8122
1422
3729
8023
0322
9427
3327
7020
5325
9624
6629
9626
4228
5727
9829
6720
7623
3125
5726
9420
9928
6721
8723
9922
0320
8722
2622
2024
5525
8125
9120
8224
6929
0129
7127
0128
6625
1927
2229
4427
8528
8220
9427
56 2831
2422
2552
2767
2457
2890
2894 2494
2634
2905
2643
2664
2054
2165 23
3921
0428
5827
0228
9720
8121
1122
7921
5224
9024
3327
35 2626
2208
2667
2790
2468
2528
2879 29
0623
5426
9324
6327
1829
9923
7926
3725
5025
0926
0027
8728
92 2541 2
154
2847
2197
2764
2377
2055
2120
2938
2825
2860
2169
2925
2149
2407
2213
2489
2672
2706
2806
2115
2697
2649
2954
2620
2888
2837
2869
2066
2703
2281
2629
2078
2284
2139
2602
2465
2521
2935 2409
2414
2964
2788
2832
2614
2737
2418
2947
2530
2097
2658
2669
2709
2878
2973
2122
2440
2720
2778
2818
2751
2592
2885
2615
2721
2123
2769
2133
2267
2400
2570
2786
2529
2305
2308
2930
2093
2641
2126
2397
2456
2752 21
6422
0624
8122
7422
3324
3524
49 2572
2740
2304
2660 2539
2363
2138
2834 2185
2234
2338 25
8325
0127
6325
4229
37 2924
2101
2119 27
5922
6421
4522
4027
75 2161
2350
2922
2650
2994
2712
2665
2057
2346
2624
2148
2499
2089
2175
2268
2707 2211
2190
2445
2843
2118
2228
2527
2573
2405
2236
2238
2370
2742
2398
2067
2765
2224
2848
2568
2746
2394
2680
2644
2209
2536
2896 2464
2361
2793
2178
2621
2902 2419
2404
2656
2438
2517
2319
2941 24
8822
0524
7524
6025
3227
04 2582
2278
2403
2246
2761 2951 22
2221
1421
3523
1226
4528
2228
6321
5623
5321
7028
6822
7322
3925
0229
3122
2323
5722
6623
3528
2425
5626
8926
2727
1421
1723
2327
9721
2425
1522
4925
6124
9221
3223
5827
0522
1728
4422
6324
5128
0224
0625
2526
3822
6228
7628
7324
6226
3924
9125
9423
1528
1223
8224
0124
9827
5524
1526
8129
9223
4326
3523
6526
2226
8623
6925
3526
5324
2725
7928
5228
8324
9329
6920
6123
5926
5429
97 2336
2766
2441
2484
2577
2725
2927
2301
2486
2442
2640
2065
2593
2192 2159
2204
2567
2743
2842
2110
2616
2295
2254
2782
2826
2258
2613 23
3020
8322
4221
4129
20 2794
2289
2510
2325
2588
2982
2340
2540
2623
2069
2589 2799
2943
2171
2729
2916 25
3422
0223
2128
74 2261
2495
2212
2287
2952
2977
2073
2696 26
7621
9426
3622
16 2144
2366
770
2940
2072
2285
2598 23
7221
5323
4225
0325
20 2432
2599
2176
2580
2732
2555
2854
2184
2727
2757 22
0022
4323
2027
74 2666
2590
2801
2605
2966
2210
2923
2663
020
4060
Dendrogram of agnes(x = dat, method = "average")
Agglomerative Coefficient = 0.99dat
Hei
ght
81
Using Dendograms
� Different ways of measuring dissimilarity result in different trees.� Dendograms are useful for getting a feel for the structure of
high-dimensional data though they don’t represent distances betweenobservations well.
� Dendograms show hierarchical clusters with respect to increasing valuesof dissimilarity between clusters, cutting a dendogram horizontally at aparticular height partitions the data into disjoint clusters which arerepresented by the vertical lines it intersects. Cutting horizontallyeffectively reveals the state of the clustering algorithm when thedissimilarity value between clusters is no more than the value cut at.
� Despite the simplicity of this idea and the above drawbacks, hierarchicalclustering methods provide users with interpretable dendograms thatallow clusters in high-dimensional data to be better understood.
82
Hierarchical Clustering on Indo-European Languages
Albanian_G
Albanian_C
Albanian_K
Albanian_T
Albanian_Top
Gypsy_Gk
Singhalese
Kashm
iriNepali_List
Khaskura
Hindi
Panjabi_ST
Lahnda
Bengali
Marathi
Gujarati
Afghan
Waziri
Ossetic
Wakhi
Baluchi
Persian_List
Tadzik Greek_K
Greek_D
Greek_Mod
Greek_ML
Greek_MD
Armenian_Mod
Armenian_List
HITTITE
TOCHARIAN_A
TOCHARIAN_B Irish_A
Irish_B
Welsh_N
Welsh_C
Breton_List
Breton_SE
Breton_ST
Latvian
Lithuanian_O
Lithuanian_STUkrainian
Byelorussian
Polish
Russian
Lusatian_L
Lusatian_U
Czech_E
Czech
Slovak
Macedonian
Bulgarian
Slovenian
Serbocroatian
Icelandic_ST
Faroese
Swedish_List
Swedish_Up
Swedish_VL Danish
Riksm
alEnglish_ST
Takitaki
German_ST
Penn_Dutch Frisian
Flem
ish
Dutch_List
Afrikaans Rom
anian_List
Vlach
Sardinian_N
Sardinian_L
Sardinian_C
French_Creole_C
French_Creole_D Provencal
French
Walloon Spanish
Portuguese_ST
Brazilian
Catalan
Italian
Ladin
0.0
0.2
0.4
0.6
0.8
1.0
83
K-meansPartition-based methods seek to divide data points into a pre-assignednumber of clusters C1, . . . ,CK where for all k, k� ∈ {1, . . . ,K},
Ck ⊂ {1, . . . , n} , Ck ∩ Ck� = ∅ ∀k �= k�,K�
k=1
Ck = {1, . . . , n} .
For each cluster, represent it using a prototype or cluster centre µk.We can measure the quality of a cluster with its within-cluster deviance
W(Ck, µk) =�
i∈Ck
�xi − µk�22.
The overall quality of the clustering is given by the total within-clusterdeviance:
W =K�
k=1
W(Ck, µk).
The overall objective is to choose both the cluster centres and allocation ofpoints to minimize the objective function.
84
K-means
W =K�
k=1
�
i∈Ck
�xi − µk�22 =
n�
i=1
�xi − µci�22
where ci = k if and only if i ∈ Ck.� Given partition {Ck}, we can find the optimal prototypes easily by
differentiating W with respect to µk:
∂W∂µk
= 2�
i∈Ck
(xi − µk) = 0 ⇒ µk =1
|Ck|
�
i∈Ck
xi
� Given prototypes, we can easily find the optimal partition by assigningeach data point to the closest cluster prototype:
ci = argmink
�xi − µk�22
But joint minimization over both is computationally difficult.
85
K-meansThe K-means algorithm is a well-known method that locally optimizes theobjective function W.Iterative and alternating minimization.
1. Randomly fix K cluster centres µ1, . . . , µK .2. For each i = 1, . . . , n, assign each xi to the cluster with the nearest centre,
ci := argmink
�xi − µk�22
3. Set Ck := {i : ci = k} for each k.4. Move cluster centres µ1, . . . , µK to the average of the new clusters:
µk :=1
|Ck|
�
i∈Ck
xi
5. Repeat steps 2 to 4 until there is no more changes.6. Return the partition {C1, . . . ,CK} and means µ1, . . . , µK at the end.
86
K-means
Some notes about the K-means algorithm.� The algorithm stops in a finite number of iterations. Between steps 2 and
3, W either stays constant or it decreases, this implies that we neverrevisit the same partition. As there are only finitely many partitions, thenumber of iterations cannot exceed this.
� The K-means algorithm need not converge to global optimum. K-meansis a heuristic search algorithm so it can get stuck at suboptimalconfigurations. The result depends on the starting configuration. Typicallyperform a number of runs from different configurations, and pick bestclustering.
87
K-means on Crabs
Looking at the Crabs data again.
library(MASS)library(lattice)data(crabs)
splom(~log(crabs[,4:8]),col=as.numeric(crabs[,1]),pch=as.numeric(crabs[,2]),main="circle/triangle is gender, black/red is species")
88
K-means on Crabscircle/triangle is gender, black/red is species
Scatter Plot Matrix
FL2.62.83.03.2
2.62.83.03.2
2.02.22.42.6
2.02.22.42.6●
●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●
●●● ●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●
●
●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●● ●●●●●
●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●
●●●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
RW2.42.62.83.0
2.42.62.83.0
1.82.02.22.4
1.82.02.22.4●
●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●
●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●
●●●●●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●
●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●● ●●
●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●
CL3.43.63.8 3.43.63.8
2.83.03.2
2.83.03.2●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●● ●
●●●●●●●●●●●●●●●●
●●● ●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●
●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●
●●● ●●
●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●
●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●
CW3.43.63.84.0
3.43.63.84.0
2.83.03.23.4
2.83.03.23.4●
●●●●●●●●●● ●
●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
● ●●●●
●●● ●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●
●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●
BD2.5
3.0 2.5 3.0
2.0
2.5
2.0 2.5
89
K-means on Crabs
Apply K-means with 2 clusters and plot results.
cl <- kmeans( log(crabs[,4:8]), 2, nstart=1, iter.max=10)
splom(~log(crabs[,4:8]),col=cl$cluster+2,main="blue/green is cluster finds big/small")
90
K-means on Crabsblue/green is cluster finds big/small
Scatter Plot Matrix
FL2.62.83.03.2
2.62.83.03.2
2.02.22.42.6
2.02.22.42.6 ●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●
●●●●●●●
●
●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●
●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●● ●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●
●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●
●●●●●●●●●●●●●●●●● ●●●●●●●
●●●●●●●●●●●●●●●●●●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●● ●●●●●
●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
RW2.42.62.83.0
2.42.62.83.0
1.82.02.22.4
1.82.02.22.4●
●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●
●
●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●
●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●
●
●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●
●
●●●●●
●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●
●
●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●
●●●●
●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●
●●●●●●
●
●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●
●
●●●●●
●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●● ●●
●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●
CL3.43.63.8 3.43.63.8
2.83.03.2
2.83.03.2●●●●●
●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●● ●
●●●●●●●●●●●●●●●●
●●● ●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●
●●●●●
●●●●●
●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●● ●●
●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●
●●●●●●●●●●●●●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●
CW3.43.63.84.0
3.43.63.84.0
2.83.03.23.4
2.83.03.23.4 ●●●●●
●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●● ●
●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●
●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
● ●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
● ●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●● ●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●
●●●●●
●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●
●
BD2.5
3.0 2.5 3.0
2.0
2.5
2.0 2.5
91
K-means on Crabs
‘Whiten’ or ‘sphere’1 the data using PCA.
pcp <- princomp( log(crabs[,4:8]) )spc <- pcp$scores %*% diag(1/pcp$sdev)splom( ~spc[,1:3],
col=as.numeric(crabs[,1]),pch=as.numeric(crabs[,2]),main="circle/triangle is gender, black/red is species")
And apply K-means again.
cl <- kmeans(spc, 2, nstart=1, iter.max=20)splom( ~spc[,1:3],
col=cl$cluster+2, main="blue/green is cluster")
1Apply a linear transformation so that covariance matrix is identity.92
K-means on Crabs
circle/triangle is gender, black/red is species
Scatter Plot Matrix
V11
2
3 1 2 3
−2
−1
0
−2 −1 0
●
●●●●● ●● ●● ●
●●● ●●
●●●
●● ●●●● ●● ●●● ●● ●●● ●● ●●● ●
●●● ●●
●●●
●
●●
●●●
● ●●●●●
● ●●●●● ●●●
●● ●● ●●●●●●
●● ●●●
●● ●●●●●●
●
●●●●●
●
●
● ●●●●● ●●
●●●
● ● ●●●
●●● ●●●●● ●●●●●●●●●●●
●●●●●
●●●● ●
●●●●
●●
●●●
●● ●●● ●
●● ●●●●●●●
● ●●●●●
●●●●●
●● ●●●●●● ●●
●●●
●●●●●
●
●
●
●●●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
●●
●
●
●
●●●●
●
●●
●
●●
●●
●
●●
●
●●●
●●
●●●
●
V20
1
2 0 1 2
−2
−1
0
−2 −1 0
●
●
●●●
●
● ●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
● ●
●
●
●
● ●●●
●
● ●
●
●●●●
●
●●
●
●●●
●●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●
●●
●
●
●●●●●●●●
●●●
●
●
●
●●●●
●●
●●●●
●
●
●●
●
●●
●●●●●
●●
●
●●●●
●
●●
●●
●
●
●
●●
●●
●●
●●
●
●●●
●●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●● ●
●●
●
●
●●●
●●●
●●
● ●●
●
●
●
●●●●
●●● ●
●●
●
●
●●
●
●●
●●●● ●
●●
●
● ●● ●
●
●●
●●
●
●
●
●●
●●
●●
●●
●
●●●
●●●●
●
V30
1
2 0 1 2
−2
−1
0
−2 −1 0
blue/green is cluster
Scatter Plot Matrix
V11
2
3 1 2 3
−2
−1
0
−2 −1 0
●
●●
●●
● ●● ●●
●●● ● ●●● ●●●●●● ● ●●●
● ●●● ●●● ●●●●● ●●●●
●● ●●●●
●
●
●●●●● ●● ●● ●
●●● ●●
●●●
●● ●●●● ●● ●●● ●● ●●● ●● ●●● ●
●●● ●●
●●●
●
●
● ●●●●
●●●
●●●●●● ●
●●●●●●
●●●
●●● ●● ● ●●●● ●●●●●●● ●
●●●● ●
●●
●●
●●●
● ●●●●●
● ●●●●● ●●●
●● ●● ●●●●●●
●● ●●●
●● ●●●●●●
●
●●●●●
●
●
●●●●
●●●●●
● ●●● ● ●●● ●●
● ●●●● ●●● ●●
●●●●● ●●● ●●● ●●
● ●●●●●
●
●
● ●●●●● ●●
●●●
● ● ●●●
●●● ●●●●● ●●●●●●●●●●●
●●●●●
●●●● ●
●●●●
●
●●●
●●●
● ●●
● ●●●● ●●● ●●●●●● ●
●●●●●● ●● ●● ●●● ●●●
●●●● ●● ●●●
●●
●●●
●● ●●● ●
●● ●●●●●●●
● ●●●●●
●●●●●
●● ●●●●●● ●●
●●●
●●●●●
●
●
●
●
●●●
●
●
●
●●●
●
●
●●
●●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●●
●●
●●●●
●
●
●
●
●●
●
●
●
●●●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●●●●
●●●●
●●●
●
●●●●
●
●
●
●
●●
●●●
●●●●
●●
●●●●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
●●
●
●
●
●●●●
●
●●
●
●●
●●
●
●●
●
●●●
●●
●●●
●
V20
1
2 0 1 2
−2
−1
0
−2 −1 0●
●
●
●●●
●
●
●
●● ●
●
●
● ●
●●
●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●●
● ●
●●●
●
●
●
●
●
●●
●
●
●
●●●
●
● ●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●●
●●●●
●●●●●●
●
●
● ●●●
●
●
●
●
●●
●●●
●●●
●
●●
●●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
● ●
●
●
●
● ●●●
●
● ●
●
●●●●
●
●●
●
●●●
●●
●●●
●
●
●
●●●●●●●
●
●
●●●●
●
●●
●●
●
●●●●
●
●
●
●●●
●●●
●
●●●●
●●
●●
●
●●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●
●●
●
●
●●●●●●●●
●●●
●
●
●
●●●●
●
●●
●
●
●
●
●●●
●
●●●●
●●
●
●●
●●
●●
●
●●
●●●
●
●
●
●●
●●
●●
●●●●
●●
●
●
●●●
●●
●●●●
●
●
●●
●
●●
●●●●●
●●
●
●●●●
●
●●
●●
●
●
●
●●
●●
●●
●●
●
●●●
●●●●
● ●
●
●●●● ●● ●
●
●
●●
●●
●
●●
●●
●
●●● ●
●
●
●
●●●
●●●
●
●●●●
●●
●●
●
● ●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●● ●
●●
●
●
●●●
●●●
●●
● ●●
●
●
●
●●●●
●
●●
●
●
●
●
●●
●●
●●●●
●●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●●
●●
●●
●●● ●●●
●
●
●●
●
●●● ●
●●
●
●
●●
●
●●
●●●● ●
●●
●
● ●● ●
●
●●
●●
●
●
●
●●
●●
●●
●●
●
●●●
●●●●
●
V30
1
2 0 1 2
−2
−1
0
−2 −1 0
Discovers gender difference...Results depends crucially on sphering the data first.
93
K-means on Crabs
Using 4 cluster centers.circle/triangle is gender, black/red is species
Scatter Plot Matrix
V11
2
3 1 2 3
−2
−1
0
−2 −1 0
●
●●●●● ●● ●● ●
●●● ●●
●●●
●● ●●●● ●● ●●● ●● ●●● ●● ●●● ●
●●● ●●
●●●
●
●●
●●●
● ●●●●●
● ●●●●● ●●●
●● ●● ●●●●●●
●● ●●●
●● ●●●●●●
●
●●●●●
●
●
● ●●●●● ●●
●●●
● ● ●●●
●●● ●●●●● ●●●●●●●●●●●
●●●●●
●●●● ●
●●●●
●●
●●●
●● ●●● ●
●● ●●●●●●●
● ●●●●●
●●●●●
●● ●●●●●● ●●
●●●
●●●●●
●
●
●
●●●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
●●
●
●
●
●●●●
●
●●
●
●●
●●
●
●●
●
●●●
●●
●●●
●
V20
1
2 0 1 2
−2
−1
0
−2 −1 0
●
●
●●●
●
● ●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
● ●
●
●
●
● ●●●
●
● ●
●
●●●●
●
●●
●
●●●
●●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●
●●
●
●
●●●●●●●●
●●●
●
●
●
●●●●
●●
●●●●
●
●
●●
●
●●
●●●●●
●●
●
●●●●
●
●●
●●
●
●
●
●●
●●
●●
●●
●
●●●
●●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●● ●
●●
●
●
●●●
●●●
●●
● ●●
●
●
●
●●●●
●●● ●
●●
●
●
●●
●
●●
●●●● ●
●●
●
● ●● ●
●
●●
●●
●
●
●
●●
●●
●●
●●
●
●●●
●●●●
●
V30
1
2 0 1 2
−2
−1
0
−2 −1 0
colors are clusters
Scatter Plot Matrix
V11
2
3 1 2 3
−2
−1
0
−2 −1 0
●
●●
●●
●●● ●●
●●● ● ●●● ● ●●●●● ● ● ●●
● ●●● ●●● ●●●●●●●●●
●● ●
●● ●
●
●
●●●●● ●● ●● ●
●●● ●●
●●●
●● ●●●● ●● ●●● ●● ●
●● ●● ●●
● ●●●
● ●●●●●
●
●
●●
●●●
●●●
●●● ●●● ●
●●●●●●
●●●
●●● ●● ● ●●●● ●●
● ●●●● ●
●●●
● ●●●
●●
●●●
●●●●●●
● ●●●●● ●●●
●●●● ●●●●●●
●● ●●●
●● ●● ●●●●
●
●●●●●
●
●
●●●●
●●●●
●● ●●● ● ●●● ●
●● ●●●● ●●
● ●●●●●●● ●●● ●
●● ●●
●●●●●●
●
●
● ●●●●● ●●
●●●
● ● ●●●
●●● ●●●●● ●●●●●●
●●●●●
●●●●●
●●●● ●
●●●●
●
●●
●●●
●● ●●
● ●●●● ●●● ●●●●●● ●
●●●●●● ●● ●● ●●
● ●●●●●
●●●
● ●●●
●●
●●●
●● ●●● ●
●● ●●●●●●●
● ●●●●
●●●●●
●●● ●●
●●●● ●●●●
●
●●●●●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●●
●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●●
●●
●●●●
●
●
●
●
●●
●
●
●
●●●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●●●●
●
●●●
●●
●
●
●●●●
●
●
●
●
●●
●●
●
●●●●
●●
●●●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
●●
●
●
●
●●●●
●
●●
●
●●
●●
●
●●
●
●●●
●●
●●●
●
V20
1
2 0 1 2
−2
−1
0
−2 −1 0
●
●
●
●●●
●
●
●
●
● ●
●
●
● ●
●
●●
●●
●
●
●
●●
●●
●
●
●
●
●●
●
●●
● ●
●●●
●
●
●
●
●
●●
●
●
●
●●●
●
● ●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●●
●●●●
●
●●●●●
●
●
● ●●●
●
●
●
●
●●
●●
●
●●●
●
●●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
● ●
●
●
●
● ●●●
●
● ●
●
●●●●
●
●●
●
●●●
●●
●●●
●
●
●
●●●
●●●●
●
●
●
●●
●
●
●●
●●
●
●●●●
●
●
●
●●●
●●●
●
●●●
●
●●
●●
●
●●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●
●●
●
●
●●●
●●●●●
●●●
●
●
●
●●●●
●
●●
●
●
●
●
●
●●
●
●●●●
●●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●●
●●
●
●
●●●●
●●
●
●
●●
●
●●
●●●●
●
●
●●
●
●
●
●●●●●
●●
●
●●●●
●
●●
●●
●
●
●
●●
●●
●●
●
●
●
●●●
●●●●
● ●
●
●●●●
●● ●
●
●
●
●●
●
●
●●
●●
●
●●● ●
●
●
●
●●●
●●●
●
●●●
●
●●
●●
●
● ●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●● ●
●●
●
●
●●●
●●●
●●
● ●●
●
●
●
●●●●
●
●●
●
●
●
●
●
●●●
●●
●●
●●
●
●●
●●
●●
●
●●
●●
●
●
●
●
●●
●●
●
●
●●● ●●●
●
●
●●
●
●●● ●
●●
●
●
●●
●
●
●
●●
●● ●●●
●
● ●● ●
●
●●
●●
●
●
●
●●
●●
●●
●
●
●
●●●
●●●●
●
V30
1
2 0 1 2
−2
−1
0
−2 −1 0
94
K-means on Spike Waveforms
library(MASS)library(lattice)spikespca <- read.table("spikes.txt")cl <- kmeans(data,6,nstart=20)splom(data,col=cl$cluster)
95
K-means on Spike Waveforms
Scatter Plot Matrix
V11234
1 2 3 4
−3−2−1
0
−3−2−1 0
●
●
●
●
●
●●●● ●
●
●●●●
●●●
●
●●
●●●●
●
●
●
●
●●
● ●●●
●●●●●
●●
●●●●●●
●●●
●
●●●●
●●
●●
●
●
●●●
●
●●
●
●
●
● ●
●
●
●
●
● ●
●●●
●
●
●
●●
●●●●
● ●●
●
●●●
●
●
●
●●
●
●●●●
●
●
●
●●
●●
●●●
●
●
●● ●
●●●
●●●●
●●●
●
● ●●●●●
●
●
●
● ●
●
● ●●●●
●
●
●
●●
●●
●
●
● ●
●
●
● ●●●● ●● ●●●●
●●●●
●●●●
●● ●●
●●
●
●
●
● ●
●●
●
● ●
●●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
● ●
●
●
●
●
●
●
● ●
●
●
●
●● ●
●●
●
●●●
●●
●●
●●
●
●
●● ●●●●
●●●● ●●● ●
●
●
●●
●
●●
●●
●
●●●●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●●●●●
●
●●
●
●
●
●
●
●
●
●●
●
●● ●●
●●
●●
●●●●
●
●●●
●
●
●
●
●
●
● ●
●
●●
●
●
●●●●
●
●
●
●●●
●
●●●●● ● ●●
●
● ●● ●
●
●● ●●
●
●
●
●● ●●
●●●
●●
●
●●●●●
●
●
●●
●
●
●
●●●●
●●●
● ●
●●
●
●
●
●
●●●●
●
●
●
● ●●
●
●
●●●●●●
●●
●
●●●●
● ●
●
●
●●●●●
●●●
●
●● ●●
●
●●●●
●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
● ●
●●●●
●●●
●
●●●
●
●
●
●
●
●
●●●●●
●●●●
●
● ●●●
●
●
●●●
●
●●
●
●●●●
●
●● ●●
●
●
●●●
●
●
●●
●●
●
●●●
●●
●
●
●●
●
●
●●
●
●
●●●●
●● ●
●
● ●● ●
●●
●
●
●●
●
●
●●●
●●
●
●
●●
●
●●
●
●
●●●●●●●●●
●●●●
●●
●
●
● ●●●●●
●
●●
●
● ●●●●●●
● ●●●
●●●
●●●●
●●
● ●●
●
●●
●
●●
●
●●●●●●●●
●
●● ●●●●●
●●
●●● ●●●●
●●
●
●
●
●
●●●
●
●●●●●●●
●
●●●●
● ●●
●●●
●● ●●
●●●●●
●
● ●●
●
●
●
●
● ●
●● ●
●
●
●
●●
●
●
●
●●●●●
●● ●
●●●
●
●
●●●
●●
●●
●
●
●
●
●
●
●●
●
●
●●●●●● ●●
●
●● ●●●●
●●
●●●●●
●
●
●●
●● ●●
●●
●
●
●●
●
●●●●
●●●●●
● ●
●
●●●●
●
●
●
●
●
●●
●
●● ●
●●
●●●●●●
●
●●●
●●
●
●●
● ●●
●
●● ●
●
●●●
●
●●●●●
●●
●●●●
●
●●●
●
●●
●●
●●
●●●●
●
●●
●
●
●●
●
●
● ●●
●●
●
●●●●
●
●
●●●●●
●●
●●●●
●
●
●
●●●●●●
●
●●●
●●
●●●●
●●●●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●
●●●●
●●●●● ●
●●●●
●
●●●●
●
●
●●●
●●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●
●●●●●
●
●●●●●●●
●
●●●●●●
●
●
●
●
●●
●●●●●●●●●●
●
●●●●●●
●●●
●
●●●●
●●
●●
●
●
●●●●
●●
●
●
●
●●
●
●
●
●
●●
●●●
●
●
●
●●
●●●●●●●
●
●●●
●
●
●
●●
●
●●●●●
●
●
●●
●●
●●●
●
●
●●●
●●●
●●●●●●●
●
●●●●●●
●
●
●
●●
●
● ●●●●
●
●
●
●●
●●
●
●
●●
●
●
●●●●●●●●
●●●
●●●●●●●●●●●
●●●
●
●
●
●●
●●
●
● ●
●●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●●●
●●
●●
●●
●
●
●●●●●●●●●●●●●●
●
●
●●
●
●●
●●
●
●●●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●●●●
●●
●
●
●
●
●
●
●
●●
●
●●●●●●●
●
●●●●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●●●●
●
●
●
●●●
●
●●●●●●●●
●
●●●●
●
●●●●
●
●
●
●●●●●●●
●●
●
●●●●●●
●
●●
●
●
●
●●●
●●●●●●
●●
●
●
●
●
●●●●●
●
●
●●●
●
●
●●●●●●
●●
●
●●●●
●●
●
●
●●●●●●●
●
●
●●●●
●
●●●●
●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●●●●
●●●
●
●●●
●
●
●
●
●
●
●●●●●●●●●
●
●●●●
●
●
●●●
●
●●
●
●●●●●
●●●●●
●
●●●
●
●
●●
●●●
●●●
●●
●
●
●●
●
●
●●
●
●
●●●●
●●●
●
●●●●●●
●
●
●●
●
●
●●●
●●●
●
●●
●
●●●
●
●●●●●●●●●
●●●●
●●
●
●
●● ●●●●
●
●●
●
●●●●● ●●●●●●
●●●●●●●●●
●●●
●
●●
●
●●
●
●●●●●●●●
●
●●●●●●●
●●
●●●●●●●
●●
●
●
●
●
●●●
●
●●●●●●●●
●●●●●●●
●●●●●●●●●
●●●
●
●●●●
●
●
●
●●
●●●●
●
●
● ●
●
●
●
●●●●●
●●●
●●●●
●
●●●
●●●●
●
●
●
●
●
●
●●
●
●
●● ●●●●●●
●
●●●●●●●●●●●●●
●
●
●●
●●●●
●●
●
●
●●
●
●●●●●●●●
●
●●
●
●●●●
●
●
●
●
●
●●
●
●●●
●●
●●●●● ●
●
●●●
●●
●
●●●●●
●
●●●
●
●●●●
●●●●●
● ●
●●●●●
●●●
●
●●
●●●●
●●●●
●
●●
●
●
●●
●
●
●●●
●●●
●●●●
●
●
●●●●●
●●
●●● ●
●
●
●
●●● ●●●
●
●●●
●●
●● ●●●●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●●●●●●●
●●●●●●●●●●
●
●●●●
●
●
●●●●●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●
●●●● ●
●
●●●●● ●●
●
●●
●●●●
●
●
●
●
●●
●●●●
●●●●●
●●
●●●●
● ●
●●●
●
●●●●
●●
●●
●
●
●●●●
●●
●
●
●
● ●
●
●
●
●
● ●
●●●
●
●
●
●●
●●
●●
●●●
●
●●●
●
●
●
●●
●
●●●●●
●
●
●●
●●
●●●
●
●
●● ●
●●●
●●●●
●●●
●
●●●●●●●
●
●
●●
●
●●●●●
●
●
●
●●
●●
●
●
●●
●
●
● ●●●●●● ●●●●
●●●●●●●●
● ●●●
●●
●
●
●
● ●
●●
●
●●
●●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●●
●●
●
●●●
●●
●●
●●
●
●
●●●●●●
●●●●● ●● ●
●
●
●●
●
●●
●●
●
●●●●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●●●● ●●
●●
●
●
●
●
●
●
●
●●
●
●●●●●●
●●
●●●●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●●●●
●
●
●
● ●●
●
●●●●●● ●●
●
●●● ●
●
●●● ●
●
●
●
●●●●●●
●●●
●
●●●●●●
●
●●
●
●
●
●●●●●
●●●●
●●
●
●
●
●
●●● ●●
●
●
● ●●
●
●
●●●● ●●
●●
●
●●●●
●●
●
●
●●●
●●● ●
●
●
●●●●
●
●●●●
●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
● ●
●●●●
●●●
●
●●●
●
●
●
●
●
●
●●●●●
●●●●
●
●● ●●
●
●
●●●
●
●●
●
●●●
●●
●●●●
●
●
●●●
●
●
●●
●●●
●●●
●●
●
●
●●
●
●
●●
●
●
●●●●
●●●
●
●●●●●
●
●
●
●●
●
●
●●●
●●
●
●
●●
●
●●●
●
●●● ●●●●●●
●●●●
●●
●
●
●●●●●●
●
●●
●
● ●●●●●●
●● ●●
●●●
●●●●
●●
●●●
●
●●
●
●●
●
●●●●●●●●
●
●●●
●●●●
●●
●●●●●●●
●●●
●
●
●
●●●
●
●●●●●●●
●
●●●●● ●
●●● ●
●●●●●●●
●●
●
●●●●
●
●
●
●●
●●●●
●
●
●●
●
●
●
●●●●●
●●●●
●●
●
●
●●●
●●
●●
●
●
●
●
●
●
●●
●
●
●●●
●● ●●●
●
●●●●●●
●●
●●●●●
●
●
●●
●● ●●
●●
●
●
●●
●
●●●●●● ●●●
●●
●
●●
●●
●
●
●
●
●
●●
●
●●●
●●
●●●●● ●
●
●●●
●●
●
●●
●●●
●
●●●
●
●●●●
●●●●●
●●
●●●●●
●●
●
●
●●
●●
●●
●● ●●
●
●●
●
●
●●
●
●
● ●●
●●●
●●●●
●
●
●●●
●●
●●
●●●●
●
●
●
●●● ●●●
●
●●●
●●
●● ●●●●● ●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●●
●●●●●
●●●●● ●●●●●
●
●●● ●
●
●
●●●
●●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●
●● ●● ●
●
●● ●●● ●
●
●
●●
●●●●
●
●
●
●
●●
● ●●●
●● ●●
●●
●
●●●●
●●
●●●
●
●●●
●
●●
●●
●
●
● ●●
●
●●
●
●
●
● ●
●
●
●
●
●●
●●●
●
●
●
●●
●●●●
●● ●
●
●●●
●
●
●
●●
●
●● ●●
●
●
●
●●
●●
●●●
●
●
●● ●
● ●●
●●●●
●●●
●
●●●●●
●●
●
●
●●
●
●●●●●
●
●
●
●●
●●
●
●
●●
●
●
● ●● ●● ●● ●●●●
●●●●●
●●●
● ●●●
●●
●
●
●
●●
● ●
●
●●
●●
●
●●
●
●
●●
●
●
● ●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●●●
●●
●
●●●
●●
●●
●●
●
●
●●● ●●●
● ●● ●●● ●●
●
●
●●
●
●●
●●
●
●●●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●●●●
●●
●
●
●
●
●
●
●
●●
●
●● ●●●
●●
●
●●●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●● ●●
●
●
●
● ●●
●
●●●●●● ●●
●
●●● ●
●
●●●●
●
●
●
●● ●●
● ●●
●●
●
●●●●●
●
●
●●
●
●
●
●●●
●●
●●●●
●●
●
●
●
●
●●●●●
●
●
● ●●
●
●
●● ● ●●●
● ●
●
●● ●●
●●
●
●
●●●
●●● ●
●
●
●●●●
●
●●
●●
●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●●●●
●●●
●
●●●
●
●
●
●
●
●
●●●●●
●●●●
●
●●● ●
●
●
●●●
●
● ●
●
●●●
●●
● ●●●
●
●
●●●
●
●
●●
●●●
●●●
●●
●
●
●●
●
●
●●
●
●
●●●●
●●●
●
●●● ●
●●
●
●
●●
●
●
●●●
●●
●
●
●●
●
●●●
●
●●● ●●●● ●●
● ● ●●
●●
●
●
●● ●●●●
●
●●
●
● ●●●●●●
●● ●●
●●●
●●●●●●
● ●●
●
●●
●
●●
●
●●●●●
● ●●
●
●●●
●●● ●
●●
●●●●
●●●
●●
●
●
●
●
● ●●
●
●● ●●●● ●●
●●●●● ●
●●
●●●●●
●●●●
●●
●
●●●
●
●
●
●
●●
●● ●
●
●
●
● ●
●
●
●
●●●● ●
●●●
●●●
●
●
●●●
●●
●●
●
●
●
●
●
●
●●
●
●
● ●●● ● ●● ●
●
●●●●● ●●
●●● ●●●
●
●
●●
●● ●●
●●
●
●
●●
●
●●●●●●● ●
●
●●
●
●●
●●
●
●
●
●
●
●●
●
●●●
●●
●●
● ●● ●
●
●●●
● ●
●
●●
●● ●
●
●●●
●
● ●●
●
●● ●● ●
● ●
●● ●●
●
●●●
●
●●
●●
●●
●●●●
●
●●
●
●
●●
●
●
●●●
●●
●
●●●●●
●
●●●●●
●●
●●● ●
●
●
●
●● ●● ●●
●
●●●
●●
●●●●
●●●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●●● ●
●
●●
●●● ●●●●●
●
●● ●●
●
●
●●●●●
●
●
●
●
● ●
●
●
●●●●
●
●
●●
●●
●●●
●
● ●●●●●
●
●●
●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●● ●●●●●
●●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●
●
●●●
●
●
●
●●●●
●●
●
●●
●
●●
●●●
●●
●
●●
● ●●●
●
●
●
●
●
●●●
●●
●
●
●●●
●
●
●●●●
●
●
●●●
●
●
●
●●●●
●
●
●
●●
●
●●
●
●
● ●
● ●
●
●●●
●
●
●
●●●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●●
●
●● ●
●
● ●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
● ●●
●● ●●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●●
●●●●
●
●
●
●
●●
●
●
●
●
●
●●●●
●●
●
●● ●
●
●
●
●
●
●
●●
●●
●
●
●●
●●●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●●● ●● ●●
●
●
●●
●
●●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●●
●
●
●
●●●●●●
●
●●
●
● ●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●●●
●●
●
●
●
●●●●●●
●
●
●
●●
●
●
●
●
●●●
●●●
●
●
●
●
●
●●●
●
●
●
●●
●●●
●●
●
●●●
●
●●● ●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●● ●
●
●
●
●●●●
●
●
●●
●
●
●●
●
●● ●●
●
●
●
●●
●
●●
●
● ●
●
●
●
●
●
●●
●●
●●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●● ●●
●●
●
●●●
●
●●●
●●
●
●●
●
●
●
●●
●
●●●●●
●
●
●
●●
●
●●●●●
●
●
●
●●●●
●●●
●
●●
●
●
●
●
●
●
●
●●●
●●●●●
●●● ●
●
●
●●●●●
●●
●
●●
●
●●●●
●
●●
●
●
●●●
●
●●●●
●
●
●
●●● ●●●
●
●
●
●●
●
●
●●●●●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●●
●
●●●●
●●
●● ●
●
●●
●
●
●
●
●●
●
●●●●
●
●●
●
●
●
●●●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●●●●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●●●
●
●●●●●
●
●●
●
●
●
●
●●
●
●
●●●●
●
●● ●
●●●
●●●
●
●
●
●●●
●●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●
●●●●●●
●●●●●●●
●
●●●●
●
●
●●
●●●●●●
●●●●
●
●●
●
●
●
●●●
●●
●●● ●●
●●● ●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●●●●●●
●●●●●
●
●
●●●●
●
●●●●
●●
●●●
●
●
●
●●
●●●
●
●
●●●● V20
12 0 1 2
−2−1
0
−2 −1 0
●
●
●●
●●
●●●
●
●●●●●●
●
●●
●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●●●●●●●
●●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●
●
●●●
●
●
●
●●●●
●●
●
●●
●
●●
●●●
●●
●
●●
● ●●●
●
●
●
●
●
●●●
●●
●
●
●●●
●
●
●●●●●
●
●●●
●
●
●
●●●●
●
●
●
●●
●
●●
●
●
●●
●●
●
●●●
●
●
●
●●●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●●
●
●●●
●
●●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●●●
●●●●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●●
●●●●
●
●
●
●
● ●
●
●
●
●
●
●●●●
●●
●
●●●
●
●
●
●
●
●
●●
●●
●
●
● ●
●●●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●●●●●●●
●
●
●●
●
●●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●●
●
●
●
●●●● ●●
●
●●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●●●
●●
●
●
●
●●●●●●
●
●
●
●●
●
●
●
●
●●●
●●●
●
●
●
●
●
● ●●
●
●
●
●●●●●
●●
●
●●●
●
●●●●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●●
●
●
●
●●●●●
●
●●
●
●
●●
●
●●●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●●●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
● ●●
●
●
●●
●
●●●●●●
●
●●●
●
●●●
●●
●
●●
●
●
●
●●
●
●●●●●
●
●
●
●●
●
●●● ●●
●
●
●
●●●●
●●●
●
●●
●
●
●
●
●
●
●
●●●
●●●●●
●●●●
●
●
●●●●●
●●
●
●●
●
●●●●
●
●●
●
●
●●●
●
●●●●
●
●
●
●●●●●●
●
●
●
●●
●
●
●●●
●●
●● ●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●●
●
●●●●
●●
●●●
●
●●
●
●
●
●
●●
●
● ●●●
●
●●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●●●●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●●●●
●
●●●●●●
●●
●
●
●
●
●●
●
●
●●●●
●
●●●
●●
●
●●●
●
●
●
●●●●●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●
●●●
●●●●
●●●●●●●
●●● ●
●
●
●●
●● ●●●●
●●●●●
●●
●
●
●
●●●
●●
●●●●●●●●●
●
●
●●
●
●
●
●●
●
●
●
●
●●●
●●●●●●
●●●●●
●
●
●●●●●
●●●●
●●
●●●
●
●
●
●●
●●●
●
●
●●●●
●
●
●●
●●
●●●
●
●●●●●●
●
● ●
●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●●●●●● ●
●●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●
●
●●●
●
●
●
●●●
●
●●
●
●●
●
●●
●●●
●●
●
●●
●●●●
●
●
●
●
●
●●●
●●
●
●
●●●
●
●
●●●●●
●
●●●
●
●
●
●●●●
●
●
●
●●
●
●●
●
●
●●
● ●
●
●●●
●
●
●
●●●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●●
●
●● ●
●
●●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●●●
● ●●●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●●
●●●●
●
●
●
●
●●
●
●
●
●
●
●● ●●
●●
●
●●●
●
●
●
●
●
●
●●
●●
●
●
● ●
●●●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●●
●
●●● ●● ●●
●
●
●●
●
●●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●●
●
●
●
●●●● ●●
●
●●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●●●●●
●
●
●
●●● ●●●
●
●
●
●●
●
●
●
●
●●●
●●●
●
●
●
●
●
● ●●
●
●
●
●●
●●●
●●
●
●●●
●
●●●●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●●
●
●
●
●●●●
●
●
●●
●
●
●●
●
●●●●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●●●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●● ●●
●●
●
●●●
●
●●●
●●
●
●●
●
●
●
●●
●
●●●●●
●
●
●
●●
●
●●●●●
●
●
●
●●●●
●●●
●
●●
●
●
●
●
●
●
●
●●●
●●●●●●●●●
●
●
●●●●●
●●
●
●●
●
●●●●
●
●●
●
●
●●
●
●
●●●●
●
●
●
●●●●●●
●
●
●
● ●
●
●
●●●●●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●●
●
●●●●
●●
●●●
●
●●
●
●
●
●
●●●
● ●●●
●
●●
●
●
●
●●●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●●
●●● ●
●
●
●
●
●
●● ●●
●
●
●
●
●
●●●
●
●
●
●●
●
●●●●●
●
●●●
●●●
●●
●
●
●
●
●●
●
●
●●●●
●
●● ●
●●
●
●●●
●
●
●
●●●
●●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●
●●●●●●●
●●●●●●
●
●●●●
●
●
●●
●● ●●●●
●●●●●
●●
●
●
●
●● ●
●●
●●●●●●●● ●●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●●●●●●
●●●●●
●
●
●●●●●
●●● ●
●●●●●
●
●
●
●●
●●●
●
●
●●●●
●
●
● ●
●●
● ●●
●
●●● ●●●
●
● ●
●
●
●●●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●●●●●
●●
●●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●●
●
●●●
●
●
●
●●●
●
●●
●
●●
●
● ●
●●●
●●
●
●●
● ●●●
●
●
●
●
●
● ●●
●●
●
●
●●●
●
●
●●●●
●
●
●●●
●
●
●
●●●●
●
●
●
●●
●
●●
●
●
● ●
●●
●
●●●
●
●
●
●●●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●●
●
● ● ●
●
● ●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
● ●●
●●●●
● ●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●●
●
●●
● ●●●
●
●
●
●
● ●
●
●
●
●
●
●●●●
●●
●
●● ●
●
●
●
●
●
●
●●
●●
●
●
●●
●●●
●
●
●● ●
●
●●
●
●
●
●
●
●●
●
●●
●
● ●● ●● ●●
●
●
● ●
●
●●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●●
●
●
●
●●●●●●
●
●●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●●●
●●
●
●
●
● ●●●●●
●
●
●
●●
●
●
●
●
●●●
●●●
●
●
●
●
●
●● ●
●
●
●
●●
● ●●
●●
●
●●●
●
●●● ●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
●
●●●
●
●
●
●●●
●●
●
●●
●
●
●●
●
●●●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●● ●●●●
●
●●●
●
●●●
●●
●
●●
●
●
●
●●
●
●●●●●
●
●
●
●●
●
●●●●
●
●
●
●
●●●●
●●●
●
●●
●
●
●
●
●
●
●
●● ●
●●●●●
● ●●●
●
●
●●●●
●
●●
●
●●
●
●●●●
●
●●
●
●
●●●
●
●●●●
●
●
●
●●●●●●
●
●
●
●●
●
●
●●●
● ●
●● ●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●●●
● ●
●
●
●●
●●
●
●●●●
●●
●● ●
●
●●
●
●
●
●
●●
●
●●● ●
●
●●
●
●
●
●●●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
● ●
●●●●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●● ●●●
●
●●●●
●●
●●
●
●
●
●
●●
●
●
● ●●●
●
●●●
●●
●
●● ●
●
●
●
●●●●●
●
●
●
●●
●
●●
●
●●
●
●
● ●
●
●
●
●●
●
●
●●●
●●●●●
●●●●●●
●●● ●
●
●
●●
● ●● ●●●
●●●●
●
●●
●
●
●
●●●
●●
●●● ●●
●●● ●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
● ●●●● ●
●●●●●
●
●
●●●●
●
●● ●●
●●
●●●
●
●
●
●●
● ● ●
●
●
●●●●
●●●● ●
●●●●●● ●●●●●●●●● ●●●●●● ●● ●●
● ●●●● ●●●●●●●● ●●●●● ●●
● ●●●●●● ●
●
●
●● ●
●●●●● ●
● ●●●
●● ●●
●●
●●●
● ●●
●●●●●●●●● ●●●●
● ●●
●●
●●●●●● ●● ●
●●
●
●●●
●
●
●●●
●●●
●●●●●●● ●●●●●●●
●
●
● ●●●
●
●
●●●●
●●●●
●● ●● ●●
●● ●●●●●●●●
●●●●●● ●●●●●●●●
●●●
● ● ●●● ●● ●
●● ●●●
●●●●
●●
●
●●●● ● ●●
●●●●●
●
● ●● ●●●● ●● ●
●
● ●●●
●● ●●●●●
● ●●●●●●
●●●●●●
●●●●●●●●
●
●●●
●●
●●●
●
●●●●
●●
● ●● ● ●●●●
●
● ●
●●●●●●● ●●●
●●
●
●
●●
●●● ●
●●●●●●
●●●●●
●●●● ●●
●
●● ●●● ●●● ●● ●●●● ●● ●●●● ●●
●●●●●●●● ●●
●
●
● ●●●●
●● ●●●●●●●●
●●● ●
●●
●
●●
●●●● ●
●●●
●●●●●●●●● ●●●●
●●●●● ●
● ●●●
●
●
●●●●●●●●●●●●●
●
● ●
●
●●●●●●●●
● ●●●
●
●
●●●●●●●●●
●● ●
●
●
●
●● ●●● ●●● ●
●●●●●● ●●● ●●●●
●●●●
●
●●●●
●●●●●●
●●●●●
●
●●●●
●●●
●
●●●●●●●●●●● ●●●●●
●
●
●
● ● ●●●●
●●
●
●● ●●
●●● ●●●● ●●●●● ●●●● ●●
●● ●●● ●
●●
●
●●●
●
●● ●●●●
●
●●●●●●●●● ●●●●
●●
●
● ●●
●●●●● ●●● ●●●●●
●
●●●●●●●●
●●●●●●
●●
●●
●●● ●
●● ●
●●●●
●●● ●●
●●●●●
●●●●●●●●●●●
●● ●
●● ●
●●
●
●●●●●●●
●●● ●●●●●
●●●●●●●●●
●
●●
●
●●●●
●
●
●●●●
●●●
●●●
●
●●● ●●●●● ●●
●●●●●
●●●●●●
●● ●●● ●●●
●●
● ●●●
●
●●●●●●●●●●●●
●●●●●
●●
●
●●●
●●●
●●●
●
●
●●● ●●●●●●●●●●●●
●●●● ●
●
●●
●
●
●●
●●●●● ●●●●
●●●
●●●●● ●●●●●● ●●●●
●
●●●●
●●● ●●●
●●●●●●
●●●
●●● ●●●●●●●●
●●●● ●●●● ●●●● ●●●●
●●●●
●●●●●●●●
●●
●●●
● ●●●●●●●●
●
●●●● ●
●
● ●●●
●●●●● ●● ●●●
●● ●●
●
● ●●●
● ●●●●
●●
●●● ●●●●
●●●●●●●●●●●●●
●●
●●●●●
●●●
●●● ●
●
●●● ●●
●●●● ● ●●●●
●●●● ●●●●●●●● ●● ●● ●●●●●● ● ●●
●● ●●●● ●●● ●●●●●●●●●●●
● ●●●
●●● ●
●
●
●● ●
●● ● ●● ●
●●●●
●● ● ●
● ●
●●●
●●●
●●●
●●●● ● ●●●●●
●●●
●●
●●●●●● ●● ●●
●
●
●●●
●
●
●●
●
●●●
●●●● ●●● ●● ●●●
●●
●
●
●●●
●
●
●
●●● ●●●●
●●
●● ●● ●●●● ●●●●
●●●
●●●
●●●●
●●●● ●● ●
●●●
●● ●●●●●●
●●●●
●●●
●●●●
●
●●
●● ●● ●●● ●●
●
●
●● ● ●●● ●● ●●
●
●●● ●
●●● ●●●●
●●●●●●●●● ●●
●●●●●
●●●● ●
●
●●●
●●
●●●
●
●●
●●●
●●●●● ● ●●
●
●
●●
●●● ●●●
●●●●
●●●
●
●●
●●●●
● ●●
●● ●●
●●
●●
●●●●●●
●
●●● ● ●●● ●●● ●●●●●●●●● ● ●●
●●●● ● ●
●●● ●●
●
● ●●●●●
● ●●● ●●●● ●●
● ●●●●
●
●●
●●●●●
●●●●●●●
●● ● ●●● ● ●
●●● ●●●●
●●●●
●
●
●●●●● ● ●●●
●●●●
●
●●
●
●●●●●●
● ●● ●
●●
●
●
●●●●●●●
●●●
●●
●
●
●
● ●● ●●●●
●●● ●
●●●●●●
●● ●●●
●●●●
●
● ●● ●
●●●●●● ●
● ●●●
●
●●●●
●●●
●
●●●● ●●
● ● ●●● ●●●●●
●
●
●
●● ●●● ●
●●
●
●● ●
●●● ●●
●●
●● ●● ●● ● ●● ●● ●
●●●●● ●
●●
●
● ●●
●
●●●●●●
●
●●●●●●●
●● ●●●●
●●
●
●●●
●●●●● ●● ●● ●●●●
●
●●●
●●●●●
●●● ● ●
●● ●
●●
●●● ●●●●
●●●●
●●●●●
●●●●●
●●●●●
●●●
●●●
●● ●
●●●
●●
●
●●●● ●●●
●●●●●
● ●●
●●● ●● ●●●
●●
●●
●
● ●● ●
●
●
● ● ●●●
●●●
●●
●
●●●●●●
●●●●●●
●●●●
●●● ●●●●●● ●●●
●●
●●●
●●
●
●●●
●● ●● ● ●●●●
●●●●●●●
●
●●●
●● ●
●● ●
●
●
●●● ●●●
●●●●● ● ● ●
●
●●●● ●
●
● ●
●
●
●●
●●● ●●●●●●●
● ●●●●●
●●●●●●
● ●●●●
●
●●● ●
●●●●●●
● ●●●
●●
●●●●●●
● ● ●●● ●●●●
●● ●●●● ●●● ●●●● ●●●●●●
●●●●●●●●
●●
●● ●
●●●●●●●●●●
●●●●●
●
● ●●●●●
●● ● ●●●●●
●●●●
●
●● ●●
● ●● ●●●
●●●
● ●●●●●● ●●●●●● ●●●●●●
●●●● ●●
●●●
●● ●●
●
●●● ●●
●●●●
V30
50 5
−10
−5
−10 −5
● ●●● ●●
●●● ●●●●●●● ●● ●
●● ●●●●●●●●●●●●●●● ●●● ●●●●●●●● ●●●
●●●●●●●●●
●
●● ●
●●●●●●●●●
●
●●● ●
● ●
●●●
●●●
●●●●●● ●●●●●●●●●
●●●
●●●●●●●● ●●
●
●
●●●
●
●
●●
●
●●●
●●●●●●●● ●●●●●●●
●
●●●●
●
●
●●● ●●●
●●
●● ●●●●●● ● ●●●●
●●●
●●●●●●●●●●●● ●●
●●●
● ●●●
●●● ●●
●●●●
●●● ●
●●
●
●●
●●●● ●●●●●●
●
●●●●●●●●●●
●
●●●●●●●●●●
●
●● ●●●
● ●●●●●
●●●●●
●● ●● ●
●
●●●
●●
●● ●
●
●●
●●●
●●●●●● ●●●
●
●●
●●●●● ●
●●●●●
●●
●
●●
●●●●●●●
●●●●●
●●●
●●●●●●
●
●●●●●●● ●●● ●●●● ●● ●● ●●●●●●●●● ●
●● ●●●
●
●●●● ●●
●● ●●●●●●● ●●●●
●●
●
●●
●●●●●
●●●●●●●
●●●●●● ●●
●●●●●●●
●●●●
●
●
●●●● ●● ●●●●●●●
●
●●
●
●●● ●●●
●●●●●
●
●
●
●●●●●●●●●●
●●
●
●
●
●●●●●●●●●● ●●
●●●●●●●●
●●●●●
●
●
● ●●●
●●●●●●●
●● ●●
●
●●●●
●●●
●
●●● ●●
●●● ●●●●● ●
● ●
●
●
●
●●●●●●
●●
●
●● ●
●●● ●●●●
●●●●●
●●●●●● ●
●●●●●●
●●
●
● ●●
●
●● ●●●●
●
●●● ●●●●●●●●●●●
●●
● ●●
●●●●● ●●●●
●●●●
●
● ●●
●●●●
●●●
●● ●●
●●
●●
●●● ●●●●●●●●●●●●
●●●●●●
●●●●●●
●●●●●
●●●●
●●●●
●
●●●● ●●●
●●●●●● ●●
●●
● ●●●●●●●
●●
●
●●●●
●
●
● ●●●●●●●●●
●
●●●●●●●●●●
●●●●●
●●●● ●●
●●●●●●● ●
●●
●●●●
●
●● ●
●● ●●●●
●●●
● ●●●●
●●
●
●● ●
●● ●
●● ●
●
●
●●● ●●●●●● ●●●●●●
●● ●●●
●
●●
●
●
●●
●●●●●●
●●●●
●●●●●
●●●●● ●
●●●●●
●
●
●●●●
●●● ●●●
●●●
●●●
●●●
●●●●●● ●
●● ●●●●●●●●●●●● ●●●●●●●●●●●
● ●●●●●●
●●
●●●●●● ●
●●●● ●●
●●●●●
●
●●●●● ●
●●●●●●●●●● ●●
●
●●●●●●●●
●●
●●●
●●● ●●●●●●●●●● ●●●
●●●●●●●●
●●●●●● ●●
●
●●● ●●●●●● ● ●
● ●●●
● ●● ●
●●● ●●● ●● ●●● ●●●●●●●●●
●● ●●●● ● ●●●●●●●●●●●●●●●●●
●● ●●●
●
●●●
●●●●● ●
● ●●●
●● ●●
●●
●●●
● ●●
●●●
●●●●● ● ●●● ●●●
●●
●●
● ● ●● ●● ●●●●
●
●●●
●
●
●●
●
● ●●
●●●●●●●●●●● ●●
●●
●
● ●●●
●
●
●●● ●
●●●●
●● ● ●●●
●●● ●● ●●●●
●●●●
●●● ●
●●●●● ●●
●●●●● ●●
●● ● ●●
● ●●●
●●●●
●●
●
●●
●●●● ●●●●●
●
●
● ●● ●●●●● ● ●
●
●●●●●●●●●●
●
●●●●●●●
●●●●
●●● ●●
●●● ●●
●
●● ●
●●
●● ●
●
●●
●●●
●● ●●● ●● ●
●
●
●●
●●●●●●● ● ●
●●
●●
●
●●●●
●●● ●
●● ●●
●●●
●●
●●● ● ●●
●
●●● ●●
●● ●● ● ●● ●● ●● ●● ●●● ●●
●●●● ●
● ●●●●
●
● ●●●●
●● ●●● ●●● ●●
●●●●
●●
●
●●●
● ●●●●
●●●
●●●●
●● ●● ●●●●
●●●● ● ●●●
●●●
●
●● ● ●●● ●●●●
● ●●
●
●●
●
● ●●●●●
●●● ●●●
●
●
●●
●●●●●●●●
●●
●
●
●
●●● ●●● ●●● ●●
●●●●●●● ●●●●
●●●
●
●
●●● ●
●● ●●●●
●●●● ●
●
●●●●●
● ●
●
●●●● ●
●●● ●● ● ●●●
● ●
●
●
●
●●●●● ●
●●
●
●●●●
● ●●●●●
●● ● ●●●●●● ●●●
●●●●● ●
●●●
● ● ●
●
●● ●●●●
●
●●● ●●●
●●●● ● ●●
●●
●
● ●●
●●
●● ●●● ●● ●●●●
●
● ●●
●●●●
●●●●●●●● ●
●●
●● ●●
●●●
●●●●
● ●●●●●
● ●●●
●●● ●●●●●
●● ●
●●●
●●●
●●
●
●● ●●●●●
●●●●●● ●●
●●
●●●●●●●
●
●●
●
●●● ●
●
●
● ●●●
●●●
●●●
●
● ●●●●●● ●● ●
●●●●●●
●●●●●●● ●
●●● ●●
●●
● ●●
●
●
●● ●
● ● ●●●●●
●●
● ●●● ●●●
●
●●●
●● ●
●●●
●
●
●● ●●● ●●●●● ●● ●●
●
●●●● ●
●
● ●
●
●
●●
●●● ●●●
●● ●●
●●●
●●●
● ●●● ●●
● ●●●●
●
●●●●
●● ●●●●
●●●
●●
●●
●●●●●
●● ●●● ●●●
●●●●
●● ●● ●●●●● ●●●●
●●●●●●
●●●●●●●
●● ●
●●●●●
●●● ●
●
●●●●●
●
●● ●●●●
●●● ●● ●●●
●● ●●
●
● ●●●● ●●● ●●
●●●
●● ●●●● ●●●●● ●● ●●●●●
●●
● ●● ●
●●●
●●●●●
●
● ● ●●●
●●●●
●
●
●
●
●●
●
●●
●
● ●●●●
●
●
●
● ●
●
●●●●● ●●
●● ● ●●
●● ●
●●●●
●●● ●●●●
●
●●●
●●●
●●
● ●●
●
●●
●●●●
●● ●● ●●●
●● ●
●
●
●●●●
●●
●
●●
●●
●●
●●● ●●●●
●●● ●●
●
●●●●●●
●
●
●
●●
●●●
●
●
●
●
●●●
●●●●●
●●●
●
●
●●
●●●●
●
●●●●
●●●●●
●●●
●●
●
●
●● ●● ●
●
●
●
●●●●●
●●●●●●● ●●●●●●
●
●
●
●●●
● ●●
● ●
●
●●
●●●
●
●●●
●
●
●
●●
●●●●
●
●
●●
●●
●●
● ●●
●●●● ●
●●
●
●●●●
●●
●●●● ●●
●
● ●●
●
● ●●●●
●●
●●●●
●
●●
●
●
●●●● ●
●
●
●●●
●
●● ●
●● ●● ● ●
●
●●
●
●
●
●
●●●
●
●● ●●
●●
●●
●
●●
●●●●
●●●●●
●
●●●●●
●●●● ●●
●
●●
●●●
●●
●
●●
●
●●●●
●
●
●
●● ●●●●●●
●
●
●●
●●
●
●●●●
●●●● ●●●●●●●●● ●
●
●●●●●
●
●
●●● ●●●●
●●●
●●●●
●●●
●●●
●●●●
● ●●
●
●● ●
●
●●●
●
●●
●●
●●●●●●
●●
●●
●●
●●
●
●●●
●●●●
●
●●●●
●●●
●● ●●
●●
●
●●
● ●●●
●●● ●
●●
●●●
●●●
●●
●●
●● ●●
●● ●
●
●●●
●●
●●●
●●●
●
●
●
● ●● ●
●●●
●
●●●
●●●
●●
●
●●●
●
●●
●
●
●
●
●●
●
●●
●● ●● ●●
●● ●
●
●
●●●● ●
●●●
●●●●● ●●
●●
●●●●
●●●●
●●
●
●●
●●●●
● ●●●
●
●●●●
●●●●●
●
● ●●
●
●●●●●
●●●● ●
●
●●●●●●
●
●●●●●
●●●
●●●
●●
●
●
●
●●●●
● ●●●●●●●●●●
●●●●●●
●
●●●●●●●●
●
●●
●●
● ●●
●
●
●●●●●●●●
●● ●●●
●●
●●●●
●●●●●●●●● ●●●●
●●
●
●● ●●
●●● ●●●
●
●● ●●●●●
●●●●
●●●
●
●●●
●●
●● ●●
●●
●
●● ●
● ●●●
●●
●
●●
●
● ●●●●●●
●
●●●●●●
●
●
●
●
●●
●●
●●
●
●● ●●
●
●●●●●●
●●
●
●
●●
●●● ●
● ●●
●
●
●●
●●●●
● ●
●●●●●● ●●●
●● ●●●
●●●●
●●
●
●
●●●●
●
●●
●●●
●●●
●●●
●
●
●●
●●
●●
●
●●
●
●●● ●●●
●●●
●●●
●●
●●●●●●●● ●●●●●●
●● ●●●●
●
●
●●●●
●●●
●●●●
●●
●●
●
●●●●
● ●●● ●● ●● ● ●●
●●●
●●
●●●●
●●
●
●●
●●
●●
●
●●●
●●●●●●●
●●●●●● ●●●
●
●●
●●●
●
●
●
●● ●●●●● ●●●● ●
●
●
●
●●
●
●●
●
●●●●●
●
●
●
● ●
●
●●●●●●
●●
● ●● ●●●●
●●●
●
●●●●●●●●
●●
●●
●●
●●
● ●●
●
●●
●●●●
●● ●●●●●
●● ●
●
●
●●
●●
●●
●
●●
●●
●●
●● ●●●●
●
●● ●●●
●
●●●●●
●●
●
●
●●●●●
●
●
●
●
●●●
●●●●
●
●●●
●
●
●●
●●●●
●
●● ●●
● ●●
●●●
●●
●●
●
●
●●● ●●
●
●
●
●●● ●
●
●●●● ●●●● ●●●●●
●
●
●
●●●
● ●●
●●
●
●●
●●●
●
●●●
●
●
●
●●
●●●●
●
●
●●
●●
●●
●●●
●●● ●●
●●
●
●●
●●
●●
● ●●●●●
●
●●●
●
●●●●●
●●
●●●●
●
●●
●
●
●●●
● ●
●
●
●● ●
●
●●●
●●●●● ●
●
●●●
●
●
●
●● ●
●
●●●●
●●
●●
●
●●
●● ●
●●
●●●●●
● ●●●●
● ●●●●●
●
●●
● ● ●
●●
●
●●
●
●●●●
●
●
●
●● ●●●●●●
●
●
●●
● ●
●
●●●
●
●●●
● ●●●
●●●● ●
●●●
●●●●●
●
●
●●●● ●
●●
●●●
●●
●●
●●●
● ●●
●●
●●
●●●
●
●●●
●
●●
●
●
●●
●●
●●●
●●●●
●
●●
●●
●●
●
● ●●
●● ●●
●
● ●●●
●●●
●● ●●
●●
●
●●
●● ●●
●●●●
●●
●●●
●●●●●
●●
●●●●
●●●
●
●●●
●●
●●●
●●
●
●
●
●
●●●●
●●
●
●
●●●
● ●●
●●
●
●●●
●
●●
●
●
●
●
●●
●
●●
●●●● ● ●
●●●
●
●
●●●
●●●●●
●●
●●●●●
●●
●●●●
●● ●●
●●
●
●●●●●●
● ●●●
●
●●●
●
●●
●●●
●
● ●●
●
●●●●●
●●● ●●
●
●●●●●
●
●
● ●●●●
●●●
●●●
● ●
●
●
●
●●●●●●●●●●●●●
● ●
● ●●●●●
●
●●●● ●●●●
●
●●
●●
●●●
●
●
●●●●●
●●
●●●●●
●
●●
●●●
●
● ●●●●●
●●● ●
●● ●●
●●
●●●●
●● ● ●●●
●
●●●●●
●●●
● ●●
●●
●
●
●●●
●●
●●●●●●
●
● ● ●
●●●●
●●
●
● ●
●
●●● ●●●
●
●
●●●
●●●
●
●
●
●
●●
●●
●●
●
●●●●
●
●●
●●●●
●●
●
●
●●
●●● ●●● ●
●
●
●●
●●● ●
●●
●●●●
● ●●●●
●●●●●
● ●●
●●●
●
●
●●● ●
●
●●
●●●● ●
●
●● ●
●
●
●●●●
●●
●
●●
●
●● ●●● ●●
●●
●●●
●●
●● ●●●●●●●●
●●●●●
●●●●●
●
●
●●
●●
●●●
●●●●
●●
●●
●
●● ●●
●●●
● ●●●●● ●●●
●●●●
●● ●
●● ●
●
●●
●●
●●
●
●●●
● ●●●●●●
●●●●●●●●●
●
●●
●●●
●
●
●
●● ●●●●●●
●●● ●
●
●
●
●●
●
●●
●
●●●●●●
●
●
●●
●
●●●●●●●●●●●●
●●●
●●●●
●●●●●●●●
●●●●
●●
●●●●●
●
●●●●●●●●●●●●
●
●●●
●
●
●●●●
●●
●
●●
●●
●●
●●●●●●●
●●●●●
●
●●●●●●●
●
●
●●
●●●
●
●
●
●
●●●
●●●●●
●●●●
●
●●●●●●
●
●●●●
● ●●●●
●●●
●●
●
●
●●●●●
●
●
●
●●●●●
●●●●●●●●●●●●●
●
●
●
●●●●●●
●●
●
●●
●●●
●
●●●
●
●
●
●●●
●●●
●
●
●●●●●
●
●●●●●●●●●●
●
●●
●●●●
●●●●●●
●
●●●
●
●●●●●
●●
●●●●
●
●●
●
●
●●● ●●
●
●
●●●
●
●●●●●●●●●
●
●●
●
●
●
●
●●●●
●●●●
●●
●●
●
●●
●●●●
●●●●●●
●●●●●
●●●●●●
●
●●●●●
●●
●
●●
●
●●●●
●
●
●
●●●●
●●●●
●
●
●●
●●
●
● ●●●
●●●
●●●●●●●●●
●●●
●●●●
●●
●
●●●●●●●
●●●
●●
●●●●●
●●●
●●●●
●●●●
●●●
●
●●●
●
●●●●●
●●●● ●●●
●●
●●
●●
●
●●●●●●●
●
● ●●●●●●
●●●●●●
●
●●●●●
●●●●●
●●
●●●●●●●●●●
●●●●●
● ●
●
●●●
●●
●●●●
●●
●
●
●
●●●●
●●●
●
●●●●●●
●●
●
●●●
●
●●
●
●
●
●
●●
●
●●
●●●● ●●
●●●
●
●
●●●●●●●●
●●●●●●●
●●●●●●
●●●
●
●●
●
●●●●●●
● ●●●
●
●●●
●
●●●●●
●
●●●
●
● ●●●●
●●●●●
●
●●●
●●●
●
●●●●●
●●●●●●
●●
●
●
●
●●●●●●●●●●●●●
● ●
●●●●●●
●
●●●●●●●●
●
●●●●
●●●
●
●
●●●●●●
●●●●●●●
●●
●●
●●
●●●●
●●●● ●●●●●
●●
●
●●●●
●●●●● ●●
●●●●●●●●●●●
●●●
●
●●●●●
●●●●●●●
●●●
●●●●
●●
●
●●
●
●●●●●●●
●
●●●●●●
●
●
●
●
●●
●●
●●
●
●●●●
●
●●●●●
●
●●●
●
●●
●●●●● ●●
●
●
●●
●●●●●●
●●●
●●●●●●
●●●●●
●●●●●●
●
●
●●●●
●
●●
●●●
●●●
●●●
●
●
●●●●
●●
●
●●●
●●●●●●●●●●●●
●●
●●●●●●●●●●●●●●●●●●
● ●
●
●
●●●
●
●●●
●●
●●
●●●●
●
●●●●
●●●●●●●●●●●●● ●●●
●●●●●●
●
●●
●●
●●
●
●●●
●●●●●●●
●●●●●●●●●
●
●●
●●●
●
●
●
●● ●●●●●●●●●
V4024
0 2 4
−6−4−2
−6 −4 −2
●
●
●
●
●●
●
●●
●
●●● ●
●●
●
●
● ●
●
●●●●●●
●●● ●● ●●●●
●●●●
●●●●●●
●●
●●
●●
●●
●●
●●●
●
●●
● ●●●
●● ●● ●●●
●● ●
●
●
●●●
●
●●
●
●●●●
●●
●● ● ●●●
●
●● ●●●
●
● ●●● ●
●●
●
●
●●
●●●
●
●
●
●
●● ●
●●●●
●
●●●●
●
●●
●●● ●
●
●●●●
●●●
●●●
●●
●●
●
●
●●●● ●
●
●
●
●●● ●●
●●●● ●●● ●●●●●
●
●
●
●
●●●● ●
●
●●
●
●●
●●●
●
●●●
●
●
●
●●
●●●
●
●
●
●●
●●
●●
● ●●
●●●●●
●●
●
●●
●●
●●
●●●●●●
●
● ●●
●
●●●● ●
●●
● ●● ●
●
●●
●
●
●● ●
●●
●
●
● ●●
●
●●●
● ● ●●● ●
●
●●
●
●
●
●
●●●●
●● ●●
●●
●●
●
●●
●●●
●●
●●● ●●
● ●●●●●●
● ● ●●
●
●●
● ●●
●●
●
● ●
●
● ●●●
●
●
●
●●● ●● ●●
●
●
●
●●
●●
●
●●●●
●● ●
● ●●●
●●●●●
●●●
●●●
●●
●
●
● ●●●●
●●
●●●
●●
●●
●●●
●●●
●●
●●
● ●●
●
●●●
●
●●
●
●
●●
●●
●●●
●● ●●
●
●●
●●
●●
●
●●●
●●●●
●
● ●●●●●●
●● ●●
●●
●
●●
●● ●●
● ●●●
●●
●●●●
●●●
●●●
●● ●●
●●●
●
●●●
●●
●●●
●●
●
●
●
●
●●●●
●●
●
●
●●●
● ●●
●●
●
● ●●
●
●●
●
●
●
●
●●
●
●●
●●● ● ●●
●●●
●
●
●●●
●●● ●●
●●●●
●●●
●●
●●●●
●●●●
●●
●
●●
●●●●
●●●●
●
●●●
●
●●
● ●●
●
●● ●
●
● ●●●●
●●● ●●
●
●●●●●
●
●
●●●●●
●●●
●●●
● ●
●
●
●
● ●●●
●●●●●●● ●●● ●
●●●
●● ●
●
● ●●●●●●●
●
●●
●●
●● ●
●
●
●● ●●●●
●●
●●●●●
●●
●●
●●
●●●●
● ●●● ●●
●● ●●
●●
●●● ●
●● ● ●● ●
●
●●●●●● ●●
●●●
●●
●
●
●●●
●●
●● ●●●
●●
●● ●
● ●● ●
●●
●
●●
●
● ●●●●●
●
●
●●●
●●●
●
●
●
●
●●
●●
●●
●
●●● ●
●
●●●●●
●
●●
●
●
●●
●●● ●●● ●
●
●
●●
●●● ●
●●
●●
●●
●●●●●
●● ●●●
●●●
●●●
●
●
●●●●
●
● ●
●●●●●
●
●● ●
●
●
●●●●
●●
●
●●
●
●● ●●●
●●
●●●
●●
●●
● ●●● ●●●●●●
●●●●●● ●●
● ●
●
●
●●●
●
● ●●
●●●●
●●●●
●
●●●●
● ●●
● ●● ●● ● ●●●
● ●●
●
●●●
●●●
●
●●
●●
●●
●
●●●
●●●●● ●●
●●●●● ●●●
●
●
●●
●●●
●
●
●
● ●● ● ●●●●
●●●
●
●
●
●
●●
●
●●
●●
●●
●●●
●
●
●●
●
●
●●●
●●● ●●
●
●
●●●
●
●
●●●
●
●●●●●
●●
●●
● ●●●
●●
●
●
●
●●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
●● ●
●●●
●●
●●
●●
●
●
●●
●●
●
●
●
●
●
●
●
● ●
●●
●
●●●
●
●
●
●
●
●
●●
●●●●●●●
●
●
●●
●
●
●
●●
●
●●
●●
●●
●●
●●
●●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●●
●
●
●
●●●
●●
●
●
●
●●●
● ●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●
●●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●●●
●●
●
●
●●
●●
●●
●●●
●
●●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
● ●
●
●●●
●●●
●
●
●●
●
●
●
●
●●●
●
●●
●●
●
●
●●
●●
●●
●●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●●
●●
●
●
●
●●
●
●●●●●
●●
●
●● ●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●
●●
●●
●●●
●●
●
●
●● ●●
●
●
●
●
●
●
●●●●●
●
●●
●
●
●●●
●●
●●●
●●● ●
●
●
●●
●●
●●
●
●
●●
●
●● ●
●
●●●●
●
●●●
●
●●●●
●
●
●● ●
●
●
●
●
●
●
●●●
● ●
●
●
●
●
●●
● ●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●● ● ●
●●
●●
●
●
●
●●●
●
●●
●
●●
●●
●
●
●
●
●●
●
●●●
●●
●●
●
●
●●●
●
●
●
●
●●
●
●●●
●
●●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●●
●●●
●
●
●
●●●●
●●●
●●●
●
●
●
●●●
●
●●
●
●●●●
●●
●●●
●●
●
●●●
●●●
●
●●●
●●●
●
●
●
●
●● ●
●●
●
●
●
●●
●●
●
●●● ●●●
●
●
●
●
●●
●●●●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●●●
●●
●
●
●●
●●
●
●
●●●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●●
●●
●●●
●●
●
●●●
●●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●●
●●
●●● ●●
●
●●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●●
●
●
●
●●
●
●
●
●●●●
●
●●●
●
●●●
●
●
●●●●●● ●●
●●●●
●●●
●
●●
●●
●●●
●
●
●●
●
●●●●
●●●●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●●
●
●
●●●
●●●
●●
●
●●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
●●●●
●
●
●
●
●●
●
●●
●●
●●
●●●
●
●
●●
●
●
●●●
●● ● ●●
●
●
●●●
●
●
●●
●
●
●●●●●
●●
●●
● ●●●
●●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
●● ●
●●●
● ●
●●
●●
●
●
●●
●●
●
●
●
●
●
●
●
● ●
● ●
●
●●●
●
●
●
●
●
●
●●
●●●● ●
●●●
●
●●
●
●
●
●●
●
● ●
●●
●●
●●
●●
●●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●●
●
●
●
●●
●●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●
●●
●●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●●● ●
●●
●
●
●●
●●
●●●
●●
●
●●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●● ●
●●●
●
●
●●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●●
●●
● ●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●●
●●
●
●
●
●●
●
●●●●●
●●
●
●●●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
●●●
●●
●
●
●●●●
●
●
●
●
●
●
●●●●●
●
● ●
●
●
●●●●
●
●●●
●●● ●
●
●
●●
●●
●●
●
●
●●
●
●●●
●
●●●●
●
●●●
●
●●●●
●
●
●●●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●●
●● ●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●● ●
●●
●●
●
●
●
●●●
●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●
●●
●●
●
●
●●●
●
●
●
●
●●
●
●●●
●
●●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●●
●●●
●
●
●
●●●●
●●●
● ●●
●
●
●
●●●
●
●●
●
●●●●
●●
●●●
●●
●
●●●
●●●
●
●● ●
●●●
●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●●●●●●
●
●
●
●
●●
● ●●●
●
●
●●
●
● ●●
●
●
●
●
●
●●●
●
●
●
●
●
● ●
●
●
●●●
●●
●
●
●●
●●
●
●
●●●
●
●
●●
●
● ●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●●
●
●
●
●●
●●●
●
●
●
●
●
●
●●●
●●
●●●
●●
●
●●●
●●
●
●
● ●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●●
●●●●●
●
●●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●●
●
●
●
●●
●
●
●
●●● ●
●
●● ●
●
● ●●
●
●
●●●●●●●●
●●●●●●
●●
●●
●●
●●●
●
●
●●
●
●●● ●
●●●●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●●
●
●
●●●
●●
●●●
●
●●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
●●●●
●
●
●
●
●●
●
●●
●●
●●
●●●
●
●
●●
●
●
●●●
●●●●●
●
●
●●●
●
●
●●●
●
●●●●●
●●
●●
●●●●
●●
●
●
●
● ●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
●●●
●●●
●●
●●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●●●
●
●
●
●
●
●
●●
●●●●●●●●
●
●●
●
●
●
●●
●
●●
●●
●●
●●
●●
●●●●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●●
●
●
●
●●●●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●
●●
●●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●●●●
●●
●
●
●●
●●
●●●●●
●
●●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●●●●●●
●
●
●●
●
●
●
●
●●●
●
●●
●●
●
●
●●
●●
●●
●●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●●
●●
●
●
●
●●●
●●● ●●
●●
●
●●●●
●
●
●●●
●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●
●●●●
●●●
●●
●
●
●●●●
●
●
●
●
●
●
●●●●●
●
●●
●
●
●●●●
●
●●●●●●●
●
●
●●
●●
●●
●
●
●●
●
●●●
●
●●●●●
●●●
●
●●●●
●
●
●● ●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●●●●●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●●●●
●●
●
●
●
●●●●
●●
●
●●
●●
●
●
●
●
●●
●
●●●
●●
●●●
●
●●●
●
●
●
●
●●
●
●●●
●
●●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●●
● ●●
●
●
●
●●●●
●●●●●●●
●
●
●●●
●
●●
●
●●●●
●●
●●●
●●
●
●●
●
●●●
●
●●●
●●●
●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●●●●
●●
●
●
●
●
●●
●●●●
●
●
●●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●●●
●●
●
●
●●
●●
●
●
●●●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●●●
●
●
●
●
●●
●●●
●
●
●
●
●
●
● ●●
●●
●●●
●●
●
●●●
●●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●●
●●●●●
●
●●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●●
●
●
●
●●
●
●
●
●●●●
●
●●●
●
●●●
●
●
●●●
●●●●●
●●●●●●●●
●●
●●
●●●
●
●
●●
●
●●●●
●●
● ●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●●
●
●
●●●
●●●●●
●
●●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●●●●
●
●
●
●
●●
●
●●
●●
●●
●●●
●
●
●●
●
●
●●●
●●●
●●
●
●
●●
●
●
●
●●
●
●
●●●●●
●●
●●
●●●●
●●
●
●
●
●●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●●
●●●
●●●
●●
●●
●●
●
●
●●●●
●
●
●
●
●
●
●
● ●
● ●
●
●●●
●
●
●
●
●
●
●●
●●●●●●●
●
●
●●
●
●
●
●●
●
●●
●●
●●
●●
●●
●●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●●
●
●
●
●●●●●
●
●
●
●●●
● ●
●
●
●
●
●
●
●
●
●●
●
●●
● ●
●●
●
●
●
●
●●
●
●
●●
●●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●●●
●●
●
●
● ●
●●
●●●●
●
●
●●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●●●
●●●
●
●
●●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●●
●●
●●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●
●
●●
●
●
●●
● ●
●
●
●
●●
●●
●
●
●
●●
●
●●● ●●
●●
●
●●●●
●
●
●●●
●●
●
●
●●
●
●●
●
●●●
●
●
●
●
●
●●● ●
●●●
●●
●
●
●●●●
●
●
●
●
●
●
●●●
●●
●
●●
●
●
●●●
●●
●●●
●●● ●
●
●
●●●
●
●●
●
●
●●
●
●●●
●
●●●●●
●●●
●
●●●●
●
●
●●●
●
●
●
●
●
●
●●●
● ●
●
●
●
●
●●●●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●●●●
●●
●
●
●
●●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●
●●
●●●
●
●●●
●
●
●
●
●●
●
●●●
●
●●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●●
●●●
●
●
●
●●●●
●●●● ●●
●
●
●
●●●
●
●●
●
●●●●
●●
●●●
● ●
●
●●●
●●●
●
●●●
●●●
●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●●●●
●●
●
●
●
●
●●
●●●●
●
●
●●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
● ●
●
●
●●●
●●
●
●
●●
●●
●
●
●●●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●
●
●
●
●●
●●●
●
●
●
●
●
●
●●●
●●
●●●
●●
●
●●●
● ●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●●
●●●●●
●
●●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●●
●
●
●
●●
●
●
●
●●●●
●
●●●
●
● ●●
●
●
●●●●●●●
●
●●●●●●●●
●●
●●
●● ●
●
●
●●
●
●●●●
●●● ●
●
●
●● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●●
●
●
● ●●
●●●●●
●
●●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
●●●●
V5123 1 2 3
−2−1
0
−2 −1 0
96
Stochastic Optimization� Each iteration of K-means requires a pass through whole dataset. In
extremely large datasets, this can be computationally prohibitive.� Stochastic optimization: update cluster means after assigning each data
point to the closest cluster.� Repeat for t = 1, 2, . . . until satisfactory convergence:
1. Pick data item xi either randomly or in order.2. Assign xi to the cluster with the nearest centre,
ci := argmink
�xi − µk�22
3. Update cluster centre:µk := µk + αt(xi − µk)
where αt > 0 are step sizes.� Algorithm stochastically minimizes the objective function. Convergence
requires slowly decreasing step sizes:
∞�
t=1
αt = ∞
∞�
t=1
α2t < ∞
97
Vector Quantization
� A related algorithm developed in the signal processing literature for lossydata compression.
� If K � n, we can store the codebook of codewords µ1, . . . , µK , and eachvector xi is encoded using ci, which only requires �log K� bits.
� As with K-means, K must be specified. Increasing K improves the qualityof the compressed image but worsens the data compression rate, sothere is a clear tradeoff.
� Some audio and video codecs use this method.� Stochastic optimization algorithm for K-means was originally developed
for VQ.
98
VQ Image Compression
3 × 3 block VQ: View each block of 3 × 3 pixels as single observation
99
VQ Image Compression
Original image (24 bits/pixel, uncompressed size 1,402 kB)
100
VQ Image Compression
Codebook length 1024 (1.11 bits/pixel, total size 88kB)
101
VQ Image Compression
Codebook length 128 (0.78 bits/pixel, total size 50kB)
102
VQ Image Compression
Codebook length 16 (0.44 bits/pixel, total size 27kB)
103
K-means Additional Comments� Sensitivity to distance measure. Euclidean distance can be greatly
affected by measurement unit and by strong correlations. Can useMahalanobis distance,
�x − y�M =�(x − y)�M−1(x − y)
where M is positive semi-definite matrix, e.g. sample covariance.� Other partition based methods. There are many other partition based
methods that employ related ideas. For example K-medoids differs fromK-means in requiring cluster centres µi to be an observation xi
2,K-medians (use median in each dimension) and K-modes (use mode).
� Determination of K. The K-means objective will always improve withlarger number of clusters K. Determination of K requires an additionalregularization criterion. E.g., in DP-means3, use
W =K�
k=1
�
i∈Ck
�xi − µk�22 + λK
2See also Affinity propagation.3DP-means paper.
104
Probabilistic Methods
� Algorithmic approach:
Analysis/Interpretation
Data
Algorithm
� Probabilistic modelling approach:
DataUnobservedprocess
GenerativeModel
AnalysisInterpretation
105
Mixture Models� Mixture models suppose that our dataset was created by sampling iid
from K distinct populations (called mixture components).� Typical samples in population k can be modelled using a distribution
F(φk) with density f (x|φk). For a concrete example, consider a Gaussianwith unknown mean φk and known symmetric covariance σ2I,
f (x|φk) = |2πσ2|−
p2 exp
�−
12σ2 �x − φk�
22
�.
� Generative process: for i = 1, 2, . . . , n:� First determine which population item i came from (independently):
Zi ∼ Discrete(π1, . . . ,πK) i.e. P(Zi = k) = πk
where mixing proportions are πk ≥ 0 for each k and�K
k=1 πk = 1.� If Zi = k, then Xi = (Xi1, . . . ,Xip)
� is sampled (independently) fromcorresponding population distribution:
Xi|Zi = k ∼ F(φk)
� We observe that Xi = xi for each i, and would like to learn about theunknown parameters of the process.
106
Mixture Models� Unknowns to learn given data are
� Parameters: π1, . . . ,πK , φ1, . . . ,φK , as well as� Latent variables: z1, . . . , zK .
� The joint probability over all cluster indicator variables {Zi} are:
pZ((zi)ni=1) =
n�
i=1
πzi =n�
i=1
K�
k=1
π (zi=k)k
� The joint density at observations Xi = xi given Zi = zi are:
pX((xi)ni=1|(Zi = zi)
ni=1) =
n�
i=1
K�
k=1
f (xi|φk)(zi=k)
� So the joint probability/density4 is:
pX,Z((xi, zi)ni=1) =
n�
i=1
K�
k=1
(πkf (xi|φk))(zi=k)
4In this course we will treat probabilities and densities equivalently for notational simplicity. Ingeneral, the quantity is a density with respect to the product base measure, where the basemeasure is the counting measure for discrete variables and Lebesgue for continuous variables.
107
Mixture Models - Posterior Distribution� Suppose we know the parameters (πk,φk)K
k=1.� Zi is a random variable, so the posterior distribution given data set X tells
us what we know about it:
Qik := p(Zi = k|xi) =p(Zi = k, xi)
p(xi)=
πkf (xi|φk)�Kj=1 πjf (xi|φj)
where the marginal probability is:
p(xi) =K�
j=1
πjf (xi|φj)
� The posterior probability Qik of Zi = k is called the responsibility ofmixture component k for data point xi.
� The posterior distribution softly partitions the dataset among the kcomponents.
108
Mixture Models - Maximum Likehood
� How can we learn about the parameters θ = (πk,φk)Kk=1 from data?
� Standard statistical methodology asks for the maximum likelihoodestimator (MLE).
� The log likelihood is the log marginal probability of the data:
�((πk,φk)Kk=1) := log p((xi)
ni=1|(πk,φk)
Kk=1) =
n�
i=1
logK�
j=1
πjf (xi|φj)
∇φk�((πk,φk)Kk=1) =
n�
i=1
πkf (xi|φk)�Kj=1 πjf (xi|φj)
∇φk log f (xi|φk)
=n�
i=1
Qik∇φk log f (xi|φk)
� A difficult equation to solve, as Qik depends implicitly on φk...
109
Mixture Models - Maximum Likehood
n�
i=1
Qik∇φk log f (xi|φk) = 0
� What if we ignore the dependence of Qik on the parameters?� Taking the mixture of Gaussian with covariance σ2I as example,
n�
i=1
Qik∇φk
�−
p2|2πσ2
|−1
2σ2 �xi − φk�22
�
=1σ2
n�
i=1
Qik(xi − φk) =1σ2
���ni=1 Qikxi
�− φk
��ni=1 Qik
��= 0
φMLE?k =
�ni=1 Qikxi�ni=1 Qik
110
Mixture Models - Maximum Likehood
� The estimate is a weighted average of data points, where the estimatedmean of cluster k uses its responsibilities to data points as weights.
φMLE?k =
�ni=1 Qikxi�ni=1 Qik
� Makes sense: Suppose we knew that data point xi came from populationzi. Then Qizi = 1 and Qik = 0 for k �= zi and:
πMLEk =
�i:zi=k xi�i:zi=k 1
� Our best guess of the originating population is given by Qik.
111
Mixture Models - Maximum Likehood
� For the mixing proportions, we can similarly derive an estimator.� Include a Lagrange multiplier λ to enforce constraint
�k πk = 1.
∇log πk
��((πk,φk)
Kk=1)− λ(
�Kk=1 πk − 1)
�
=n�
i=1
πkf (xi|φk)�Kj=1 πjf (xi|φj)
− λπk
=n�
i=1
Qik − λπkπMLE?k =
�ni=1 Qik
n
� Again makes sense: the estimate is simply (our best guess of) theproportion of data points coming from population k.
112
Mixture Models - The EM Algorithm
� Putting all the derivations together, we get an iterative algorithm forlearning about the unknowns in the mixture model.
� Start with some initial parameters (π(0)k ,φ(0)
l )Kk=1.
� Iterate for t = 1, 2, . . .:� Expectation Step:
Q(t)ik :=
π(t−1)k f (xi|φ(t−1)
k )�K
j=1 π(t−1)j f (xi|φ(t−1)
j )
� Maximization Step:
π(t)k =
�ni=1 Q(t)
ik
nφ(t)
k =
�ni=1 Q(t)
ik xi�n
i=1 Q(t)ik
� Will the algorithm converge?� What does it converge to?
113
Likelihood Surface for a Simple Example01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
320 compBody.tex
!30 !20 !10 0 10 20 300
5
10
15
20
25
30
35
(a)
mu1
mu
2
!15.5 !10.5 !5.5 !0.5 4.5 9.5 14.5 19.5
!15.5
!10.5
!5.5
!0.5
4.5
9.5
14.5
19.5
(b)
Figure 11.6: Left: N = 200 data points sampled from a mixture of 2 Gaussians in 1d, with πk = 0.5, σk = 5, µ1 = −10 and µ2 = 10.Right: Likelihood surface p(D|µ1, µ2), with all other parameters set to their true values. We see the two symmetric modes, reflecting theunidentifiability of the parameters. Produced by mixGaussLikSurfaceDemo.
11.4.2.5 Unidentifiability
Note that mixture models are not identifiable, which means there are many settings of the parameters which have the samelikelihood. Specifically, in a mixture model with K components, there are K! equivalent parameter settings, which differ merelyby permuting the labels of the hidden states. See Figure 11.6 for an illustration. The existence of equivalent global modes doesnot matter when computing a single point estimate, such as the ML or MAP estimate, but it does complicate Bayesian inference,as we will in Section 12.5.6.3. Unfortunately, even finding just one of these global modes is computationally difficult. The EMalgorithm is only guaranteed to find a local mode. A variety of methods can be used to increase the chance of finding a goodlocal optimum. The simplest, and most widely used, is to perform multiple random restarts.
11.4.2.6 K-means algorithm
There is a variant of the EM algorithm for GMMs known as the K-means algorithm, which we now discuss.Consider a GMM in which we make the following assumptions: Σk = σ2ID is fixed, and πk = 1/K is fixed, so only the
cluster centers, µk ∈ RD, have to be estimated. Now consider an approximation to EM in which we make the approximation
p(zi = k|xi,θ) ≈ I(k = z∗i ) (11.61)
where zi∗ = argmaxk p(zi = k|xi,θ). This is sometimes called hard EM, since we are making a hard assignment of pointsto clusters. Since we assumed an equal spherical covariance matrix for each cluster, the most probable cluster for xi can becomputed by finding the nearest prototype:
z∗i = argmink
||xi − µk||2 (11.62)
Hence in each E step, we must find the Euclidean distance between N data points and K cluster centers, which takes O(NKD)time. However, this can be sped up using various techniques, such as applying the triangle inequality to avoid some redundantcomputations [Elk03]. Given the hard cluster assignments, the M step updates each cluster center by computing the mean of allpoints assigned to it:
µk =1
Nk
�
i:zi=k
xi (11.63)
The resulting method is equivalent to the K-means algorithm. See Algorithm 5 for the pseudo-code.
Algorithm 3: K-means algorithminitialize mk, k ← 1 to K1
repeat2
Assign each data point to its closest cluster center: zi = argmink ||xi − µk||23
Update each cluster center by computing the mean of all points assigned to it: µk = 1Nk
�i:zi=k xi4
until converged5
Since K-means is not a proper EM algorithm, it is not maximizing likelihood. Instead, it can be interpreted as a greedyalgorithm for approximately minimizing the reconstruction error created by using vector quantization, as discussed in Sec-tion 8.5.3.3. (See also Section 20.2.1.)
c� Kevin P. Murphy. Draft — not for circulation.
(left) n = 200 data points from a mixture of two 1D Gaussians withπ1 = π2 = 0.5, σ = 5 and µ1 = 10, µ2 = −10.(right) Log likelihood surface � (µ1, µ2), all the other parameters beingassumed known.
114
Example: Mixture of 3 GaussiansAn example with 3 clusters.
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●●
●
●
●●
●●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
−5 0 5 10
−50
5
X1
X2
115
Example: Mixture of 3 GaussiansAfter 1st E and M step.
!5 0 5 10
!5
05
data[,1]
data[,2]
Iteration 1
116
Example: Mixture of 3 GaussiansAfter 2nd E and M step.
!5 0 5 10
!5
05
data[,1]
data[,2]
Iteration 2
117
Example: Mixture of 3 GaussiansAfter 3rd E and M step.
!5 0 5 10
!5
05
data[,1]
data[,2]
Iteration 3
118
Example: Mixture of 3 GaussiansAfter 4th E and M step.
!5 0 5 10
!5
05
data[,1]
data[,2]
Iteration 4
119
Example: Mixture of 3 GaussiansAfter 5th E and M step.
!5 0 5 10
!5
05
data[,1]
data[,2]
Iteration 5
120
The EM Algorithm� In a maximum likelihood framework, the objective function is the log
likelihood,
�(θ) =n�
i=1
logK�
j=1
πjf (xi|φj)
Direct maximization is not feasible.� Consider another objective function F(θ, q) such that:
F(θ, q) ≤ �(θ) for all θ, q,max
qF(θ, q) = �(θ)
F(θ, q) is a lower bound on the log likelihood.� We can construct an alternating maximization algorithm as follows:
For t = 1, 2 . . . until convergence:
q(t) := argmaxq
F(θ(t−1), q)
θ(t) := argmaxθ
F(θ, q(t))
121
EM Algorithm
� The lower bound we use is called the variational free energy.� q is a probability mass function for some distribution over (Zi) and
F(θ, q) =Eq[log p((xi, zi)ni=1)− log q((zi)
ni=1)]
=Eq
��n�
i=1
K�
k=1
(zi = k) (logπk + log f (xi|φk))
�− log q(z)
�
=�
zq(z)
��n�
i=1
K�
k=1
(zi = k) (logπk + log f (xi|φk))
�− log q(z)
�
Using z := (zi)ni=1 to shorten notation.
122
EM Algorithm - Solving for q� Introducing Lagrange multiplier to enforce
�z q(z) = 1, and setting
derivatives to 0,
∇q(z)F(θ, q) =n�
i=1
K�
k=1
(zi = k) (logπk + log f (xi|φk))− log q(z)− 1 − λ
=n�
i=1
(logπzi + log f (xi|φzi))− log q(z)− 1 − λ = 0
q∗(z) =�n
i=1 πzi f (xi|φzi)�z��n
i=1 πz�i f (xi|φz�i )=
n�
i=1
πzi f (xi|φzi)�k πkf (xi|φk)
=n�
i=1
p(zi|xi, θ)
� Optimal q∗ is simply the posterior distribution.� Plugging in optimal q∗ into the variational free energy,
F(θ, q∗) =n�
i=1
logK�
k=1
πkf (xi|φk) = �(θ)
123
EM Algorithm - Solving for θ� Setting derivative with respect to φk to 0,
∇φkF(θ, q) =�
zq(z)
n�
i=1
(zi = k)∇φk log f (xi|φk)
=n�
i=1
q(zi = k)∇φk log f (xi|φk) = 0
� This equation can be solved quite easily. E.g., for mixture of Gaussians,
φ∗
k =
�ni=1 q(zi = k)xi�ni=1 q(zi = k)
� If it cannot be solved exactly, we can use gradient ascent algorithm:
φ∗
k = φk + αn�
i=1
q(zi = k)∇φk log f (xi|φk)
� This leads to generalized EM algorithm. Further extension usingstochastic optimization method leads to stochastic EM algorithm.
� Similar derivation for optimal πk as before.124
EM Algorithm� Start with some initial parameters (π(0)
k ,φ(0)l )K
k=1.� Iterate for t = 1, 2, . . .:
� Expectation Step:
q(t)(zi = k) :=π(t−1)
k f (xi|φ(t−1)k )
�Kj=1 π
(t−1)j f (xi|φ(t−1)
j )= Ep(zi|xi,θ(t−1))[ (zi = k)]
� Maximization Step:
π(t)k =
�ni=1 q(t)(zi = k)
nφ(t)
k =
�ni=1 q(t)(zi = k)xi�n
i=1 q(t)(zi = k)
� Each step increases the log likelihood:
�(θ(t−1)) = F(θ(t−1), q(t)) ≤ F(θ(t), q(t)) ≤ F(θ(t), q(t+1)) = �(θ(t)).
� Additional assumption, that ∇2θF(θ(t), q(t)) are negative definite with
eigenvalues < −� < 0, implies that θ(t) → θ∗ where θ∗ is a local MLE.
125
Notes on Probabilistic Approach and EM Algorithm
Some good things:� Guaranteed convergence to locally optimal parameters.� Formal reasoning of uncertainties, using both Bayes Theorem and
maximum likelihood theory.� Rich language of probability theory to express a wide range of generative
models, and straightforward derivation of algorithms for ML estimation.
Some bad things:� Can get stuck in local minima so multiple starts are recommended.� Slower and more expensive than K-means.� Choice of K still problematic, but rich array of methods for model
selection comes to rescue.
126
Flexible Gaussian Mixture Models
� We can allow each cluster to have its own mean and covariance structureallows greater flexibility in the model.
Different covariances
Identical covariances
Different, but diagonal covariances
Identical and spherical covariances
127
Probabilistic PCA� A probabilistic model related to PCA has the following generative model:
for i = 1, 2, . . . , n:� Let k < n, p be given.� Let Yi be a k-dimensional normally distributed random variable with 0 mean
and identity covariance:Yi ∼ N (0, Ik)
� We model the distribution of the ith data point given Yi as a p-dimensionalnormal:
Xi ∼ N (µ+ LYi,σ2I)
where the parameters are a vector µ ∈ Rp, a matrix L ∈ R
p×k and σ2 > 0.� EM algorithm can be used for ML estimation, but PCA can more directly
give a MLE (note this is not unique).� Let λ1 ≥ · · · ≥ λp be the eigenvalues of the sample covariance and let
V ∈ Rp×k have columns given by the eigenvectors of the top k
eigenvalues. Let R ∈ Rk×k be orthogonal. Then a MLE is:
µMLE = x̄ (σ2)MLE = 1p−k
�pj=k+1 λj
LMLE = V diag((λ1 − (σ2)MLE)12 , . . . , (λk − (σ2)MLE)
12 )R
128
Mixture of Probabilistic PCAs
� We have learnt two types of unsupervised learning techniques:� Dimensionality reduction, e.g. PCA, MDS, Isomap.� Clustering, e.g. K-means, linkage and mixture models.
� Probabilistic models allow us to construct more complex models fromsimpler pieces.
� Mixture of probabilistic PCAs allows both clustering and dimensionalityreduction at the same time.
Zi ∼ Discrete(π1, . . . ,πK)
Yi ∼ N (0, Id)
Xi|Zi = k, Yi = yi ∼ N (µk + Lyi,σ2Ip)
� Allows flexible modelling of covariance structure without using too manyparameters.
Ghahramani and Hinton 1996 129
Mixture of Probabilistic PCAs� PCA can reconstruct x given low dimensional embedding z, but is linear.� Isomap is non-linear, but cannot reconstruct x given any z.
tion to geodesic distance. For faraway points,geodesic distance can be approximated byadding up a sequence of “short hops” be-tween neighboring points. These approxima-tions are computed efficiently by findingshortest paths in a graph with edges connect-ing neighboring data points.
The complete isometric feature mapping,or Isomap, algorithm has three steps, whichare detailed in Table 1. The first step deter-mines which points are neighbors on themanifold M, based on the distances dX (i, j)between pairs of points i, j in the input space
X. Two simple methods are to connect eachpoint to all points within some fixed radius !,or to all of its K nearest neighbors (15). Theseneighborhood relations are represented as aweighted graph G over the data points, withedges of weight dX(i, j) between neighboringpoints (Fig. 3B).
In its second step, Isomap estimates thegeodesic distances dM (i, j) between all pairsof points on the manifold M by computingtheir shortest path distances dG(i, j) in thegraph G. One simple algorithm (16 ) for find-ing shortest paths is given in Table 1.
The final step applies classical MDS tothe matrix of graph distances DG " {dG(i, j)},constructing an embedding of the data in ad-dimensional Euclidean space Y that bestpreserves the manifold’s estimated intrinsicgeometry (Fig. 3C). The coordinate vectors yi
for points in Y are chosen to minimize thecost function
E ! !#$DG% " #$DY%!L2 (1)
where DY denotes the matrix of Euclideandistances {dY(i, j) " !yi & yj!} and !A!L2
the L2 matrix norm '(i, j Ai j2 . The # operator
Fig. 1. (A) A canonical dimensionality reductionproblem from visual perception. The input consistsof a sequence of 4096-dimensional vectors, rep-resenting the brightness values of 64 pixel by 64pixel images of a face rendered with differentposes and lighting directions. Applied to N " 698raw images, Isomap (K" 6) learns a three-dimen-sional embedding of the data’s intrinsic geometricstructure. A two-dimensional projection is shown,with a sample of the original input images (redcircles) superimposed on all the data points (blue)and horizontal sliders (under the images) repre-senting the third dimension. Each coordinate axisof the embedding correlates highly with one de-gree of freedom underlying the original data: left-right pose (x axis, R " 0.99), up-down pose ( yaxis, R " 0.90), and lighting direction (slider posi-tion, R " 0.92). The input-space distances dX(i, j )given to Isomap were Euclidean distances be-tween the 4096-dimensional image vectors. (B)Isomap applied to N " 1000 handwritten “2”sfrom the MNIST database (40). The two mostsignificant dimensions in the Isomap embedding,shown here, articulate the major features of the“2”: bottom loop (x axis) and top arch ( y axis).Input-space distances dX(i, j ) were measured bytangent distance, a metric designed to capture theinvariances relevant in handwriting recognition(41). Here we used !-Isomap (with ! " 4.2) be-cause we did not expect a constant dimensionalityto hold over the whole data set; consistent withthis, Isomap finds several tendrils projecting fromthe higher dimensional mass of data and repre-senting successive exaggerations of an extrastroke or ornament in the digit.
R E P O R T S
22 DECEMBER 2000 VOL 290 SCIENCE www.sciencemag.org2320
on D
ecem
ber
5, 2008
ww
w.s
cie
ncem
ag.o
rgD
ow
nlo
aded fro
m
� We can learn a probabilistic mapping between the k-dimensional Isomapembedding space and the p-dimensional data space.
� Demo: [Using LLE instead of Isomap, and Mixture of factor analysersinstead of Mixture of PPCAs.]
Teh and Roweis 2002 130
Further Readings—Unsupervised Learning
� Hastie et al, Chapter 14.� James et al, Chapter 10.� Venables and Ripley, Chapter 11.� Tukey, John W. (1980). We need both exploratory and confirmatory. The
American Statistician 34 (1): 23-25.
131
