Apache Kylin W OLAP - open.qiniudn.comopen.qiniudn.com/ecug-2016/apache-kylin.pdf• The company was...

Apache Kylin

W�OLAPW¬å: | Apache Kylin PMC Chair

[email protected]

2

!�…

� O-I C LA? C C pfyW

E LA?C pfyW 0 ?F

B B F 0

CHDF D E 2D H 1 E K

? FDGD H 0

g La e w V W

3

!�Apache Kylin

� S Tt E

� R C D DFA: M w

cxM P DD A CGDF AD

x

� d bo

ApacheKylin� � IY�¨2�

<²=Y��-£}£¯�0�W�—TedDunning,ApacheI,:+lÅ

4

Apache Kylin;B

/2

y

1

h Eis

C D DFA: DGG? F:

P SQ B

QE

5432612A DO

/2 0

C D DFA: DGG? F:

P SQ B

/2

rQ 0l

3

5

Apache Kylin�r

6

!�Kyligence

• Kyligence’svisionistounleashbigdataproductivityforeveryone's

analyticsneeds.

• ThecompanywasfoundedbytheteamwhocreatedApache

Kylin™,atopopensourceOLAPenginebuiltforinteractive

analyticsatpetabyte-scaledataonHadoop.Kyligenceisthe

primarycontributortotheopensourceKylinprojectglobally.

• Kyligenceprovidesaleadingintelligentdataplatformtosimplify

bigdataanalyticsfromon-premisestocloud.

¬©�Hadoop��5�j3'/Ú�

Hadoop��ç

�Îk¿a

e�ß�ê

�FSv

�m�~�àé

�~A�

¤»ädQ

rj��ka

rj$��Û

l!w�p�ê

SQ N SQ K

k

SQL

kh

/

e Q

nL CDH

DC :DDE

Q R e

Kylin³mX�rGQ�z�á®��

e Q

nSQ

SQ NSQ K & H

I& HI

& HI

& HI

•e

• M w

l

e

• /

u m

9

Apache Kylin��~

MapReduce

Kylin

BITools,WebApp…

ANSISQL

V1.5�^·�Æ¾

All rights reserved ©Kyligence Inc.http://kyligence.io

11

V1.5�StreamingCubing

MapReduce/Spark

Kylin

BITools,WebApp…

ANSISQL

StandaloneJVM

processtoatime

window

PuttoHBase

cron

V1.5:�rj(� –��áÕ�\s


P��¦¤DcpQSegment


�7Dc

�7Dc

pÁ=½â


• ¡% �u*y�àé • ~X��7�¼

• ¥��(�s��R�z

• ~X�6ä�¯{

• ÞÐaA;ä

• s�¤»p�ê

15

v1.6Streaming³lf�~��K�W¬

MapReduce/Spark

Kylin

BITools,WebApp…

ANSISQL

Adapter

Adapter

Tables

Topics

CubeSegmentioffset)(


• Segmentioffset)(,��yª5• [�Hiverj�²(-�á#segment�offset²�Ì�E4®*Z• [�Kafkarj�²4partition�offset�5²��segment�offset

• Segment�á&Ê��á"ª5• ��offset²�� rj��`9�Î,±k

1:00– 1:05 1:04– 1:11 1:08-1:40

Seg[0-100) Seg[100-400) Seg[400-2000)

Whenqueryfortime1:10²KylinwillscanallSegmentscontainthistime

K 2�¼


• C©(�°Û��eh|��»t

• �'.|*'�|rj²��>2Ò��e

• 2¬��$�mO�A�o�~�eèª

"JqØ


• �7\sfTG¸�®offset

• ogSD[JSON�b

• og�L��á�g

• �ÊPsegmentV�~X/5V

• §ÔRestAPI¢>

KylinStreaming�%|�


• Kafka�{0.10d��

• Kafka�b�JSON�[²_æ�� timestampG�

• �?��iÄ�}xP Topic�join²��og

topic�Hive �join±

KylinStreaming�%|�


��Kylin(�Twitter�b


• Demo«N� AWS 53vm�Hadoop�²Kafka 3broker

• ¢Í�z�Twitter samplerj�²��TwitterSDK1AÁ?�+(�J

�Kafkatopic;

• Topic“TwitterSample”H%<T£n�Topic“TWITTER_TAG_STREAM2”H

%�<T£n�u-®HASH_TAG�b´

Twitter�b


./bin/kafka-console-consumer.sh --zookeeper10.0.0.205:2181--bootstrap-server\

10.0.0.207:6667,10.0.0.208:6667,10.0.0.209:6667--topicTwitterSample

TwitterTag�b


./bin/kafka-console-consumer.sh --zookeeper10.0.0.205:2181--bootstrap-server\

10.0.0.207:6667,10.0.0.208:6667,10.0.0.209:6667--topicTWITTER_TAG_STREAM2

OJSON�bvNc ¸�


Ó(Kafka��b


Ó(£nparser�b


¸��]Up


×��i�¥�Kafka�TableR�c °�JYF1eModel9Cube

1e�L


�w§�?@{�yUZ

• æyV� �á/90�H�¡gK��á�b�Î��

E�(Ý¶2Ù��yVMINUTE_START��/90

1eCube


�w§Cube�yUZ´

hL��inmem��2�k¨cube~X

[�á0yVhierarchy`¶�[Ñ M�»dyVjoint²&Q·D�

~XCube


WebGUI�.Ç>�q

RESTful API¢>³

curl-XPUT--userADMIN:KYLIN-H"Content-Type:application/json;charset=utf-8"

-d'{"sourceOffsetStart":0,"sourceOffsetEnd":9223372036854775807,

"buildType":"BUILD"}'

http://localhost:7070/kylin/api/cubes/twitter_tag_cube5/build2

“sourceOffsetStart”:0 � ��~X�¸�¦fT�

“sourceOffsetEnd”:9223372036854775807� ~X*kafka]�xt��b��

Kylin��Kafka APIÁ?C partition�xtoffset�b²�c�6

~XCube- continued


• 87MR job²V��partitionÁ?�£n

• 7º�e^M�HDFS��bØÂ�Hiveµ

6

Twitter�b�(�K�


• ¸DTwitter�b�#M²ËÉ/�K�

• K�1:i�á�ÏÈ�/�£n¡Ü@8

• K�2:i�á�Xr³�/�¡Ü@8

• K�3:i�á�ÏÈ�/��§ß®Tag

• K�4:[¹VTag²�²)ã�á®èª@8

�j®�ËÉ3Cube


• Cube1:ogK�1&2²»d��á�ÏÈ�Xr³´�

W«�count(*)

• Cube2:ogK�3²»d��á�ÏÈ�dÜ�TopN

• Cube3:ogK�4²»d��á�Tag²W«�count(*)

Dashboard


StreamingÖUBQµ


z}T�


• �8MapReduce\n²k/~X¨W

• _Íxº�eh|�SparkStreaming/Flink

• ÃKW�À¦

ThanksSee you on our next meeting

Apache Kylin W OLAP - open.qiniudn.comopen.qiniudn.com/ecug-2016/apache-kylin.pdf• The company was...

Documents

Transcript of Apache Kylin W OLAP - open.qiniudn.comopen.qiniudn.com/ecug-2016/apache-kylin.pdf• The company was...