genn.ai v2
-
Upload
takeshi-nakano -
Category
Technology
-
view
738 -
download
0
description
Transcript of genn.ai v2
genn.ai (源内)Realtime processing for everyone
1
Takeshi Nakano@tf0054
http://genn.ai/
Who am I?Takeshi NAKANO
Senior Researcher / Architect.Co-authered Getting started with Solr in Japanese.Co-authered Hadoop Hacks in Japanese.
2
Writing “Getting started with Kafka” in Japanese.
http://genn.ai/
Agenda
What is genn.ai? Realtime processingUsecases
3
Let’s use genn.aiDownload & get your keyDefine a schema and a filterMake it real!
A usecase of genn.ai
4
CustomerA
CustomerB
CustomerC
Realtime Analysis Platform
じゃらんWeb Sites
Client(Hotels)
MarketerAnalyst
Visual editor for genn.ai
http://genn.ai/
GungnirThrift Server
GungnirRest Server
GungnirCosole
Stormnimbus
Stormsupervisor
GungnirClient
on finagle
Kafkabroker
Stormsupervisor
StormsupervisorKafka
broker
Kafkabroker
GungnirThrift Server
on finagle
GungnirRest Server
GungnirRest Serveron finagle
MongoDBMetaStore
TupleTopic
Query
genn.ai UIQuery design
view
Query
User accountTuple schema
Topology
JSON Tuple Tuple Tuple
Kafka
MongoDB
HDFS
User accountTuple schema
6
Structure of genn.ai
http://dev.genn.ai/
Preparing for starting.
http://dev.genn.ai/
Preparing for starting.
> cd gungnir-client> bin/gungnir -u tf0054 -p 2e2232073109
Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-82013/10/14 22:34:58 com.twitter.finagle.Init$ apply情報: Finagle version 6.5.1 (rev=59..9cde41) built at 20130626-111057
Gungnir server connected... ec2-xxx.amazonaws.com/54.238.99.212:9290Welcome tf0054gungnir>
Define your schema!CREATE TUPLE userAction ( userId STRING, hotelId STRING);
CREATE TUPLE commitAction ( userId STRING, hotelId STRING, checkin TIMESTAMP('yyyy-MM-dd'), nights INT);
STRING Java type string
TINYINT Java type byteSMALLINT Java type shortINT Java type intBIGINT Java type longFLOAT Java type floatDOUBLE Java type double
TIMESTAMP long value
LISTMAPSTRUCT
FROM userAction AS ua, commitAction AS ca USING kafka_spout()EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user');
From EMIT
Define your filter! (1/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo')JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel')EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user');
From JOIN(user)
JOIN(hotel)
EMIT
Define your filter! (2/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo')JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel')INTO stream_0;FROM stream_0EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user');
From JOIN(user)
JOIN(hotel) stream_0
EMIT(user)
Define your filter! (3/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image USING mongo_fetch('getusers', 'userInfo')JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo USING mongo_fetch('hotelscom', 'hotel')INTO stream_0;FROM stream_0EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user');FROM stream_0BEGIN GROUP BY hotelId EACH count() AS count, * EMIT hotelId, hotelname, photo, count USING kafka_emit('${TOPOLOGY_ID}_hotel')END GROUP;
From JOIN JOIN
EMIT(user)
stream_0
GROUPBY
EMIT(hotel)
Define your filter! (4/4)
EACH sum(nights)EACH avg(nights)
SUBMIT TOPOLOGY;
DESC TOPOLOGY;
SHOW TOPOLOGIES;
STOP TOPOLOGY xxx;
DROP TOPOLOGY xxx;
Register your filter! and ...gungnir> SUBMIT TOPOLOGY;OKgungnir> DESC TOPOLOGY;{
"id":"52373363e4b0a525adadb4f2", "explain":" ... ", "status":"RUNNING", "owner":"[email protected]", "createTime":"2013-09-16T16:35:47.432Z", "summary":{
...
}
}
gungnir> STOP TOPOLOGY 52373363e4b0a525adadb4f2 ;
gungnir> DELETE TOPOLOGY 52373363e4b0a525adadb4f2 ;
TRACK userAction {"userId":"siniida","hotelId":"226979"};TRACK commitAction {"userId":"siniida","hotelId":"226979","checkin":"2013-10-18","nights":1};
Testing it!
> bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_user ; or> bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_hotel;
gungnir> DESC TOPOLOGY;{
"id":"52373363e4b0a525adadb4f2", "explain":" ... ", "status":"RUNNING", "owner":"[email protected]", ...
... USING kafka_emit('${TOPOLOGY_ID}_user')
"hotelId" : "128767"
"hotelId" : "356564"
"hotelId" : "226979"
"hotelId" : "325073"
"hotelId" : "357090"
"hotelId" : "126942"
"hotelId" : "310294"
"hotelId" : "367746"
"hotelId" : "232896"
"hotelId" : "220231"
"hotelId" : "163175"
"hotelId" : "367856"
"hotelId" : "360863"
"hotelId" : "119747"
*** We are changing this name because “TRACK” isn’t fit to its function! (advised in GOTO berlin 2013. Thx.)*** The function of this command is to make and post a event to gennai.
Knowing http tuple receiver
> curl -v -H "Content-type: application/json" -X POST \
-d '{"userId":"tf0054","hotelId":"226979"}' \
http://dev.genn.ai:9191/gungnir/v1.0/track/525e67f8e4b0e7cfca98d040/userAction
gungnir> DESC USER;{
"id":"525e67f8e4b0e7cfca98d040", "name":"siniida", "password":"CmGOdN2UU/Xtmk0wRCZKzDy/BN65LAHPCp2+1KMDOOM=", "createTime":"2013-10-16T10:18:32.849Z"
}
gungnir>
Design of genn.aiBuilt on Storm
Converts properly (e.g. building AST)The Good Parts (e.g. Supporting parallelism setting) ..
genn.ai wants friends!genn.ai codes will be open-sourced in the future.We are looking for co-challenging partners for improving genn.ai.
You can use genn.ai on your web sites completely for free (infrastructure etc..) if we can share the dream together.
Please feel free to contact us!http://genn.ai/ [email protected]
The future of genn.ai
http://genn.ai/
Thanks!
19
20