Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
ZeroMQ as scriptable sockets
Realtimeconf EU 2013Lyon - http://realtimeconf.eu/
Lourens Naudé
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Why this talk ?
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Operations / Platform
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Credentials
ZeroMQ monitoring supportczmq co-maintainerRuby bindingExperimental TLS support
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
99% of deployments don’t need enterprise
reliability
Of reliability
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Then ...
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
... now
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Zero broker, admin, cost, waste, price tag and (almost) zero latency
What is ZeroMQ ?
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
What it isn’t
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
IM for apps
Not a server or message queueA libraryCommunication / concurrencyMessaging semantics for sockets
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
BSD API
sock.bind(“inproc://workers”)sock.connect (“tcp://0.0.0.0:30”)sock.send (“message”)msg = sock.recv ()sock.close ()
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Community
30+ language bindings ...
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Super (scriptable) sockets
What is a ZeroMQ socket ?
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
bind () or connect () to multiple endpoints,
simultaneously
Multiple endpoints
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
bind () or connect () in any order
Async connect
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Socket types
PUB SUB
REQ REP
PUSH PULL
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Request / reply
REQ
REP
REQ
REP REP
REP
REQ REQ
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Request / Reply usage
Lock stepBi-directionalRoutableFlow control, orchestration etc.
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Publish / subscribe
PUB
SUB SUB SUB
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Inversion - logging
SUB
PUB PUB PUB
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Publish / subscribe usage
Market data, game state etc.PUB socket distributesSUB socket filtersSubscription forwarding (3.x)
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Pipeline
PUSH
PULL PULL PULL
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Pipeline usage
Extendable - UNIX pipesRound-robin distributionTasks with variable processing
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Transport agnostic
inproc:// - threads in a processipc:// - processes on a boxtcp:// - nodes in a networkUniform API
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Lightweight switchesDevices
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
pub/sub persistence
PERSISTENCEDEVICE
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Device main loop
loop do m = sub_socket.recv () persist (m) pub_socket.send (m)end
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Use cases
ProxyRouting (LRU)Filtering / de-dupTranslation
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Work distribution
PUSH
PULL
PUSH
PULL
PULL
PUSH
PULL
PUSH
VENTILATOR
SINK
Monday, April 22, 13
PUSH
PULL
PUSH
PULL
PULL
PUSH
PULL
PUSH
VENTILATOR
SINK
PUSH
PULLBear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Scalable work distribution
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Interjection Principle
Scalable topologies
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Saving bandwidth
SUBPUB
PUB
SUB SUB
SUB SUB
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Asynchronous external dependencies.
Resiliency
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Atomicity
Atomic message deliveryUnbound message sizeSend 5GB, recv all or nothing
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Availability
Delayed retry on failureMulti bind() and connect()Unordered bind() or connect()
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Reliability as patterns
Lazy Pirate (REQ / REP)Majordomo (queue)Titanic (persistence)Binary Star (High Availability)See http://zguide.zeromq.org
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
ZeroMQ concurrency
PUSH
PULL PULL PULL
MAIN THREAD
THREAD 1 THREAD 2 THREAD 3
I/OTHREAD
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
s = ctx.bind(:PUSH, "inproc://w")3.times{ spawn_worker(ctx) }loop do msg = work.shift break if msg == “exit” s.send(msg)end
Master (work distributor)
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Thread.new do s = ctx.connect(:PULL, “inproc://w”) loop do msg = s.recv break if msg == "quit" process(msg) endend
Workers (consumer threads)
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Mongrel2Case study
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
How it works
Multi-protocolZeroMQ for backendsAsync backend requestsLanguage agnostic (bindings)
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Mongrel2 topology
PULL
PUB
PULL
PUB
PUSH
SUB
BACKEND A BACKEND B
MONGREL2
CLIENT
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Mongrel2 protocol
UUID ID PATH SIZE:HEADERS,SIZE:BODY
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Mongrel2 HTTP request
PUSH PULL
AB23 4 /uri SIZE:HEADERS,SIZE:BODY
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Mongrel2 HTTP response
SUB PUB
AB23 4 /uri SIZE:HEADERS,SIZE:BODY
Monday, April 22, 13
Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | [email protected]
Questions ?(... or sync up after)
O fim
fork @methodmissingfollow @methodmissing
Monday, April 22, 13
Top Related