Integrating PostgreSql with RabbitMQ
-
Upload
gavin-roy -
Category
Data & Analytics
-
view
3.079 -
download
0
description
Transcript of Integrating PostgreSql with RabbitMQ
Integrating PostgreSQL with RabbitMQGavin M. Roy
April 4th, 2014
About MeVP of Architecture
AWeber Communications !
Blog: http://gavinroy.com Twitter: @Crad
GitHub: https://github.com/gmr !
RabbitMQ in Depth (MEAP) http://manning.com/roy
RabbitMQ Primer
About RabbitMQ• Message Oriented Middleware
• Primary identity is defined by AMQP 0-9-1
• Supports other protocols with Plugins: AMQP 1.0, HTTP, MQTT, STOMP, XMPP, UDP, WebStomp, and more
• Written in Erlang/OTP, is Open Source (MPL), and is developed/maintained by Pivotal
Why RabbitMQ?• Create loosely coupled architectures
• Decouple database write operations
• Communicate across application platforms
• Tap into pre-existing message flow for new purposes
• Scale-out clustering for growth, throughput & HA
• Federation for WAN latencies & network partitions
Native AMQP ClientsC
Clojure
Cobol
Common Lisp
Delphi
Erlang
Go
Groovy
Haskell
Java
JavaScript
.NET
OCaml
Perl
PHP
Python
Ruby
Scala
Who Uses It?Agora Games
Chef
Google AdMob
MeetMe
Mercado Libre
Mozilla
Nasa
New York Times
NSF
Openstack
RabbitMQ Concepts
AMQP Message
Body
Properties
Message Propertiesapp-id
content-encoding
content-type
correlation-id
delivery-mode
expiration
headers
message-id
priority
reply-to
timestamp
type
user-id
Body
Properties
Publishers and Consumers
CP
Multiple Publishers
P
C
P
P
Multiple Consumers
P
C
C
C
AMQ Model
Exchange defines routing behavior
Queuestores messages in memory and optionally on disk
Bindingdefines relationship between exchanges and queues
X
Queue
Binding
Routing Keys• Provided when publishing a message
• Compared against binding keys by exchanges
• Example uses of a Routing Key:
• Connotate the type of message
• Designate the destination of a message
• Categorize the content in the message
Built-In Exchange TypesDirectString matching on Routing Key
Fanout No routing key, messages sent to all bound queues
TopicPattern matching in Routing Key
HeadersNo routing key, string matching in the message headers property
Topic Exchange Binding Keysnamespace.delimited.keys
# Receive all messages
namespace.# Receive all messages in the namespace
namespace.delimited.* Receive all namespace.delimited messages
namespace.*.keys Receive all namespace messages ending in keys
Example Exchange PluginsConsistent HashingDistribute messages via routing key hashed value
Pulse Publish internal RabbitMQ metrics via AMQP
Random Distribute messages across all bound queues randomly
Recent History Stores last 20 messages to any bound queue
Riak Storage Store received messages in Riak"
Script Exchange Calls out to external scripts for routing behavior
Exchange to Exchange Bindings
X
X
X
Queue
Queue
Queue
to RabbitMQ
pg_amqp• PostgreSQL Extension
• User-defined functions to publish via AMQP
• Developed by OmnTI
• Invoke from user-defined function
• Available on PGXN
Using pg_amqp• Has table of AMQP broker connections
• Publishes body only AMQP messages
• Transactional publishing via amqp.publish
• Non-transactional via amqp.autonomous_publish
• Needs some love for additional features
https://github.com/gmr/On-Rabbits-and-ElephantsExample:
SELECT amqp.publish(broker_id, 'name', 'routing-key', 'message');
Publishing a Message
PgSQL Listen Exchange• Exchange that issues LISTEN and publishes received
notifications to bound queues
• When routing messages, bindings are checked for the routing key to match the NOTIFY channel
• Unobtrusive to PostgreSQL environment, native constructs for publishing messages
• Requires PostgreSQL 9.0 and greater
• Body only messages*
Sending a Notificationpsql (9.3.2) Type "help" for help. !postgres=# \timing Timing is on. postgres=# NOTIFY channel_name, ‘Test notification'; NOTIFY Time: 0.160 ms
Receiving a Notificationpsql (9.3.2) Type "help" for help. !postgres=# LISTEN channel_name; !(after NOTIFY issued) !Asynchronous notification "channel_name" with payload "Test notification" received from server process with PID 16749.
Example Notification Message
to PostgreSQL
No direct integrations yet
Chapter 14Creating a pgsql-storage
RabbitMQ plugin
Other ways?
via Consumer apps in the language of your choice
via Apache Flume with RabbitMQ Source, PostgreSQL Sink
Questions?rabbitmq-pgsql-listen-exchange https://github.com/aweber/rabbitmq-pgsql-listen-exchange
pg_ampq http://pgxn.org/dist/pg_amqp/
RabbitMQ in Depth 44% off code pgcf (includes other books as well)http://manning.com/roy/
Follow me on Twitter @Crad