Introduction to ActiveMQ Apollo

A Progress So3ware Company

Introduc6on to Ac6veMQ Apollo

Bosanac Dejan May 2011

About me

 Bosanac Dejan  Senior So3ware Engineer at FUSESource ‐ hNp://fusesource.com  Apache Ac6veMQ commiNer and PMC member  Co‐author of Ac6veMQ in Ac6on

What we are going to cover?

 Why Apollo?   HawtDispatch   Connec6vity

•  Stomp 1.1, MQTT, JMS, ...

  LevelDB Store   Features

•  REST Based Management

  Future

Why Apollo?

Apache Apollo

  An experiment to beNer u6lize high core counts on modern processors

  A completely new broker core that is much more determinis6c, stable, and scaleable

  Branched as a new project and a chance for a clean start

Resulted

Apollo Architecture

  Reactor Based Thread Model   Scala implementa6on   Protocol Agnos6c   REST Based Management

HawtDispatch

HawtDispatch ‐ Introduc6on

  Small (less than 100k) thread pooling and NIO event no6fica6on framework API

  hNp://hawtdispatch.fusesource.org/   Java clone of Grand Central Dispatch   Avoid explicit usage of threads and synchroniza6on points in mul6threaded applica6ons

  Applica6on developer submit tasks to dispatch queues   Fixed‐sized thread pool execute tasks

HawtDispatch ‐ Dispatch Queues

  Global Dispatch Queue •  3 global queues shared • Non determinis6c order •  3 priori6es

  Serial Dispatch Queue •  Serial FIFO queues •  Synchronize certain task execu6ons

DispatchQueue queue = getGlobalQueue(HIGH);

DispatchQueue queue = createQueue("My queue");

HawtDispatch ‐ Submijng Tasks

  Java

  Scala

  Tasks •  non‐blocking •  lock‐free • wait‐free

queue.execute(new Runnable(){ public void run() { System.out.println("Hi!"); } });

queue { System.out.println("Hi!"); } // or queue.execute(^{ System.out.println("Hi!"); })

HawtDispatch ‐ Dispatch Sources

  Trigger a task execu6on based on external event   Ideal for dealing with NIO events

SelectableChannel channel = ... DispatchQueue queue = createQueue() DispatchSource source = createSource(channel, OP_READ, queue); source.setEventHandler(new Runnable(){ public void run() { ByteBuffer buffer = ByteBuffer.allocate(1024); int count; while( (c=channel.read(buffer)) > 0 ) { // just dump it to the console System.out.write(buffer.array(), buffer.offset(), buffer.position()); } } }); source.resume();

HawtDispatch ‐ conclusion

  Ideal for developing highly concurrent applica6ons   Ideal for handling NIO events   Ideal for implemen6ng broker cores   Impressive performances

  New development paradigm means it couldn’t be fiNed into exis6ng Ac6veMQ broker

Connec6vity

Message protocols ‐ Overview

  Broker Core is Protocol Agnos6c   Protocols are Plugins

•  STOMP 1.0/1.1 • MQTT v3.1 • Openwire

  Transports are Pluggable too. •  TCP, WebSockets etc.

  Protocol detec6on (all protocols can use 1 port)

Feature: Mul6ple Transports

  TCP   SSL  WebSockets   Secure WebSockets   UDP

Stomp ‐ basics

  hNp://stomp.github.com/   Simple Text Orientated Messaging Protocol   HTTP for the messaging realm   Very simple, so it’s easy to write clients and servers in prac6cally any language

  A lot of exis6ng clients in C, Ruby, Pyhton, JS, PHP, etc.   Provides a way to connect different plamorms in asynchronous way   Also Implemented by Ac6veMQ, RabbitMQ, HornetQ, and many others.

Stomp ‐ Nutshell

  Based on text frames, similar to HTTP ones

  Can transport binary bodies   Frame command for every messaging concept, like CONNECT, MESSAGE, SUBSCRIBE, ACK, etc.

  New in version 1.1 •  Protocol nego6a6on • Heartbeats • NACK • Virtual hosts

MESSAGE subscription:0 message-id:007 destination:/queue/a content-type:text/plain hello queue a^@

Apollo and Stomp

  Both 1.0 and 1.1 Supported   Des6na6on Types

• Queues ‐ /queue/a •  Topics ‐ /topic/b• Durable Subscrip6ons ‐ /dsub/c

  Reliable messaging

SEND destination:/queue/a receipt:001 persistent: true hello queue a ^@ RECEIPT receipt-id:001 ^@

Apollo and Stomp

 More messaging features • Message expira6on •  Topic retained messages •  Topic durable subscrip6ons • Queue browsing • Queue message sequences •  Exclusive subscrip6ons •  Temporary des6na6ons • Des6na6on wildcards •  Composite des6na6ons • Message selectors

Connec6vity ‐ MQTT

  Get at hNps://github.com/fusesource/fuse‐extra/   Focused on:

•  low bandwidth networks •  unreliable networks •  Small footprint / Embedded Devices

  3 QoS Op6ons   Also Implemented by WebsphereMQ, MosquiNo and others.

Connec6vity – JMS API

  StompJMS is a JMS 1.1 client implemented using the STOMP protocol.

  Get it at hNps://github.com/fusesource/stompjms   Client implemented with HawtDispatch

•  Constant number of threads no maNer how many client connec6ons are established.

import org.fusesource.stomp.jms.*;import javax.jms.*;StompJmsConnectionFactory factory = new StompJmsConnectionFactory();factory.setBrokerURI("tcp://localhost:61613”);Connection connection = factory.createConnection(“admin”, “password”);Destination example = new StompJmsDestination(“/queue/example”);

Connec6vity – OpenWire

 OpenWire is the na6ve binary protocol implemented by Ac6veMQ   API op6ons:

•  JMS 1.1 Client of Ac6veMQ 5.x • NMS Client for C# Apps •  CMS Client for C++ Apps

 Working Features • Queues, Topics, Durable Subscrip6ons •  Temporary Des6na6ons •  Transac6ons

  Not Yet Supported •  XA Transac6ons (distributed transac6ons) • Network of Brokers style clustering •  0 sized consumer prefetches

LevelDB Store

Message Store ‐ overview

 Message Stores are Plugins   Ships with 2 Op6ons

•  LevelDB Store •  BDB Store

  Used to store •  persistent messages •  non‐persistent messages that needs to be swapped out of memory

  Non‐persistent messages that get swapped out do not get dropped on restart

  Delayed Writes

LevelDB ‐ Basics

 What is LevelDB •  LevelDB is a fast key‐value storage library • WriNen at Google •  Provides an ordered mapping from string keys to string values •  Based on SSTable and Log Structured Merge (LSM) Trees

LevelDB ‐ Basics

  Designed to efficiently store large numbers of key‐value pairs while op6mizing for high throughput, sequen6al read/write workloads

  Ideal for implemen6ng message store index  Much beNer performance over tradi6onal B‐Tree indexes (used in KahaDB)

LevelDB ‐ Basics

  It’s a C++ library (no client‐server support)   Batching writes   Forward and backward itera6on over data   Uses Snappy compression

LevelDB Store

  Uses LevelDB to maintain indexes into log files holding the messages

  Uses a JNI driver on Linux and OS X, but falls back to a pure Java version on other plamorms

  Supports replica6on to get High Availability   Default store in Apollo. Available in Apache Ac6veMQ 5.6.0

LevelDB vs KahaDB

  It maintains fewer index entries per message than KahaDB which means it has a higher persistent throughput.

  Faster recovery when a broker restarts   LevelDB based index provide a much beNer performance than the B‐Tree for sequen6al access

  LevelDB indexes support concurrent read access   Pauseless data log file garbage collec6on cycles

LevelDB vs KahaDB

  Uses fewer read IO opera6ons to load stored messages.   It will only journal the payload of the message once   Exposes it's status via JMX for monitoring   Supports replica6on to get High Availability

LevelDB store ‐ HA

  HA version of store works with Hadoop based file systems  Message log is mirrored to HDFS   It can sync on HDFS file instead of local file system   LevelDB indexes are immutable on disk (SSTables)  On checkpoint .sst files are uploaded to HDFS

LevelDB store ‐ HA Recovery

 On master failure, slave will download • message log •  .sst files associated with the latest uploaded index

  Con6nue with regular recovery process

LevelDB store – HA locking

 Master elec6on is done externally  Mul6ple brokers should never use the same HDFS path   Apache ZooKeeper good op6on for implemen6ng distributed locking

  FuseFabric (hNp://fuse.fusesource.org/fabric/) can be used as well

BDB store

 Not ASL 2.0! You have to Agree to the BDB license & download from Oracle.

 Pure Java implementa6on  Very robust  The BDB library supports advanced features like replica6on (not yet exploited)

Features

Feature: JAAS Authen6ca6on

 Use any 3rd party JAAS login module  Ships with security enabled by default • Default id/password is admin/password

 File based user and group configura6on  Supports IP address white and black lists.  X509 Cer6ficates  Op6onal guest login support

Feature: Authoriza6on rules

 Fine grained control of who can • admin, monitor, configure, connect, create, destroy, • send, receive, consume

 On broker resources like: • broker, connector, virtual host, topic, queue or durable subscrip6ons

<access_rule allow="bartenders" action="send,consume” kind=”queue topic” id=”BAR.*”/> <access_rule deny="guests" action="consume"/>

Feature: Config Updates

 All Configura6on files are watch and changes are applied at run6me: • Broker config: etc/apollo.xml • JAAS config files like: etc/user.proper6es • Logging config: etc/log4j.proper6es

 No need to restart to apply config changes

Feature: Config Updates

 All Configura6on files are watch and changes are applied at run6me: • Broker config: etc/apollo.xml • JAAS config files like: etc/user.proper6es • Logging config: etc/log4j.proper6es

 No need to restart to apply config changes

Feature: REST Management

curl -u "admin:password" \ http://localhost:61680/broker/virtual-hosts/default.json

Where is it going?

Where has it been?

Feature diff vs Ac6veMQ

Missing in Apollo  Networks of Brokers  Priority Support  Message Groups  Message Scheduling  XA Transac6ons  JMX Management API

Only in Apollo  REST Management API  Secure WebSockets  Message Sequences  Con6nuous Queue Browsing  Run6me Config Updates  Per Consumer Store Prefetch

Future

  Start integra6ng it in Ac6veMQ • Apollo should be a new broker engine for 6.0 • We should try to port all exis6ng features (networks, priority queues, XA, etc.)

  Tons of interes6ng work ahead of us ‐ Join us

Ques6ons?

  Links: • Apache Apollo hNp://ac6vemq.apache.org/apollo/ •  STOMP Benchmarks hNp://hiramchirino.com/stomp‐benchmark/ • MQTT Protocol Plugin for Apollo hNps://github.com/fusesource/fuse‐extra • HawtDispatch hNp://hawtdispatch.fusesource.org/ •  StompJMS hNps://github.com/fusesource/stompjms

  Blog: hNp://www.nighNale.net

  TwiNer: •  hNp://twiNer.com/dejanb •  hNp://twiNer.com/fusenews

Introduction to ActiveMQ Apollo

Technology

Transcript of Introduction to ActiveMQ Apollo

Apollo x16 Hardware Manual - Universal Audiomedia.uaudio.com/support/manuals/hardware/Apollo x16 Hardware … · Apollo x16 Hardware Manual 8 Introduction Apollo x16 Documentation

Apache ActiveMQ and Apache ServiceMix

Using Apache ActiveMQ with Ruby

Using Apache ActiveMQ with COBOL

Using Apache ActiveMQ with Lisp

Tutorial Jms Activemq

Polyglot Messaging with Apache ActiveMQ

Apache Beam Introduction to - events.static.linuxfound.org · Introduction to Apache Beam JB Onofr ... Apache Kafka, Apache ActiveMQ, tailing filesystem...

Apollo Apollo Scott Brown | Andrew Levy. 350 Mission Introduction | Lighting Design | Lighting Controls | Energy Savings | Conclusions Apollo Apollo.

Apache Camel + Apache ActiveMQ persistence

Apollo Workshop AGS2017 Introduction

Advanced messaging with Apache ActiveMQ

ActiveMQ Startup Guide · 2020-01-07 · ActiveMQ Startup Guide 1 ActiveMQ Basic Guide ActiveMQ is a low level messaging services that internally uses JMS Java Messaging Services

IBM MQ vs Apache ActiveMQ

read.pudn.comread.pudn.com/downloads418/ebook/1774626/ActiveMQ+in+Action.… · I. An Introduction to Messaging and ActiveMQ .............................................1 1. Introduction

Apollo Software Introduction

Messaging With ActiveMQ

Using the JNBridge JMS Adapter for .NET with ActiveMQ › guides › UsingDotNetAdapterActiveMQ.pdf · 4 Using the JMS Adapter with ActiveMQ Version 4.0 Quick Config for ActiveMQ

ActiveMQ 5.9.x new features

Introduction to Apache ActiveMQ Webinar Slides