Post on 06-Jan-2017
@PatrickMcFadin
Patrick McFadinChief Evangelist for Apache Cassandra
How to Build Data Pipelines for Real-Time Applications with SMACK & Apache Kafka
1
The problem
Your Magical
App
Sad solutions
SMACK
Spark
Mesos
Akka
Cassandra
Kafka
CassandraAkka
SparkKafka
Organize Process Store
Mesos
KafkaKafkaKafka SparkSparkSpark
AkkaAkkaAkka CassandraCassandraCassandra
CassandraAkka
SparkKafka
Organize Process Store
Managing Weather Data
Windsor California67.3 FRainfall total: 1.2cm
Today:
High: 73.4FLow : 51.4F
Yesterday:
High: 75.2FLow : 52.3F
Our Magical
App
Reactive and immediate
Batch
KillrWeather
KillrWeather
Windsor California67.3 FRainfall total: 1.2cm
Today:
High: 73.4FLow : 51.4F
Yesterday:
High: 75.2FLow : 52.3F
Spark
Mesos
Akka
Cassandra
Kafka
Kafka
Kafka decouples data pipelines
The problem
Kitchen
Hamburgerplease
Meat diskon breadplease
The problem
Kitchen
The problem
Kitchen
Order Queue
Hamburgerplease
Order
The problem
Kitchen
Order Queue
The problem
Kitchen
Order Queue
Meat diskon breadplease
You mean aHamburger?
Uh yeah. That.
Order
Order from chaosProducer
Consumer
Topic = FoodOrder
Order from chaosProducer
Topic = Food
Order
1
Consumer
Order from chaosProducer
Topic = Food
Order
1
Order
Consumer
Order from chaosProducer
Topic = Food
Order
1
Order
2
Consumer
Order from chaosProducer
Topic = Food
Order
1
Order
2
Consumer
Order
Order from chaosProducer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order from chaosProducer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order from chaosProducer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order from chaosProducer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
Order from chaosProducer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
4
Order from chaosProducer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
4
Order
Order from chaosProducer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
4
Order
5
Order from chaosProducer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
4
Order
5
Order from chaosProducer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
4
Order
5
Order from chaosProducer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
4
Order
5
ScaleProducer
Topic = Hamburgers
Order
1
Order
2
Consumer
Order
3
Order
4
Order
5
Topic = Pizza
Order
1
Order
2
Order
3
Order
4
Order
5
Topic = Food
KafkaProducer
Topic = Temperature
Temp
1
Temp
2
Consumer
Temp
3
Temp
4
Temp
5
Collection API
Temperature Processor
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5Precipitation Processor
Broker
KafkaProducer
Topic = Temperature
Temp
1
Temp
2
Consumer
Temp
3
Temp
4
Temp
5
Collection API
Temperature Processor
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5Precipitation Processor
Broker
Partition 0
Partition 0
KafkaProducer Consumer
Collection API
Temperature Processor
Precipitation Processor
Topic = Temperature
Temp
1
Temp
2
Temp
3
Temp
4
Temp
5
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5
Broker
Partition 0
Partition 0
Temp
1
Temp
2
Temp
3
Temp
4
Temp
5Partition 1 Temperature
Processor
KafkaProducer Consumer
Collection API
Temperature Processor
Precipitation Processor
Topic = Temperature
Temp
1
Temp
2
Temp
3
Temp
4
Temp
5
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5
Broker
Partition 0
Partition 0
Temp
1
Temp
2
Temp
3
Temp
4
Temp
5Partition 1
Temperature Processor
Topic = Temperature
Temp
1
Temp
2
Temp
3
Temp
4
Temp
5
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5
Broker
Partition 0
Partition 0
Temp
1
Temp
2
Temp
3
Temp
4
Temp
5Partition 1
Topic TemperatureReplication Factor = 2
Topic PrecipitationReplication Factor = 2
KafkaProducer
Consumer
Collection API
Temperature Processor
Precipitation Processor
Topic = Temperature
Temp
1
Temp
2
Temp
3
Temp
4
Temp
5
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5
Broker
Partition 0
Partition 0
Temp
1
Temp
2
Temp
3
Temp
4
Temp
5Partition 1 Temperature
Processor
Topic = Temperature
Temp
1
Temp
2
Temp
3
Temp
4
Temp
5
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5
Broker
Partition 0
Partition 0
Temp
1
Temp
2
Temp
3
Temp
4
Temp
5Partition 1
Temperature Processor
Temperature Processor
Precipitation Processor
Topic TemperatureReplication Factor = 2
Topic PrecipitationReplication Factor = 2
GuaranteesOrder•Messages are ordered as they are sent by the producer•Consumers see messages in the order they were inserted by the producer
Durability•Messages are delivered at least once•With a Replication Factor N up to N-1 server failures can be tolerated without losing committed messages
CassandraAkka
SparkKafka
Organize Process Store
Mesos
Coming soon!• May 4: How to Achieve High Throughput for Real-Time Applications with SMACK,
Apache Kafka and Spark Streaming
• May 18: How to Build Data Pipelines with SMACK: Storage Strategy using Cassandra and DSE
• June 1: How to Build Data Pipelines with SMACK: Analyzing Data with Spark
• For the latest schedule of webinars, check out our Webinars page: http://www.datastax.com/resources/webinars
Go get your SMACK on
Thank you!
Follow me on twitter: @PatrickMcFadin