Kappa Architecture, IoT of the cars - LibreCon 2016
23
Kappa Architecture: IoT of the cars Juantomás García
Transcript of Kappa Architecture, IoT of the cars - LibreCon 2016
Kappa Architecture: IoT of the cars
Juantomás García
Who am I• Data Solutions Manager at Open Sistemas
• Co-author of the first spanish book about free software “La pastilla roja”
• Former president of Hispalinux (Spanish Linux User Group).
• Co-organizer of Machine Learning Spain Meetup
What’s Kappa ArchitectureJuly 2, 2014 Jay Kreps coined the term Kappa Architecture in an article for O’reilly Radar
Who is Jay KrepsJay has been involved in lots of projects:
Author of the essay:
The Log: What every software engineer should know about real-time data's unifying abstraction (12/16/2013)
https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
Who is Jay KrepsAuthor of the book: I ♥ Logs
Who is Jay KrepsInvolved with projects as:
Apache Kafka
Apache Samza
Voldemort
Azkaban
Ex-Linkedin
Now co-founder and CEO of Confluent
Usual data flow
Kappa Architecture Way
Tools we use to implement Kappa Architecture
Tools we use to implement Kappa Architecture
Tools we use to implement Kappa Architecture
• Our favorite kafka features.
• Easy to use
• Robust & Resilient
• Kafka connectors
• Kafka streams
• Compact logs
• Very fast writes and reads
Tools we use to implement Kappa Architecture
Some favorite Spark features • If you have an schema spark SQL is
perfect.
• Spark streaming work very fine with spark and almost all streaming sources.
• Structured queries will be a huge advance.
• We love scala and scala is the spirit of spark.
Some favorite Spark features • We love code like this:
A real case• One of our clients want to monitor all the
cars informations via OBD II
• OBD II is a car interface with the car electronics.
• Client develop an app to read all the car information throw a ODB II with bluetooth
A real case
First Problems• We need to scale the rest interfaces too
many requests.
• MySQL don’t scale
• Client wants to make realtime expensive queries.
Some Numbers
Architecture V2
Architecture KappaV3
Now we’re more flexible• We can have queries like:
• What are the drivers that are not client of the X gas brand, has a few gas and are near of gas station of the brand X and if true, send a notification with a discount coupon and a link with the route.
Takeaways• Kappa architecture is not a silver bullet but
helps in a lot of solutions.
• Kafka + spark streaming is our favorite tools
• There a lots of improvements:
• OLAP like Apache Druid
• Graph databases like neo4j
• Kafka streams and compacts logs
