Breakthrough OLAPPerformance with
Cassandra and SparkEvan Chan
September 2015
Who am I?
Distinguished Engineer, @evanfchan
User and contributor to Spark since 0.9, Cassandra since 0.6Co-creator and maintainer of
Tuplejump
http://velvia.github.io
Spark Job Server
About Tuplejump is a big data technology leader providing solutions for
rapid insights from data.Tuplejump
- the first Spark-Cassandra integration - an open source Lucene indexer for Cassandra - open source HDFS for Cassandra
CalliopeStargateSnackFS
Didn't I attend the same talk last year?Similar title, but mostly new materialWill reveal new open source projects! :)
Problem SpaceNeed analytical database / queries on structured big data
Something SQL-like, very flexible and fastPre-aggregation too limiting
Fast data / constant updatesIdeally, want my queries to run over fresh data too
Example: Video analyticsTypical collection and analysis of consumer events3 billion new events every dayVideo publishers want updated stats, the sooner the betterPre-aggregation only enables simple dashboard UIsWhat if one wants to offer more advanced analysis, or ageneric data query API?
Eg, top countries filtered by device type, OS, browser
RequirementsScalable - rules out PostGreSQL, etc.Easy to update and ingest new data
Not traditional OLAP cubes - that's not what I'm talkingabout
Very fast for analytical queries - OLAP not OLTPExtremely flexible queriesPreferably open source
ParquetWidely used, lots of support (Spark, Impala, etc.)Problem: Parquet is read-optimized, not easy to use for writes
Cannot support idempotent writesOptimized for writing very large chunks, not small updatesNot suitable for time series, IoT, etc.Often needs multiple passes of jobs for compaction of smallfiles, deduplication, etc.
People really want a database-like abstraction, not a file format!
Cassandra
Horizontally scalableVery flexible data modelling (lists, sets, custom data types)Easy to operatePerfect for ingestion of real time / machine dataBest of breed storage technology, huge communityBUT: Simple queries onlyOLTP-oriented
Apache Spark
Horizontally scalable, in-memory queriesFunctional Scala transforms - map, filter, groupBy, sortetc.SQL, machine learning, streaming, graph, R, many more pluginsall on ONE platform - feed your SQL results to a logisticregression, easy!Huge number of connectors with every single storagetechnology
Spark provides the missing fast, deepanalytics piece of Cassandra!
...tying together fast event ingestion and rich deepanalytics!
How to make Spark and CassandraGo Fast
Spark on Cassandra: No Caching
Not Very Fast, but Real-Time UpdatesSpark does no caching by default - you will always be readingfrom C*!Pros:
No need to fit all data in memoryAlways get the latest data
Cons:Pretty slow for ad hoc analytical queries - using regular CQLtables
How to go Faster?Read less dataDo less I/OMake your computations faster
Spark as Cassandra's Cache
Caching a SQL Table from CassandraDataFrames support in Cassandra Connector 1.4.0 (and 1.3.0):
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val df = sqlContext.read .format("org.apache.spark.sql.cassandra") .option("table", "gdelt") .option("keyspace", "test").load()df.registerTempTable("gdelt")sqlContext.cacheTable("gdelt")sqlContext.sql("SELECT count(monthyear) FROM gdelt").show()
How Spark SQL's Table Caching Works
Spark Cached Tables can be Really FastGDELT dataset, 4 million rows, 60 columns, localhost
Method secsUncached 317
Cached 0.38
Almost a 1000x speedup!
On an 8-node EC2 c3.XL cluster, 117 million rows, can runcommon queries 1-2 seconds against cached dataset.
Problems with Cached TablesStill have to read the data from Cassandra first, which is slowAmount of RAM: your entire data + extra for conversion tocached tableCached tables only live in Spark executors - by default
tied to single context - not HAonce any executor dies, must re-read data from C*
Caching takes time: convert from RDD[Row] to compressedcolumnar formatCannot easily combine new RDD[Row] with cached tables(and keep speed)
Problems with Cached TablesIf you don't have enough RAM, Spark can cache your tablespartly to disk. This is still way, way, faster than scanning an entireC* table. However, cached tables are still tied to a single Sparkcontext/application.
Also: rdd.cache() is NOT the same as SQLContext'scacheTable!
Faster Queries Through Columnar StorageWait, I thought Cassandra was columnar?
How Cassandra stores your CQL TablesSuppose you had this CQL table:
CREATE TABLE ( department text, empId text, first text, last text, age int, PRIMARY KEY (department, empId));
How Cassandra stores your CQL TablesPartitionKey 01:first 01:last 01:age 02:first 02:last 02:ageSales Bob Jones 34 Susan O'Connor 40
Engineering Dilbert P ? Dogbert Dog 1
Each row is stored contiguously. All columns in row 2 come afterrow 1.
To analyze only age, C* still has to read every field.
Cassandra is really a row-based, OLTP-oriented datastore.
Unless you know how to use it otherwise :)
The traditional row-based data storageapproach is dead- Michael Stonebraker
Columnar StorageName column
0 10 1
Dictionary: {0: "Barak", 1: "Hillary"}
Age column
0 146 66
Columnar Format solves I/OHow much data can I query interactively? More than you think!
Columnar Storage Performance Studyhttp://github.com/velvia/cassandra-gdelt
Scenario Ingest Read allcolumns
Read onecolumn
Narrowtable
1927sec
505 sec 504 sec
Widetable
3897sec
365 sec 351 sec
Columnar 93 sec 8.6 sec 0.23 sec
On reads, using a columnar format is up to 2190x faster, whileingestion is 20-40x faster.
Columnar Format solves CachingUse the same format on disk, in cache, in memory scan
Caching works a lot better when the cached object is thesame!!
No data format dissonance means bringing in new bits of dataand combining with existing cached data is seamless
So, why isn't everybody doing this?No columnar storage format designed to work with NoSQLstoresEfficient conversion to/from columnar format a hard problemMost infrastructure is still row oriented
Spark SQL/DataFrames based on RDD[Row]Spark Catalyst is a row-oriented query parser
All hard work leads to profit, but mere talk leadsto poverty.- Proverbs 14:23
Introducing FiloDBDistributed. Versioned. Columnar. Built for Streaming.
github.com/tuplejump/FiloDB
FiloDB - What?
DistributedApache Cassandra. Scale out with no SPOF. Cross-datacenterreplication. Proven storage and database technology.
VersionedIncrementally add a column or a few rows as a new version. Easilycontrol what versions to query. Roll back changes inexpensively.
Stream out new versions as continuous queries :)
ColumnarParquet-style storage layoutRetrieve select columns and minimize I/O for analyticalqueriesAdd a new column without having to copy the whole tableVectorization and lazy/zero serialization for extremeefficiency
What's in the name?
Rich sweet layers of distributed, versioned database goodness
100% ReactiveBuilt completely on the Typesafe Platform:
Scala 2.10 and SBTSpark (including custom data source)Akka Actors for rational scale-out concurrencyFutures for I/OPhantom Cassandra client for reactive, type-safe C* I/OTypesafe Config
Spark SQL Queries!CREATE TEMPORARY TABLE gdeltUSING filodb.sparkOPTIONS (dataset "gdelt");
SELECT Actor1Name, Actor2Name, AvgTone FROM gdelt ORDER BY AvgTone DESC LIMIT 15
Read to and write from Spark DataframesAppend/merge to FiloDB table from Spark StreamingUse Tableau or any other JDBC tool
FiloDB - Why?Fast Streaming Data + Big Data, All in One!
Analytical Query PerformanceUp to 200x Faster Queries for Spark on Cassandra 2.x
Parquet Performance with Cassandra Flexibility(Stick around for the demo)
Fast Event/Time-Series Ad-Hoc Analytics
New rows appended via KafkaWrites are idempotent - no need to dedup!Converted to columnar chunks on ingest and stored in C*Only necessary columnar chunks are read into Spark forminimal I/O
Fast Event/Time-Series Ad-Hoc AnalyticsEntity Time1 Time2US-0123 d1 d2
NZ-9495 d1 d2
Model your time series with FiloDB similarly to Cassandra:
Sort key: Timestamp, similar to clustering keyPartition Key: Event/machine entity
FiloDB keeps data sorted while stored in efficient columnarstorage.
FiloDB = Streaming + Columnar
Extend your Cassandra (2.x) Investment
Make it work for batch and ad-hoc analytics!
Simplify your Lambda Architecture...
( )https://www.mapr.com/developercentral/lambda-architecture
With Spark, Cassandra, and FiloDB
Ma, where did all the components go?You mean I don't have to deal with Hadoop?Use Cassandra as a front end to store IoT data first
FiloDB vs ParquetComparable read performance - with lots of space to improve
Assuming co-located Spark and CassandraOn localhost, both subsecond for simple queries (GDELT1979-1984)FiloDB has more room to grow - due to hot column cachingand much less deserialization overhead
Lower memory requirement due to much smaller block sizesMuch better fit for IoT / Machine / Time-series applications
Idempotent writes by PK with no deduplicationLimited support for types
array / set / map support not there, but will be added
FiloDB - How?
FiloDB Architecture
Ingestion and Storage?Current version:
Each dataset is stored using 2 regular Cassandra tablesIngestion using Spark (Dataframes or SQL)
Future version?
Automatic ingestion of your existing C* data using customsecondary index
Towards Extreme Query Performance
The filo project is a binary data vector library
designed for extreme read performance with minimaldeserialization costs.
http://github.com/velvia/filo
Designed for NoSQL, not a file formatrandom or linear accesson or off heapmissing value supportScala only, but cross-platform support possible
What is the ceiling?This Scala loop can read integers from a binary Filo blob at a rateof 2 billion integers per second - single threaded:
def sumAllInts(): Int = { var total = 0 for { i <- 0 until numValues optimized } { total += sc(i) } total }
Vectorization of Spark QueriesThe project.Tungsten
Process many elements from the same column at once, keep datain L1/L2 cache.
Coming in Spark 1.4 through 1.6
Hot Column Caching in TachyonHas a "table" feature, originally designed for SharkKeep hot columnar chunks in shared off-heap memory for fastaccess
FiloDB - RoadmapSupport for many more data types and sort and partition keys -please give us your input!Non-Spark ingestion API. Your input is again needed.In-memory caching for significant query speedupProjections. Often-repeated queries can be sped upsignificantly with projections.Use of GPU and SIMD instructions to speed up queries
You can help!Send me your use cases for fast big data analysis on Spark andCassandra
Especially IoT, Event, Time-SeriesWhat is your data model?
Email if you want to contribute
Thanks...to the entire OSS community, but in particular:
Lee Mighdoll, Nest/GoogleRohit Rai and Satya B., TuplejumpMy colleagues at Socrata
If you want to go fast, go alone. If you want to gofar, go together.-- African proverb
DEMO TIMEGDELT: Regular C* Tables vs FiloDB
Extra Slides
The scenarios dataset
1979 to now60 columns, 250 million+ rows, 250GB+Let's compare Cassandra I/O only, no caching or Spark
Narrow table - CQL table with one row per partition key
Wide table - wide rows with 10,000 logical rows per partitionkeyColumnar layout - 1000 rows per columnar chunk, wide rows,with dictionary compression
Global Database of Events, Language, and Tone
First 4 million rows, localhost, SSD, C* 2.0.9, LZ4 compression.Compaction performed before read benchmarks.
Disk space usageScenario Disk usedNarrow table 2.7 GB
Wide table 1.6 GB
Columnar 0.34 GBThe disk space usage helps explain some of the numbers.
Connecting Spark to CassandraDatastax's Tuplejump
Spark Cassandra ConnectorCalliope
Get started in one line with spark-shell!bin/spark-shell \ --packages com.datastax.spark:spark-cassandra-connector_2.10:1.4.0-M3 \ --conf spark.cassandra.connection.host=127.0.0.1
What about C* Secondary Indexing?Spark-Cassandra Connector and Calliope can both reduce I/O byusing Cassandra secondary indices. Does this work with caching?
No, not really, because only the filtered rows would be cached.Subsequent queries against this limited cached table would notgive you expected results.
Turns out this has been solved before!
Even .Facebook uses Vertica
MPP Databases
Easy writes plus fast queries, with constant transfersAutomatic query optimization by storing intermediate queryprojectionsStonebraker, et. al. - paper (Brown Univ)CStore
When in doubt, use brute force- Ken Thompson
Top Related