Breakthrough OLAPPerformance with

Cassandra and SparkEvan Chan

September 2015

Who am I?

Distinguished Engineer, @evanfchan

User and contributor to Spark since 0.9, Cassandra since 0.6Co-creator and maintainer of

Tuplejump

http://velvia.github.io

Spark Job Server

About Tuplejump is a big data technology leader providing solutions for

rapid insights from data.Tuplejump

- the first Spark-Cassandra integration - an open source Lucene indexer for Cassandra - open source HDFS for Cassandra

CalliopeStargateSnackFS

Didn't I attend the same talk last year?Similar title, but mostly new materialWill reveal new open source projects! :)

Problem SpaceNeed analytical database / queries on structured big data

Something SQL-like, very flexible and fastPre-aggregation too limiting

Fast data / constant updatesIdeally, want my queries to run over fresh data too

Example: Video analyticsTypical collection and analysis of consumer events3 billion new events every dayVideo publishers want updated stats, the sooner the betterPre-aggregation only enables simple dashboard UIsWhat if one wants to offer more advanced analysis, or ageneric data query API?

Eg, top countries filtered by device type, OS, browser

RequirementsScalable - rules out PostGreSQL, etc.Easy to update and ingest new data

Not traditional OLAP cubes - that's not what I'm talkingabout

Very fast for analytical queries - OLAP not OLTPExtremely flexible queriesPreferably open source

ParquetWidely used, lots of support (Spark, Impala, etc.)Problem: Parquet is read-optimized, not easy to use for writes

Cannot support idempotent writesOptimized for writing very large chunks, not small updatesNot suitable for time series, IoT, etc.Often needs multiple passes of jobs for compaction of smallfiles, deduplication, etc.

 

People really want a database-like abstraction, not a file format!

Cassandra

Horizontally scalableVery flexible data modelling (lists, sets, custom data types)Easy to operatePerfect for ingestion of real time / machine dataBest of breed storage technology, huge communityBUT: Simple queries onlyOLTP-oriented

Apache Spark

Horizontally scalable, in-memory queriesFunctional Scala transforms - map, filter, groupBy, sortetc.SQL, machine learning, streaming, graph, R, many more pluginsall on ONE platform - feed your SQL results to a logisticregression, easy!Huge number of connectors with every single storagetechnology

Spark provides the missing fast, deepanalytics piece of Cassandra!

...tying together fast event ingestion and rich deepanalytics!

How to make Spark and CassandraGo Fast

Spark on Cassandra: No Caching

Not Very Fast, but Real-Time UpdatesSpark does no caching by default - you will always be readingfrom C*!Pros:

No need to fit all data in memoryAlways get the latest data

Cons:Pretty slow for ad hoc analytical queries - using regular CQLtables

How to go Faster?Read less dataDo less I/OMake your computations faster

Spark as Cassandra's Cache

Caching a SQL Table from CassandraDataFrames support in Cassandra Connector 1.4.0 (and 1.3.0):

val sqlContext = new org.apache.spark.sql.SQLContext(sc)

val df = sqlContext.read .format("org.apache.spark.sql.cassandra") .option("table", "gdelt") .option("keyspace", "test").load()df.registerTempTable("gdelt")sqlContext.cacheTable("gdelt")sqlContext.sql("SELECT count(monthyear) FROM gdelt").show()

 

How Spark SQL's Table Caching Works

Spark Cached Tables can be Really FastGDELT dataset, 4 million rows, 60 columns, localhost

Method secsUncached 317

Cached 0.38

 

Almost a 1000x speedup!

On an 8-node EC2 c3.XL cluster, 117 million rows, can runcommon queries 1-2 seconds against cached dataset.

Problems with Cached TablesStill have to read the data from Cassandra first, which is slowAmount of RAM: your entire data + extra for conversion tocached tableCached tables only live in Spark executors - by default

tied to single context - not HAonce any executor dies, must re-read data from C*

Caching takes time: convert from RDD[Row] to compressedcolumnar formatCannot easily combine new RDD[Row] with cached tables(and keep speed)

Problems with Cached TablesIf you don't have enough RAM, Spark can cache your tablespartly to disk. This is still way, way, faster than scanning an entireC* table. However, cached tables are still tied to a single Sparkcontext/application.

Also: rdd.cache() is NOT the same as SQLContext'scacheTable!

Faster Queries Through Columnar StorageWait, I thought Cassandra was columnar?

How Cassandra stores your CQL TablesSuppose you had this CQL table:

CREATE TABLE ( department text, empId text, first text, last text, age int, PRIMARY KEY (department, empId));

How Cassandra stores your CQL TablesPartitionKey 01:first 01:last 01:age 02:first 02:last 02:ageSales Bob Jones 34 Susan O'Connor 40

Engineering Dilbert P ? Dogbert Dog 1

 

Each row is stored contiguously. All columns in row 2 come afterrow 1.

To analyze only age, C* still has to read every field.

Cassandra is really a row-based, OLTP-oriented datastore.

Unless you know how to use it otherwise :)

The traditional row-based data storageapproach is dead- Michael Stonebraker

Columnar StorageName column

0 10 1

 

Dictionary: {0: "Barak", 1: "Hillary"}

 

Age column

0 146 66

Columnar Format solves I/OHow much data can I query interactively? More than you think!

 

Columnar Storage Performance Studyhttp://github.com/velvia/cassandra-gdelt

Scenario Ingest Read allcolumns

Read onecolumn

Narrowtable

1927sec

505 sec 504 sec

Widetable

3897sec

365 sec 351 sec

Columnar 93 sec 8.6 sec 0.23 sec 

On reads, using a columnar format is up to 2190x faster, whileingestion is 20-40x faster.

Columnar Format solves CachingUse the same format on disk, in cache, in memory scan

Caching works a lot better when the cached object is thesame!!

No data format dissonance means bringing in new bits of dataand combining with existing cached data is seamless

So, why isn't everybody doing this?No columnar storage format designed to work with NoSQLstoresEfficient conversion to/from columnar format a hard problemMost infrastructure is still row oriented

Spark SQL/DataFrames based on RDD[Row]Spark Catalyst is a row-oriented query parser

All hard work leads to profit, but mere talk leadsto poverty.- Proverbs 14:23

Introducing FiloDBDistributed. Versioned. Columnar. Built for Streaming.

 

github.com/tuplejump/FiloDB

FiloDB - What?

DistributedApache Cassandra. Scale out with no SPOF. Cross-datacenterreplication. Proven storage and database technology.

VersionedIncrementally add a column or a few rows as a new version. Easilycontrol what versions to query. Roll back changes inexpensively.

Stream out new versions as continuous queries :)

ColumnarParquet-style storage layoutRetrieve select columns and minimize I/O for analyticalqueriesAdd a new column without having to copy the whole tableVectorization and lazy/zero serialization for extremeefficiency

What's in the name?

Rich sweet layers of distributed, versioned database goodness

100% ReactiveBuilt completely on the Typesafe Platform:

Scala 2.10 and SBTSpark (including custom data source)Akka Actors for rational scale-out concurrencyFutures for I/OPhantom Cassandra client for reactive, type-safe C* I/OTypesafe Config

Spark SQL Queries!CREATE TEMPORARY TABLE gdeltUSING filodb.sparkOPTIONS (dataset "gdelt");

SELECT Actor1Name, Actor2Name, AvgTone FROM gdelt ORDER BY AvgTone DESC LIMIT 15

Read to and write from Spark DataframesAppend/merge to FiloDB table from Spark StreamingUse Tableau or any other JDBC tool

FiloDB - Why?Fast Streaming Data + Big Data, All in One!

Analytical Query PerformanceUp to 200x Faster Queries for Spark on Cassandra 2.x

Parquet Performance with Cassandra Flexibility(Stick around for the demo)

Fast Event/Time-Series Ad-Hoc Analytics

New rows appended via KafkaWrites are idempotent - no need to dedup!Converted to columnar chunks on ingest and stored in C*Only necessary columnar chunks are read into Spark forminimal I/O

Fast Event/Time-Series Ad-Hoc AnalyticsEntity Time1 Time2US-0123 d1 d2

NZ-9495 d1 d2 

Model your time series with FiloDB similarly to Cassandra:

Sort key: Timestamp, similar to clustering keyPartition Key: Event/machine entity

FiloDB keeps data sorted while stored in efficient columnarstorage.

FiloDB = Streaming + Columnar

Extend your Cassandra (2.x) Investment

Make it work for batch and ad-hoc analytics!

Simplify your Lambda Architecture...

( )https://www.mapr.com/developercentral/lambda-architecture

With Spark, Cassandra, and FiloDB

Ma, where did all the components go?You mean I don't have to deal with Hadoop?Use Cassandra as a front end to store IoT data first

FiloDB vs ParquetComparable read performance - with lots of space to improve

Assuming co-located Spark and CassandraOn localhost, both subsecond for simple queries (GDELT1979-1984)FiloDB has more room to grow - due to hot column cachingand much less deserialization overhead

Lower memory requirement due to much smaller block sizesMuch better fit for IoT / Machine / Time-series applications

Idempotent writes by PK with no deduplicationLimited support for types

array / set / map support not there, but will be added

FiloDB - How?

FiloDB Architecture

Ingestion and Storage?Current version:

Each dataset is stored using 2 regular Cassandra tablesIngestion using Spark (Dataframes or SQL)

Future version?

Automatic ingestion of your existing C* data using customsecondary index

Towards Extreme Query Performance

The filo project is a binary data vector library

designed for extreme read performance with minimaldeserialization costs.

http://github.com/velvia/filo

Designed for NoSQL, not a file formatrandom or linear accesson or off heapmissing value supportScala only, but cross-platform support possible

What is the ceiling?This Scala loop can read integers from a binary Filo blob at a rateof 2 billion integers per second - single threaded:

def sumAllInts(): Int = { var total = 0 for { i <- 0 until numValues optimized } { total += sc(i) } total }

Vectorization of Spark QueriesThe project.Tungsten

Process many elements from the same column at once, keep datain L1/L2 cache.

Coming in Spark 1.4 through 1.6

Hot Column Caching in TachyonHas a "table" feature, originally designed for SharkKeep hot columnar chunks in shared off-heap memory for fastaccess

FiloDB - RoadmapSupport for many more data types and sort and partition keys -please give us your input!Non-Spark ingestion API. Your input is again needed.In-memory caching for significant query speedupProjections. Often-repeated queries can be sped upsignificantly with projections.Use of GPU and SIMD instructions to speed up queries

You can help!Send me your use cases for fast big data analysis on Spark andCassandra

Especially IoT, Event, Time-SeriesWhat is your data model?

Email if you want to contribute

Thanks...to the entire OSS community, but in particular:

Lee Mighdoll, Nest/GoogleRohit Rai and Satya B., TuplejumpMy colleagues at Socrata

 

If you want to go fast, go alone. If you want to gofar, go together.-- African proverb

DEMO TIMEGDELT: Regular C* Tables vs FiloDB

Extra Slides

The scenarios dataset

1979 to now60 columns, 250 million+ rows, 250GB+Let's compare Cassandra I/O only, no caching or Spark

Narrow table - CQL table with one row per partition key

Wide table - wide rows with 10,000 logical rows per partitionkeyColumnar layout - 1000 rows per columnar chunk, wide rows,with dictionary compression

Global Database of Events, Language, and Tone

First 4 million rows, localhost, SSD, C* 2.0.9, LZ4 compression.Compaction performed before read benchmarks.

Disk space usageScenario Disk usedNarrow table 2.7 GB

Wide table 1.6 GB

Columnar 0.34 GBThe disk space usage helps explain some of the numbers.

Connecting Spark to CassandraDatastax's Tuplejump

Spark Cassandra ConnectorCalliope

 

Get started in one line with spark-shell!bin/spark-shell \ --packages com.datastax.spark:spark-cassandra-connector_2.10:1.4.0-M3 \ --conf spark.cassandra.connection.host=127.0.0.1

What about C* Secondary Indexing?Spark-Cassandra Connector and Calliope can both reduce I/O byusing Cassandra secondary indices. Does this work with caching?

No, not really, because only the filtered rows would be cached.Subsequent queries against this limited cached table would notgive you expected results.

Turns out this has been solved before!

Even .Facebook uses Vertica

MPP Databases

Easy writes plus fast queries, with constant transfersAutomatic query optimization by storing intermediate queryprojectionsStonebraker, et. al. - paper (Brown Univ)CStore

When in doubt, use brute force- Ken Thompson