CFSCassandra-backed storage for HadoopNick Bailey@nickmbaileynick@datastax.com

©2012 DataStax

Motivation

2

©2012 DataStax

Help me Cassandra, you’re my only hope

3

©2012 DataStax

Cassandra• Distributed architecture

• No SPOF

• Scalable

• Real time data

• No ad-hoc query support

4

©2012 DataStax

Cassandra, why can’t you...

5

©2012 DataStax

...do the things Hadoop was built for.

6

©2012 DataStax

Cassandra + Hadoop = <3

7

©2012 DataStax

The Solution• InputFormat/OutputFormat

• Unfortunately, still need a DFS

• Run tasktrackers/datanodes locally• Data Locality FTW!

• Run namenode/jobtracker somewhere

• Since Cassandra 0.6 (the dark ages)

8

©2012 DataStax

Ok, but what about these parts that suck...

9

©2012 DataStax

Do not want...• Multiple hadoop stacks?

• SPOF?

• 3 JVMS?

10

©2012 DataStax

CFS

11

©2012 DataStax

Cassandra Data model in 1 minute

12

©2012 DataStax

Column Families• Column Family ~= Table

• Row Key + columns

• Columns are sparse

13

©2012 DataStax

Static - Users Column Family

14

Row Key

nickmbailey password: * name: Nick

zznate password: * name: Nate phone: 512-7777

©2012 DataStax

Select * from Users where name=Nick;

Secondary Indexes

15

©2012 DataStax

Dynamic - Friends

16

Row Key

nickmbailey zznate: thobbs:

zznate jbeiber: thobbs: steve_watt:

©2012 DataStax

So what about CFS...

17

©2012 DataStax

Simple...

18

©2012 DataStax 19

©2012 DataStax

CF: inode• Essentially, namenode replacement

• File metadata

20

©2012 DataStax 21

©2012 DataStax

CF: inode• Row Key = UUID

• Allows for file renames

• Secondary indexes for file browsing

• Columns:

22

Columnfilename /home/nick/data.txt

parent_path /home/nick/attributes nick:nick:777

TimeUUID1 <block metadata>TimeUUID2 <block metadata>TimeUUID3 <block metadata>

...

©2012 DataStax 23

©2012 DataStax

CF: sblocks• Essentially, datanode replacement

• Stores actual contents of files

• Each row is an hdfs block

• Row Key = Block ID

24

Column

TimeUUID1 <compressed file data>

TimeUUID2 <compressed file data>

TimeUUID3 <compressed file data>

...

©2012 DataStax 25

©2012 DataStax

Writes• Write file metadata

• Split into blocks• Still controlled by ‘dfs.block.size’• also ‘cfs.local.subblock.size’

• Read in a block• split into sub blocks

• Update inode, sblocks

• rinse, repeat

26

©2012 DataStax 27

©2012 DataStax

Reads• Check for file in inode

• Determine appropriate blocks

• Request blocks via thrift

• If data is local...• ...get location on local filesystem

• If data is remote...• ...get actual file content via thrift

28

©2012 DataStax

What Else?• Current Implementation: 1.0.4• <property>

<name>fs.cfs.impl</name> <value>com.datastax.bdp.hadoop.cfs.CassandraFileSystem</value> </property>

• Supports HDFS append()

• Immutability makes things easy

• See the first incarnation• https://github.com/riptano/brisk

29

Want a job?

nick@datastax.com

Questions?