Hadoop Demystified 101 ETL + Automation Smackdown

Learn Big Data: Learn manually, then ask “Which approach makes me the most valuable as developer?”

Slides, code, youtube, resources at end

Q&A at end

Bio - Pete Carapetyan

• Java dev last 15 years, dev 20 years

• Grew up automating in a different industry

• Almost involuntary obsession with systems & automation

• Since 2000 as dataFundamentals, now a 2 man shop

Contact info at last slide

Special Skills - Special Snowflakes

• Let me show you these Hadoop basics.

• Then, we code for special snowflakes. (data sets)

• Thus we are more valuable, and can up our bill rates!

• This is Approach #1: Special Snowflake (manual)

My 2013 Manual Hadoop Benchmark

• 15 ETL jobs [Partial scope]

• Brilliant, ninja level team

• 1 year of competitive NIH* copy paste spaghetti coding - AKA special snowflake approach

• Is this the best I can do?

*NIH: Not Invented Here

Hadoop sidebar: which Serialization Protocol?

• [text - native via SequenceFile]

• Binary protocols include

• Thrift (Facebook, Evernote)

• Protocol Buffers (Google)

• Avro (Hadoop author, Cloudera)

• What about character based?

• XML

• JSON

• etc

http://www.slideshare.net/IgorAnishchenko/pb-vs-thrift-vs-avroLess than complementary view of Avro:

Transform to Avro

• Not detailed in this talk

• Demo’d here as a binary

• Code listed at end of talk

[Demo Basics of Hadoop ETL Job]

Whoops - [lots of moving parts!]

• What if I make a misteak?

• Dig through log files

• Obtuse messages

• Scripts for logs are critical

• Budget lots of time

• Error UI

I don’t get it. What makes Hadoop so cool?• Expands to thousands of machines

• Placement of my data across those machines (uses HDFS)

• Moves program to data, not data to program

• Tooling/ecosystem

• Much of which is now usable outside Hadoop

• Examples:

• Hive

• Pig

• Zookeeper

Map Reduce 101?

• Makes more sense as ”MapShuffleReduce”

• API for handing program to the data.

• Primary feature is the two pass heuristic for dealing with data on clusters

• You can avoid understanding Map Reduce if Hive is all you use :(

• Yes, MapReduce runs on other systems than Hadoop!

Special Snowflake Approach:Human drama!

What limitations of this manual special skills special snowflakes

approach do we observe?

How To Un-Pack Either Approach?

What if we remove the human drama?

Now, what happens if we automate?

Automated Approach

Carrie

Our own internal project for automating big data.Name inspired by the horror film…

How to drive focus? The Phoenix Project

• Results, not drama

• Focus only on bottleneck

• Brent as bottleneck

Brent: The bad guy?

• Brent is a team’s best asset! Brent is a ninja.Brent is not the bad guy.

• Brent is bottleneck only when treating every situation like a special snowflake.

• Brent enjoys attention???

• Brent is not the drama queen, others bring the drama to him.

• Often victim of his own success.

Brent?

Automation Basics

1. Brent spends time on clean design, PIE*, not NIH*

• Uses [Camel] - Integration Server

2. Brent automates the rule, codes the exception

• Apply metadata to templates

• Infrastructure as code: servers(Devops)

* NIH: Not Invented Here especially as opposed to PIE “Proudly Invented Elsewhere”

Demo Integration Server

• Raw linux OS (Centos)

• Java

• Maven

• Ruby

• networking

• maven repo - binaries

• [created with vagrant]

youtube link

Test your Chef and Vagrant knowledge:

1. What is Vagrant?

2. Name 4 other tools like Chef.

3. Dev, Test, Prod all identical ?????

4. In Chef box as ‘run list' of ?????

5. Idempotent in Chef defined as ????

6. Extra credit: VirtualBox is to VM as Docker is to ?????

Chef and Vagrant Basic Answers

1. Vagrant is a command line front end for creating VMs

2. Chef as (1 of 5) Chef, Puppet, Ansible, Salt, CFEngine

3. Dev, Test, Prod all identical ‘code’

4. Box as ‘run list' of features or recipes

5. Idempotent creates or updates same code

6. Virtualbox is to VM as Docker is to container

Demo Metadata Collection

• Simple properties

• Collected using a cheesy UI

• UI and code generation bothwritten in Ruby

youtube link

Demo Generated Code

• Camel ETL binary

• OSGi, versioned, modular jar

• Only 3 primary outputs!

• simple

• clean

• well designed (?)

• JUnit/integration tested

• Supporting scripting

• messy

youtube link

Demo Server Deploy

• One line deploy/run command

• Compiles on server with Maven

• Also runnable as jar

youtube link

Does it work?

• Make custom file

• Drop into ETL folder

• Inspect

youtube link

Demo - Review

• Schema created

• DDL run

• Avro binary (JSON) transform

• Data Migration

• FTP to server

• Into HDFS partition

• Alter Table: Date Partition

youtube link

Takeaways

• Brent coding the exception manually, rule by template.

• Brent has time to focus on design & exceptions.

• Brent may lose some personal attention and status.

• Resulting code is

• clean

• consistent, easy to maintain

• But is there a Home Run?

• defined as anything not possible via special snowflake approach

Home Run 1: Instant, identical, dev/test/prod

Home Run 2: Big Data, Beyond Hadoop!

1. Pick your provider

• Hadoop

• Cassandra

• Couchbase

• any of hundreds…

2. Adopt your templates, VMs, etc

3. Even stick with Avro…

Home Run 3: Effort as Idempotent

• Idempotent effort? No penalty for discontinuous development.

• Walkup - The 10 minute test

• Walkaway - Requirements

• Features

• Testing, technical debt, already in place for code

• VMs and recipes for dev, test, prod

• OSGi etc modularity for binaries

• Does what we see here pass this test?

What to leave with

• De-mystify: how to Avro/Hadoop a delimited file

• Review motives for automating this process

• Code automation basics

• Infrastructure automation basics

• Code for above

Further Hadoop Tutuorial Resources

• Hortonworks

• best free stuff? Except networking vas

• Cloudera

• Lots but appear to prefer to get paid

• Apache Hadoop

• haven’t tried but it is Apache

Further Camel Resources

• Gerald Cantor of this group, Mark of this group (AMD)

• Camel In Action Book

• Camel mail list

• Red Hat support (Fuse)

6 week deep dives: Candidates• Apache Camel

• Serialization choices, formats, code, tools

• Big Data: NoSQL and newSQL variants and choices

• Test Driven Development

• Jenkins, CI, etc

• Hive, Impala, Hawq other Hadoop sql engines

• Pig, and MapReduce for Hadoop

• Hadoop clustering

• OSGi, Felix

• Maven, Gradle etc

• bash

• Chef or Puppet, Salt, Ansible, CFEngine

• Devops, Phoenix Project

Wish To See More?

• In office demos

• Your sample data

Code, Content, Contacts• This Slide Deck: http://www.slideshare.net/datafundamentals/hadoop-big-data-35762308

• or just remember slideshare.net/datafundamentals it may be the only one there

• Youtube - 11 minute slide-less version of code demo - https://www.youtube.com/playlist?list=PLO_T9AjxEaYeByfqBqHVCmg4GbLFkYCJe

• Dev Code

• Carrie (ruby UI and generator) https://github.com/datafundamentals/df_ui_carrie

• Avro from delimited https://bitbucket.org/datafundamentals/avro_from_delimited

• Camel-Avro https://bitbucket.org/datafundamentals/camel-avro-etl

• Ops Code - cookbook recipes

• https://github.com/datafundamentals

• Contact

• pete@datafundamentals.com, jeff@datafundamentals.com

Be careful! It’s a competitive world out there!