Tackling Big Data with Hadoop

TACKLING BIG DATA WITH HADOOP

David Howell

Sunday, September 11, 11

WHAT IS BIG DATA?


WHAT IS BIG DATA?Google web crawl


WHAT IS BIG DATA?stream of Twitter messages


WHAT IS BIG DATA?Annoying Farmville requests on Facebook


WHAT IS BIG DATA?terabyte-scale data sets

awkward to work with using traditional tools


WHAT IS BIG DATA?requires distributed computing


MEDIUM DATAdozens to hundreds of gigabytes

still awkward to work with using traditional tools


MAP-REDUCEhttp://labs.google.com/papers/mapreduce.html


http://labs.google.com/papers/mapreduce.html

http://labs.google.com/papers/mapreduce.html

COUNTING AT SCALE


function map_1(t, search_phrase)emit(search_phrase, 1)

function reduce_1(search_phrase, counts)total = 0for count in countstotal += count

emit(search_phrase, total)

function map_2(search_phrase, total)emit(total, search_phrase)

function reduce_2(total, search_phrases)for search_phrase in search_phrasesemit(search_phrase, total)

sort and shuffle

sort and shuffle


cat IN | sort | uniq -c > OUTmap shuffle reduce

awk ‘{print $2,$1}’ OUT | sort > FINAL map shuffle reduce


WHY BOTHER?


HADOOP


DISTRIBUTED COMPUTING PLATFORM


TOOLS IN THE PLATFORM

Higher Level APIs•Hive•Cascading•Pig

Map-Reduce APIs•Java•C++•UNIX pipes


THE ORIGIN STORY


WHO’S USING IT?


HADOOPHow does it work?


DEMO!


YOUR DATA PLATFORM

ad hocunstructuredprototypingexperimentdata-driven

curiosityplay


LEARN MORE

http://hadoop.apache.org/http://www.cloudera.com/

Hadoop: The Definitive Guide

@[email protected]

http://github.com/dehowell/hadoop-crypto-demo


mailto:[email protected]

mailto:[email protected]



Tackling Big Data with Hadoop

Technology

Transcript of Tackling Big Data with Hadoop