HDFS & MapReduce

Let us change our traditional attitude to the construction of programs: Instead of imagining that

our main task is to instruct a computer what to do, let us concentrate rather on explaining to humans what

we want the computer to do

Donald E. Knuth, Literate Programming, 1984

Drivers

2

Central activity

3

Dominant logics

4

Economy Subsistence Agricultural Industrial Service Sustainable

Question How to survive? How to farm?How to manage resources?

How to create customers?

How to reduce impact?

Dominant issue

Survival

Production

Customer service

Sustainability

Key information systems

GestureSpeech

WritingCalendar

AccountingERPProject management

CRMAnalytics

SimulationOptimizationDesign

Data sources

5

Operational

6

Social

7

Environmental

8

Digital transformation

9

Data

Data are the raw material for informationIdeally, the lower the level of detail the better

Summarize up but not detail down

Immutability means no updatingAppend plus a time stamp

Maintain history

10

Data types

StructuredUnstructured

Can structure with some effort

11

Requirements for Big Data

Robust and fault-tolerantLow latency reads and updatesScalableSupport a wide variety of applicationsExtensibleAd hoc queriesMinimal maintenanceDebuggable 12

Bottlenecks

13

Solving the speed problem

14

Lambda architecture

Speed layer

Serving layer

Batch layer

15

Batch layer

Addresses the cost problemThe batch layer stores the master copy of the dataset• A very large list of records• An immutable growing dataset

Continually pre-computes batch views on that master dataset so they available when requested

Might take several hours to run

16

Batch programmingAutomatically parallelized across a cluster of machines

Supports scalability to any size dataset

If you have an x nodes cluster, the computation will be about x times faster compared to a single machine

17

Serving layer

A specialized distributed databaseIndexes pre-computed batch views and loads them so they can be efficiently queriedContinuously swaps in newer pre-computed versions of batch views

18

Serving layer

Simple databaseBatch updatesRandom readsNo random writes

Low complexityRobustPredictableEasy to configure and manage

19

Speed layer

The only data not represented in a batch view are those data collected while the pre-computation was runningThe speed layer is a real-time system to top-up the analysis with the latest data

Does incremental updates based on recent data Modifies the view as data are collectedMerges the two views as required by queries

20

Lambda architecture

21

Speed layer

Intermediate results are discarded every time a new batch view is receivedThe complexity of the speed layer is “isolated”

If anything goes wrong, the results are only a few hours out-of-date and fixed when the next batch update is received

22

Lambda architecture

23

Lambda architectureNew data are sent to the batch and speed layers

New data are appended to the master dataset to preserve immutability

Speed layer does an incremental update

24

Lambda architectureBatch layer pre-computes using all dataServing layer indexes batch created views

Prepares for rapid response to queries

25

Lambda architectureQueries are handled by merging data from the serving and speed layers

26

Master dataset

Goal is to preserve integrityOther elements can be recomputed

Replication across nodesRedundancy is integrity

27

CRUD to CRCreateReadUpdateDelete

CreateRead

28

Immutability exceptions

Garbage collectionDelete elements of low potential value• Don’t keep some histories

Regulations and privacyDelete elements that are not permitted• History of books borrowed

29

Fact-based data modelEach fact is a single piece of data

Clare is femaleClare works at BloomingdalesClare lives in New York

Multi-valued facts need to be decomposed

Clare is a female working at Bloomingdales in New York

A fact is data about an entity or a relationship between two entities

30

Fact-based data modelEach fact has an associated timestamp recording the earliest time that the fact is believed to be true

For convenience, usually the time the fact is capturedCreate a new data type of time series or attributes become entities

More recent facts override older factsAll facts need to be uniquely identified

Often a timestamp plus other attributesUse a 64 bit nonce (number used once) field, which is a a random number, if timestamp plus attribute combination could be identical

31

Fact-based versus relational

Decision-making effectiveness versus operational efficiency

Days versus seconds

Access many records versus access a fewImmutable versus mutable

History versus current view

32

Schemas

Schemas increase data quality by defining structureCatch errors at creation time when they are easier and cheaper to correct

33

Fact-based data model

Graphs can represent facts-based data models

Nodes are entitiesProperties are attributes of entitiesEdges are relationships between entities

34

Graph versus relational

Keep a full historyAppend onlyScalable?

35

Solving the speed and cost problems

36

Hadoop

Distributed file systemHadoop distributed file system (HDFS)

Distributed computationMapReduce

Commodity hardwareA cluster of nodes

37

Hadoop

Yahoo! uses Hadoop for data analytics, machine learning, search ranking, email anti-spam, ad optimization, ETL, and moreOver 40,000 servers170 PB of storage

38

Hadoop

Lower costCommodity hardware

SpeedMultiple processors

39

HDFS

Files are broken into fixed sized blocks of at least 64MBBlocks are replicated across nodes

Parallel processingFault tolerance

40

HDFS

Node storageStore blocks sequentially to minimize disk head movementBlocks are grouped into filesAll files for a dataset are grouped into a single folderNo random access to recordsNew data are added as a new file

41

HDFS

Scalable storageAdd nodesAppend new data as files

Scalable computationSupport of MapReduce

PartitioningGroup data into folders for processing at the folder level

42

Vertical partitioning

43

MapReduceA distributed computing method that provides primitives for scalable and fault-tolerant batch computationAd hoc queries on large datasets are time consuming

Distribute the computation across multiple processorsPre-compute common queries

Move the program to the data rather than the data to the program

44

MapReduce

45

MapReduce

46

MapReduce

InputDetermines how data are read by the mapperSplits up data for the mappers

MapOperates on each data set individually

PartitionDistributes key/value pairs to reducers

47

MapReduce

SortSorts input for the reducer

ReduceConsolidates key/value pairs

OutputWrites data to HDFS

48

Shuffle

49

Programming MapReduce

Map

A Map function converts each input element into zero or more key-value pairsA “key” is not unique, and many pairs with the same key are typically generated by the Map functionThe key is the field about which you want to collect data

51

Map

Compute the square of set of numbers

Input is (null,1), (null,2), …Output is (1,1), (2,4), …

52

mapper <- function(k,v) { key <- v value <- key^2 keyval(key,value)}

Reduce

A Reduce function is applied, for each input key, to its associated list of values The result is a new pair consisting of the key and whatever is produced by the Reduce functionThe output of the MapReduce is what results from the application of the Reduce function to each key and its list 53

Reduce

Report the number of items in a list

Input is (key, value-list), … Output is (key, length(value-list)), …

54

reducer <- function(k,v) { key <- k value <- length(v) keyval(key,value)}

MapReduce API

A low-level Java implementationCan gain additional compute efficiency but tedious to programTry out highest-level options first and descend to lower levels if required

55

R & Hadoop

Compute squares

56

R

# create a list of 10 integersints <- 1:10# equivalent to ints <- c(1,2,3,4,5,6,7,8,9,10)# compute the squaresresult <- sapply(ints,function(x) x^2)result[1] 1 4 9 16 25 36 49 64 81 100

Key-value mapping

58

Input Map   Reduce Output

(null,1) (1,1)     (1,1)

(null,2) (2,4)     (2,4)

… …     …

(null,10) (10,100)     (10,100)

MapReduce

MapReduce library(rmr2)rmr.options(backend = "local") # local or hadoop# load a list of 10 integers into HDFS hdfs.ints = to.dfs(1:10)# mapper for the key-value pairs to compute squaresmapper <- function(k,v) { key <- v value <- key^2 keyval(key,value)}# run MapReduce out = mapreduce(input = hdfs.ints, map = mapper)# convert to a data framedf1 = as.data.frame(from.dfs(out))colnames(df1) = c('n', 'n^2')#display the resultsdf1

No reduce

Exercise

Use the map component of the mapreduce() to create the cubes of the integers from 1 to 25

60

R & Hadoop

Tabulation

R

library(readr)url <- "http://people.terry.uga.edu/rwatson/data/centralparktemps.txt"t <- read_delim(url, delim=',')#convert and round temperature to an integert$temperature = round((t$temperature-32)*5/9,0)# tabulate frequenciestable(t$temperature)

Key-value mapping

63

Input Map (F to C)

  Reduce Output

(null,35.1) (2,1)   (-7,c(1)) (-7,1)

(null,37.5) (3,1)   (-6,c(1)) (-6,1)

… …   … …

(null,43.3) (6,1)   (27,c(1,1,1,1,1,1,1,1))

(27,8)

MapReduce (1)

MapReduce library(rmr2)library(readr)rmr.options(backend = "local") #local or hadoopurl <- "http://people.terry.uga.edu/rwatson/data/centralparktemps.txt"t <- read_delim(url, delim=',')# save temperature in hdfs filehdfs.temp <- to.dfs(t$temperature)# mapper for conversion to Cmapper <- function(k,v) { key <- round((v-32)*5/9,0) value <- 1 keyval(key,value)}

MapReduce (2)

MapReduce # reducer to count frequenciesreducer <- function(k,v) { key <- k value = length(v) keyval(key,value)}out = mapreduce( input = hdfs.temp, map = mapper, reduce = reducer)df2 = as.data.frame(from.dfs(out))colnames(df2) = c('temperature', 'count')df3 <- df2[order(df2$temperature),]print(df3, row.names = FALSE) # no row names

R & Hadoop

Basic statistics

Rlibrary(readr)url <- "http://people.terry.uga.edu/rwatson/data/centralparktemps.txt"t <- read_delim(url, delim=',')a1 <- aggregate(t$temperature,by=list(t$year),FUN=max)colnames(a1) = c('year', 'value')a1$measure = 'max'a2 <- aggregate(t$temperature,by=list(t$year),FUN=mean)colnames(a2) = c('year', 'value')a2$value = round(a2$value,1)a2$measure = 'mean'a3 <- aggregate(t$temperature,by=list(t$year),FUN=min)colnames(a3) = c('year', 'value')a3$measure = 'min'# stack the resultsstack <- rbind(a1,a2,a3) library(reshape)# reshape with year, max, mean, min in one rowstats <- cast(stack,year ~ measure,value="value")head(stats)

Key-value mapping

68

Input Map   Reduce Output

(null,record) (year, temperature)

   (year, vector of temperatures)

(year, max)(year, mean)(year, min)

MapReduce (1)

MapReduce library(rmr2)library(reshape)library(readr)rmr.options(backend = "local") # local or hadoopurl <- "http://people.terry.uga.edu/rwatson/data/centralparktemps.txt"t <- read_delim(url, delim=',')# convert to hdfs filehdfs.temp <- to.dfs(data.frame(t))# mapper for computing temperature measures for each yearmapper <- function(k,v) { key <- v$year value <- v$temperature keyval(key,value)}

MapReduce (2)

MapReduce #reducer to report statsreducer <- function(k,v) { key <- k #year value <- c(max(v),round(mean(v),1),min(v)) #v is list of values for a year keyval(key,value)}out = mapreduce( input = hdfs.temp, map = mapper, reduce = reducer)df3 = as.data.frame(from.dfs(out))df3$measure <- c('max','mean','min')# reshape with year, max, mean, min in one rowstats2 <- cast(df3,key ~ measure,value="val")head(stats2)

R & Hadoop

Word counting

R

library(stringr)# read as a single character stringt <- readChar("http://people.terry.uga.edu/rwatson/data/yogiquotes.txt", nchars=1e6)t1 <- tolower(t[[1]]) # convert to lower caset2 <- str_replace_all(t1,"[[:punct:]]","") # get rid of punctuationwordList <- str_split(t2, "\\s") #split into stringswordVector <- unlist(wordList) # convert list to vectortable(wordVector)

Key-value mapping

73

Input Map   Reduce Output

(null, text)

(word,1)(word,1)…

   (word, vector)…

word, length(vector)…

MapReduce (1)

MapReduce library(rmr2)library(stringr)rmr.options(backend = "local") # local or hadoop# read as a single character stringurl <- "http://people.terry.uga.edu/rwatson/data/yogiquotes.txt"t <- readChar(url, nchars=1e6)text.hdfs <- to.dfs(t)mapper=function(k,v){ t1 <- tolower(v) # convert to lower case t2 <- str_replace_all(t1,"[[:punct:]]","") # get rid of punctuation wordList <- str_split(t2, "\\s") #split into words wordVector <- unlist(wordList) # convert list to vector keyval(wordVector,1)}

MapReduce (2)

MapReduce reducer = function(k,v) {keyval(k,length(v))}out <- mapreduce (input = text.hdfs,map = mapper,reduce = reducer,combine=T)# convert output to a framedf1 = as.data.frame(from.dfs(out))colnames(df1) = c('word', 'count')#display the resultsprint(df1, row.names = FALSE) # no row names

Hortonworks data platform

76

HBase

A distributed databaseDoes not enforce relationshipsDoes not enforce strict column data typingPart of the Hadoop ecosytem

77

Applications

FacebookTwitterStumbleUpon

78

Hiring: learning from big data

People with a criminal background perform a bit better in customer-support call centersCustomer-service employees who live nearby are less likely to leaveHonest people tend to perform better and stay on the job longer but make less effective salespeople

79

Outcomes

Scientific discoveryQuasarsHiggs Boson

Discovering linkages among humans, products, and servicesAn ecological sustainable society

Energy Informatics

80

Critical questions

What’s the business problem?What information is needed to make a high quality decision?What data can be converted into information?

81

ConclusionsFaster and lower cost solutions for data-driven decision makingHDFS

Reduces the cost of storing large data setsBecoming the new standard for data storage

MapReduce is changing the way data are processed

CheaperFasterNeed to reprogram for parallelism

82