Big Data - Hadoop and MapReduce - Aditya Garg

Confidential | Copyright © QAAgility Technologies

Big Data - Hadoop and MapReduce - new age tools for aid to testing and

QAby Aditya Garg

Big Data - Hadoop and MapReduce - new age tools

for aid to testing and QA

Topic for the presentation

What is this

Confidential | Copyright © QA Agility Technologies

1. How to test Big Data applications ?

2. How can QA and Testing team use Big Data tools for their testing needs ?

What are we going to discuss ?


What is Big Data ?

Is it just too much Hype or reality ?

Here is latest one from yesterday on #Bigdata


Let us start with what exactly is BigData

Which Search Engine do you use ?

https://www.cirrusinsight.com/blog/how-much-data-does-google-store

http:

//se

arch

stor

age.

tech

targ

et.c

om/

defin

ition

/Kilo

-meg

a-gi

ga-te

ra-p

eta-

and-

all-

that

How much data does Google store ?

Key Points in Big Data

1.Volume – Data Explosion

2.Velocity3.Variety4.Veracity

Ref: IBM.com

Key Points in Big Data

Definition

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization.

http://www.forbes.com/sites/gilpress/2014/09/03/12-big-data-definitions-whats-yours/#379879e621a9

Ref: goo.gl/iWZhjJ

http://goo.gl/iWZhjJ

http://goo.gl/iWZhjJ

Big Data Application

1. Finance2. Insurance3. Health Care4. Agriculture5. Defense6. Manufacturing7. Aero Space8. Oil and Gas9. Advertisement and Marketing10.Election Campaigns11. List goes on --- applicability across industries

http://snip.ly/UKNB#http://bit.ly/1OF5nhF


http://www.forbes.com/sites/bernardmarr/2016/02/03/how-the-super-bowl-uses-big-data-to-change-the-game/?


http://andrewshamlet.com/2015/12/03/who-will-win-the-2016-us-presidential-nominations/

Lets go back to definition

Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization.


Tools solving Big Data Challenge

Tool solving the Big Data Challenge

*Source Udacity

Hadoop – Key components HDFS and MR

*Source Udacity

1. Sqoop takes data from regular RDBMS and puts it into HDFS

2. Flume ingests data into HDFS as it is generated by external systems

3. HBASE is real time database on top of HDFS

4. Hue is a graphical front end to the cluster

5. Oozie is workflow management tool

6. Mahout is Machine Learning library

Hadoop Ecosystem

HDFS

• HDFS stands for Hadoop Distributed File System, which is the storage system used by Hadoop. The following is a high-level architecture that explains how HDFS works.

Map Reduce

Ref: Emanuele Della Valle@manudellavalle

https://twitter.com/manudellavalle






Understanding MapReduce

Demo – Word Count

Given an input file, count unique words

WordCount – Map Reduce

Reference : http://wearecloud.cz/media/files/prezentace-biz/Big%20Data%20v%20Cloudu.ppt


How can QA and Testing team use Big Data tools for their testing needs ?


Problem Statement and Solution using Hadoop

and MapReduce

MTBT – Multicast Tick by Tick Adapter

Input was exchange feed – Output given to HFT Engine

Exchange TAP – Co-location servers listen to it at high speed

Legacy Adaptor (3rd Party) connects to the TAP – and converts to a format which can be used by HFT Platforms (Algorithmic Trading Platforms)

New Adaptor – being made Inhouse – to increase the

speed by 10 Times

HFT Engine

MTBT - Adaptor

MTBT – Multicast Tick by Tick Adapter

•Client was trying to build a brand new MTBT Exchange Adaptor•The adaptor was being developed in C and Unix and was to run in a co-location with NSE (National Stock Exchange)•The new adaptor was supposed to increase the overall speed by more than 10 times from the existing adaptor•The Goal was to test the new adaptor

LEGACY

INHOUSE (NEW)

Input OutputOutput over time

MTBT - Adaptor

Sample

Sample

Sample

Sample

Sample

Do A Reverse Comparison

MTBT – Testing Strategy - Sampling

LEGACY

INHOUSE (NEW)


MTBT - Adaptor Challenges--------------------------------------------------1. Manually next to impossible2. Even few seconds samples were

running into large MegaBytes (MB) files

3. Manually impossible to compare the legacy records with the New code processed records

4. Daily processed data ran into 150 Giga Bytes (GB) plus files

MTBT – Challenges

LEGACY

INHOUSE (NEW)


MTBT - Adaptor BIG DATA Problem--------------------------------------------------1. LARGE 150 GB files (legacy and New

applications) – VOLUME

2. Testing to compare the output and measure the functional effectiveness in real time data environment – VELOCITY

3. Packet drops may happen – (VERACITY)

4. Variety was not there – except the format of the output file generated was not in similar format – the content/information was there

MTBT – It was a BIG DATA Testing problem

MTBT – SOLUTION

1 Reduce LEGACY MTBT - Output file into a standard format

2 Reduce NEW INHOUSE MTBT output file into a standard format

3 Compare the two files

4 Generate Report


QA team can use the tools in multiple scenarios1. Beta Testing2. Repeated execution effectiveness –

applying analytics ( R)3. Capturing Customer feedback and

channeling the same for smarter test execution

4. Extracting relevant information from repeated regression cycles from QC

5. Adding intelligence on the data generated by the testing team

Other scenarios – Big Data Tool implementation

Thank you and Jai Hind

Questions ?@adigIndia@AgileTA#GTR2016

ContactPlease contact us at

[email protected]

Confidential | Copyright © QAAgility Technologies

MUMBAI711, Rupa SolitaireMBP, MahapeNavi Mumbai-400701

DENMARK1 Lindebo 7 Lej - 42,2630 Tasstrup, [email protected]

USA 200 E Campus View Blvd.Suite 200, Columbus, OH

Big Data - Hadoop and MapReduce - Aditya Garg

Technology

Transcript of Big Data - Hadoop and MapReduce - Aditya Garg