Analyzing Yellowstone’s Network with a Raspberry Pi Cluster
description
Transcript of Analyzing Yellowstone’s Network with a Raspberry Pi Cluster
![Page 1: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/1.jpg)
Analyzing Yellowstone’s Network with a Raspberry Pi Cluster
Lauren Patterson
![Page 2: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/2.jpg)
Using a low cost Raspberry Pi cluster to find the interconnect path between two nodes on Yellowstone in order to analyze the performance of jobs.
Objective of the Project
![Page 3: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/3.jpg)
![Page 4: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/4.jpg)
Assembling the Raspberry Pi cluster
![Page 5: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/5.jpg)
![Page 6: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/6.jpg)
Yellowstone Interconnect
Credit: Siddhartha Ghosh
![Page 7: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/7.jpg)
Files Used
• job1_nodes.txt– Gives the job ID and nodes used
• ibnetdiscover.log (Discover File)– Lists connections between switches
• LFTS.txt– Routing table for each switch
![Page 8: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/8.jpg)
What is Hadoop?• HDFS • MapReduce
![Page 9: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/9.jpg)
HDFSName Node
Job TrackerTask TrackerMap/Reduce
Data Node
Task TrackerMap/Reduce
Data NodeTask TrackerMap/Reduce
Data NodeTask TrackerMap/Reduce
![Page 10: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/10.jpg)
InputData
Map Phase
Shuffle phase
Reduce phase
OutputData
MapReduce
![Page 11: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/11.jpg)
Pig• Apache Pig • Pig Latin • Grunt
![Page 12: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/12.jpg)
Pig Latin Script• Created Pig Latin Script to find the path
between two nodes in Yellowstone
http://www.edureka.in/blog/pig-programming-create-your-first-apache-pig-script/
![Page 13: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/13.jpg)
JOIN Operations in PIG Default, Inner Join returns intersection of A and B
Set BSet A A BU
Full, Right and Left Outer Joins return A and B with different parts nulled out (white)
Full
Right
Left
Join
![Page 14: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/14.jpg)
Path Finder Code Flow
![Page 15: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/15.jpg)
Six Hops (seconds) Four Hops (seconds) Two Hops (seconds)
Raspberry Pi Cluster
3216 2514 1584
Virtual Box 450 351 214
250
750
1250
1750
2250
2750
3250
Pig (Single Path)
Seco
nds
Results
±3
±82 ±19 ±15
±3±4
![Page 16: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/16.jpg)
Python• Single Path Python• Parallel Python– Mpi4py 1.3.1
![Page 17: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/17.jpg)
Six Hop (seconds) Four Hop (seconds) Two Hop (seconds)
Raspberry Pi 6.39 4.136 2.226
Virtual Box 0.273 0.174 0.099
0.5
1.5
2.5
3.5
4.5
5.5
6.5
Serial Python (Single Path)Se
cond
s
±0.02
±0.07
±0.006
±0.11
±0.004
±0.11
![Page 18: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/18.jpg)
16*15 Paths (sec-onds)
12*11 Paths (sec-onds)
8*7 Paths (seconds)
One Proces-sor
933 503 197
Two Proces-sors
493 275 108
Four Proces-sors
258 134 57
50250450650850
Parallel Python on Raspberry PiSe
cond
s
±18
±4
±20
±2±7 ±4
±1
±2 ±0.5
![Page 19: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/19.jpg)
What Do All Of These Have In Common?
• Raspberry Pi• Hadoop • Pig • Python
![Page 20: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/20.jpg)
Acknowledgments
Richard LoftKarina Hauser Stephanie BarrBruce Chittenden Amogh SimhaRaghu Raj Prasanna Kumar
![Page 21: Analyzing Yellowstone’s Network with a Raspberry Pi Cluster](https://reader038.fdocuments.net/reader038/viewer/2022110103/56814358550346895dafd5ca/html5/thumbnails/21.jpg)
Questions?