Big Data Universe - How we design architectures
-
Upload
mate-gulyas -
Category
Software
-
view
265 -
download
1
Transcript of Big Data Universe - How we design architectures
![Page 1: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/1.jpg)
How we design data architectureMate Gulyas
![Page 2: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/2.jpg)
CTO & Co-FounderGULYÁS MÁTÉ
@gulyasm
![Page 3: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/3.jpg)
ARCHITECTURE?●CODE ARCHITECTURE●GENERAL
INFRASTRUCTURE●DATA INFRASTRUCTURE
@gulyasm
![Page 4: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/4.jpg)
ON THE NEXT EPISODE OF BIG DATA...
1.WHAT DO WE DESIGN FOR?
2.OUR STORY, OUR FAILURES
@gulyasm
![Page 5: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/5.jpg)
WHAT DO WE DESIGN FOR?
![Page 6: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/6.jpg)
WHAT DO WE DESIGN FOR?
●SCALABILITY●MAINTAINABILITY●COST
@gulyasm
![Page 7: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/7.jpg)
SCALABILITY AND MAINTAINABILITY
ARE RESULTS OF A GOOD DESIGN
![Page 8: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/8.jpg)
WHAT DO WE REALLY DESIGN FOR?
●SIMPLICITY
●RESILIENCY
●SMALL ITERATIONS
●SELF SERVICE
@gulyasm
![Page 9: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/9.jpg)
WHAT DO WE REALLY DESIGN FOR?
●SIMPLICITY
●RESILIENCY
●SMALL ITERATIONS
●SELF SERVICE
@gulyasm
![Page 10: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/10.jpg)
SIMPLICITY
SIMPLE THINGS
SCALE WELL
@gulyasm
![Page 11: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/11.jpg)
SIMPLICITY
SIMPLE THINGS ARE EASY TO UNDERSTAND
@gulyasm
![Page 12: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/12.jpg)
SIMPLICITY
BORING TECHNOLOGY IS GOOD TECHNOLOGY
@gulyasm
![Page 13: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/13.jpg)
SMALL ITERATIONS
THE UNKNOWNS● THE UNKNOWNS
● THE UNKNOWN UNKNOWNS
@gulyasm
![Page 14: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/14.jpg)
SMALL ITERATIONS @gulyasm
![Page 15: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/15.jpg)
END RESULT @gulyasm
![Page 16: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/16.jpg)
SMALL ITERATIONS @gulyasm
![Page 17: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/17.jpg)
SMALL ITERATIONS @gulyasm
![Page 18: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/18.jpg)
SMALL ITERATIONS @gulyasm
![Page 19: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/19.jpg)
SMALL ITERATIONS @gulyasm
![Page 20: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/20.jpg)
SMALL ITERATIONS @gulyasm
![Page 21: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/21.jpg)
SMALL ITERATIONS @gulyasm
![Page 22: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/22.jpg)
SELF SERVICE
YOUR SOFTWARE/IT INFRASTRUCTURE
IMPACTS THE WHOLE ORGANIZATION
![Page 23: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/23.jpg)
![Page 24: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/24.jpg)
![Page 25: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/25.jpg)
ENBRITELY
DATA
PLATFORM
![Page 26: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/26.jpg)
Product placeholder
![Page 27: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/27.jpg)
Luigi TOOLS
Luigi + enbrite.ly extensions = Gabo Luigi
WORKFLOW ENGINE
![Page 28: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/28.jpg)
Tools we created
GABO LUIGI
![Page 29: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/29.jpg)
Spark TOOLS
0.5-4TB daily data1-10B events
Ad-hoc batch queries: 20TB data
![Page 30: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/30.jpg)
Spark TOOLS
●SPENT 3 MONTHS OPTIMIZING IT
●20+ NODE CLUSTERS●UNIT TESTS
![Page 31: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/31.jpg)
AWS TOOLS
●16 services●110+ machines●1-4 EMR clusters (1-20 node)●100TB+ on S3●All clients has separate
infrastructure
![Page 32: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/32.jpg)
![Page 33: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/33.jpg)
HOW WE GOT HERE?
2014
MONOLITHIC PYTHON ANALYTICS
2015
JAN
EVALUATE BIG DATA TECHNOLOGIES
2015
SEP
STARTED WORK ON DP
2016FEB
DPPRODUCTION READY
2016JULSAAS DP
@gulyasm
![Page 34: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/34.jpg)
HAVE FUN!
@gulyasm
![Page 35: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/35.jpg)
PRACTICE AT HOME
@gulyasm
![Page 36: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/36.jpg)
WE ARE HIRING!
![Page 37: Big Data Universe - How we design architectures](https://reader031.fdocuments.net/reader031/viewer/2022030304/5876f1891a28abf3398b493d/html5/thumbnails/37.jpg)
WE ARE HIRING!