Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large...
Transcript of Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large...
![Page 1: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/1.jpg)
Cyber Analytics Service Constraints and Solutions
Tristan Vanderbruggen
CISC850Cyber Analytics
![Page 2: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/2.jpg)
Range of Internet Services
CISC850Cyber Analytics
2
MonolithicUntil early 2000s
Scaling: Larger computerNot reliable
Weekly update => Debugging Hell
Increased Traffic
Micro-servicesServerless
ScalesLow down time
Emergence
![Page 3: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/3.jpg)
What the really large players do:
CISC850Cyber Analytics
Mastering Chaos - A Netflix
Guide to Microservices
What I Wish I Had Known Before
Scaling Uber to 1000 Services3
![Page 4: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/4.jpg)
Middle Ground Solution
CISC850Cyber Analytics
Web-basedUser Interface
Edge Services(dispatch)
Platform(compute intensive)
Persistence Layer
4
![Page 5: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/5.jpg)
WebUI
CISC850Cyber Analytics
● Static: ○ HTML○ JavaScript○ CSS
● Content:○ REST API: edge services○ Media: persistence layer
● Short lifecycle
5
![Page 6: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/6.jpg)
Edge
CISC850Cyber Analytics
6
● Implement: transaction logic○ REST API
● Micro-services○ Serverless: AWS Lambda○ Lightweight: AWS Elastic Beanstalk
■ WSGI application (Flask)● Short Lifecycle
![Page 7: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/7.jpg)
Platform
CISC850Cyber Analytics
7
● Compute hungry○ Actual application○ Independent tasks○ Embarrassingly parallel
● Somewhat monolithic○ Large code base○ Many dependencies
● Long lifecycle
![Page 8: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/8.jpg)
Persistence Layer
CISC850Cyber Analytics
8
● Your data!○ Relational Database○ NoSQL Database
■ Key-value stores● Storage: AWS S3● Database: AWS DynamDB
Introduction to NoSQL - Martin Fowler
![Page 9: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/9.jpg)
Middle Ground Solution
CISC850Cyber Analytics
Web-basedUser Interface
Edge Services(dispatch)
Platform(compute intensive)
Persistence Layer
9
![Page 10: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/10.jpg)
Cyber 20/20 Analytics Service
CISC850Cyber Analytics
FCAS Frontend(Electron)
FCAS Backend(EBS + Flask)
AMP(homemade)
DynamoDB + S3
10
MySQL
![Page 11: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/11.jpg)
11
![Page 12: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/12.jpg)
12
![Page 13: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/13.jpg)
13
![Page 14: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/14.jpg)
14
![Page 15: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/15.jpg)
File Capture and Analysis Service
CISC850Cyber Analytics
15
● Tightly coupled frontend and backend○ Web-based UI: Electron
■ Presents analysis and prediction results■ Use D3 to provide visual insights
○ REST server: Flask + MySQL■ Dispatch analysis and prediction workload■ Gather results in relational DB
![Page 16: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/16.jpg)
FCAS Frontend
CISC850Cyber Analytics
16
● Constraints
○ Controlled Environment
○ Visually Appealing
● Solutions
○ Electron
○ D3js
![Page 17: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/17.jpg)
17
![Page 18: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/18.jpg)
FCAS Backend
CISC850Cyber Analytics
18
● Constraints
○ Deploy and Scale
○ Complex Queries
○ Short Lifecycle
● Solutions
○ Elastic Beanstalk
○ Relational Database
○ Python + Flask
![Page 19: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/19.jpg)
19
![Page 20: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/20.jpg)
Analysis and Machine Learning Platform
CISC850Cyber Analytics
20
● Analyses files○ Basic: crypto hash, strings, PE, …○ Bytes-Entropy Histograms○ Reverse Engineering with Radare2
● Make predictions○ DNN applied to various analysis results○ DNN ensemble for consensus
=> Lots of dependencies <=
![Page 21: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/21.jpg)
Analysis and Machine Learning Platform
CISC850Cyber Analytics
21
● Analysis Tools○ Independent○ Lots of dependencies (Radare2, ssdeep, pefile, scipy, ...)
● Machine Learning (Theano + Scikit Learn)○ Handle big data (training)○ Fast inception (predictions)
● Glue code○ receive workload○ dispatch to subprocesses
![Page 22: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/22.jpg)
Analysis and Machine Learning Platform
CISC850Cyber Analytics
22
● Constraints
○ Highly scalable
○ Cheap
○ Reliable
○ Low latency
● Solutions
○ ASG + S3 + DynamoDB
○ SPOT instances
○ Simple Queue Service
○ Hard work !!!
![Page 23: Cyber Analytics Service Constraints and Solutionscavazos/cisc850-spring... · What the really large players do: CISC850 Cyber Analytics Mastering Chaos - A Netflix Guide to Microservices](https://reader035.fdocuments.net/reader035/viewer/2022062916/5ec466257cd94d146313fb6f/html5/thumbnails/23.jpg)
23