AWS January 2016 Webinar Series - Managing your Infrastructure as Code
Managing terabytes on aws
81
-
Upload
baruch-sadogursky -
Category
Technology
-
view
644 -
download
0
description
JFrog is handling huge amount of binaries files for all our customers. Since each customer has its own space and domain, using a global Object Store can be tricky. Also our application cannot work with an “eventually consistent” storage, and cannot deliver customer requirements with current S3 performance. Learn in this session how we managed fast upload, critical replication and backups, and global download availability of the terabytes of JFrog customer binaries files.
Transcript of Managing terabytes on aws
Request
Storage File Listing Checksum Storage
Request
Storage File Listing Checksum Storage
Metadata requests
Cache check, FS request
DB request
Request
Storage File Listing Checksum Storage
Metadata requests
Cache check, FS request
DB request
HEAD requests Cache check, FS request
DB request
Request
Storage File Listing Checksum Storage
Metadata requests
Cache check, FS request
DB request
HEAD requests Cache check, FS request
DB request
GET request Cache check, FS request
DB + FS request
Request
Storage File Listing Checksum Storage
Metadata requests
Cache check, FS request
DB request
HEAD requests Cache check, FS request
DB request
GET request Cache check, FS request
DB + FS request
PUT request FS request DB + FS request
EBS + LVM
= WIN!
/data
250GB 250GB
/data
250GB 250GB 250GB 250GB
/data
250GB 250GB 250GB 250GB 250GB 250GB
Backup files
Backup files
Backup database
Backup files
Backup database
Keep them in sync!
Backup files
Backup database
Keep them in sync!
Backup only deltas
Backup only deltas
Manage threads to
ec2 efficiently
Backup only deltas
Manage threads to
ec2 efficiently
Keep the data
isolated
Backup only deltas
Manage threads to
ec2 efficiently
Keep the data
isolated
Customer EBS
Customer EBS
Customer EBS
De-duped
EBS
HA Storage
Customer EBS
Customer EBS
Customer EBS
De-duped Glacier
De-duped
EBS Backup
HA Storage
Customer EBS
Customer EBS
Customer EBS
Create empty Artifactory instance
Create empty Artifactory instance
Populate from back
storage on demand
Create empty Artifactory instance
Populate from back
storage on demand
Drink
Cosmopolitan
Conclusions time
1. Be lazy.
Conclusions time
1. Be lazy.
2. Be smart.
Conclusions time
1. Be lazy.
2. Be smart.
3. Be agile.
Drink
Cosmopolitan
![JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]](https://static.fdocuments.net/doc/165x107/58a359261a28ab14598b5ed5/javaone2016-microservices-terabytes-in-microseconds-con4516.jpg)
