Managing terabytes on aws

Post on 19-May-2015

644 views 0 download

Tags:

description

JFrog is handling huge amount of binaries files for all our customers. Since each customer has its own space and domain, using a global Object Store can be tricky. Also our application cannot work with an “eventually consistent” storage, and cannot deliver customer requirements with current S3 performance. Learn in this session how we managed fast upload, critical replication and backups, and global download availability of the terabytes of JFrog customer binaries files.

Transcript of Managing terabytes on aws

Request

Storage File Listing Checksum Storage

Request

Storage File Listing Checksum Storage

Metadata requests

Cache check, FS request

DB request

Request

Storage File Listing Checksum Storage

Metadata requests

Cache check, FS request

DB request

HEAD requests Cache check, FS request

DB request

Request

Storage File Listing Checksum Storage

Metadata requests

Cache check, FS request

DB request

HEAD requests Cache check, FS request

DB request

GET request Cache check, FS request

DB + FS request

Request

Storage File Listing Checksum Storage

Metadata requests

Cache check, FS request

DB request

HEAD requests Cache check, FS request

DB request

GET request Cache check, FS request

DB + FS request

PUT request FS request DB + FS request

EBS + LVM

= WIN!

/data

250GB 250GB

/data

250GB 250GB 250GB 250GB

/data

250GB 250GB 250GB 250GB 250GB 250GB

Backup files

Backup files

Backup database

Backup files

Backup database

Keep them in sync!

Backup files

Backup database

Keep them in sync!

Backup only deltas

Backup only deltas

Manage threads to

ec2 efficiently

Backup only deltas

Manage threads to

ec2 efficiently

Keep the data

isolated

Backup only deltas

Manage threads to

ec2 efficiently

Keep the data

isolated

Customer EBS

Customer EBS

Customer EBS

De-duped

EBS

HA Storage

Customer EBS

Customer EBS

Customer EBS

De-duped Glacier

De-duped

EBS Backup

HA Storage

Customer EBS

Customer EBS

Customer EBS

Create empty Artifactory instance

Create empty Artifactory instance

Populate from back

storage on demand

Create empty Artifactory instance

Populate from back

storage on demand

Drink

Cosmopolitan

Conclusions time

1. Be lazy.

Conclusions time

1. Be lazy.

2. Be smart.

Conclusions time

1. Be lazy.

2. Be smart.

3. Be agile.

Drink

Cosmopolitan