(PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014
-
Upload
amazon-web-services -
Category
Technology
-
view
6.535 -
download
3
description
Transcript of (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014
![Page 1: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/1.jpg)
![Page 2: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/2.jpg)
in data transfer from S3
not including Amazon Web Services use
![Page 3: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/3.jpg)
Architecture
Choosing a region
Building a naming scheme
Considering LISTs
Optimizing PUTs
Multipart upload
Demo
Optimizing GETs
Using CloudFront
Range-based GETs
Demo
Customer Case
BigData Corp
![Page 4: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/4.jpg)
![Page 5: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/5.jpg)
![Page 7: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/7.jpg)
1 2
58
100/8 = 12.5 events/sec
100,000 users @ 10 events an hour = 224 TPS
![Page 8: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/8.jpg)
<my_bucket>/2013_11_13-164533125.jpg<my_bucket>/2013_11_13-164533126.jpg<my_bucket>/2013_11_13-164533127.jpg<my_bucket>/2013_11_13-164533128.jpg<my_bucket>/2013_11_12-164533129.jpg<my_bucket>/2013_11_12-164533130.jpg<my_bucket>/2013_11_12-164533131.jpg<my_bucket>/2013_11_12-164533132.jpg<my_bucket>/2013_11_11-164533133.jpg<my_bucket>/2013_11_11-164533134.jpg<my_bucket>/2013_11_11-164533135.jpg<my_bucket>/2013_11_11-164533136.jpg
![Page 9: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/9.jpg)
1 2 N1 2 N
Partition Partition Partition Partition
![Page 10: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/10.jpg)
<my_bucket>/521335461-2013_11_13.jpg<my_bucket>/465330151-2013_11_13.jpg<my_bucket>/987331160-2013_11_13.jpg<my_bucket>/465765461-2013_11_13.jpg<my_bucket>/125631151-2013_11_13.jpg<my_bucket>/934563160-2013_11_13.jpg<my_bucket>/532132341-2013_11_13.jpg<my_bucket>/565437681-2013_11_13.jpg<my_bucket>/234567460-2013_11_13.jpg<my_bucket>/456767561-2013_11_13.jpg<my_bucket>/345565651-2013_11_13.jpg<my_bucket>/431345660-2013_11_13.jpg
![Page 11: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/11.jpg)
1 2 N1 2 N
Partition Partition Partition Partition
![Page 12: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/12.jpg)
• Store objects as a hash of their name– add the original name as metadata
• “deadmau5_mix.mp3” 0aa316fb000eae52921aab1b4697424958a53ad9
– prepend key name with short hash
• 0aa3-deadmau5_mix.mp3
• Epoch time (reverse)– 5321354831-deadmau5_mix.mp3
![Page 13: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/13.jpg)
![Page 14: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/14.jpg)
<my_bucket>/images/521335461-2013_11_13.jpg<my_bucket>/images/465330151-2013_11_13.jpg<my_bucket>/movies/293924440-2013_11_13.jpg<my_bucket>/movies/987331160-2013_11_13.jpg<my_bucket>/thumbs-small/838434842-2013_11_13.jpg<my_bucket>/thumbs-small/342532454-2013_11_13.jpg<my_bucket>/thumbs-small/345233453-2013_11_13.jpg<my_bucket>/thumbs-small/345453454-2013_11_13.jpg
![Page 16: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/16.jpg)
![Page 17: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/17.jpg)
![Page 18: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/18.jpg)
![Page 19: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/19.jpg)
faster flexible
set of parts
presents all parts as
a single object
parallel pausing resuming
beginning uploads before
you know the total object size
![Page 20: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/20.jpg)
![Page 21: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/21.jpg)
![Page 22: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/22.jpg)
DEMOMultipart Uploads
![Page 23: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/23.jpg)
![Page 24: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/24.jpg)
![Page 25: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/25.jpg)
![Page 26: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/26.jpg)
DEMOAmazon CloudFront vs. Amazon S3 download performance
![Page 27: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/27.jpg)
• Align your ranges with your parts!
![Page 28: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/28.jpg)
DEMORange based GETs
![Page 29: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/29.jpg)
![Page 30: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/30.jpg)
![Page 31: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/31.jpg)
DynamoDB Amazon RDS Amazon
CloudSearchAmazon EC2
![Page 32: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/32.jpg)
![Page 33: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/33.jpg)
![Page 34: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/34.jpg)
![Page 35: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/35.jpg)
![Page 36: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/36.jpg)
Maestro
(Reserved Instance)
List of crawl
URLs Main workers
Execute crawling
and process data
Spot Instances
Secondary workers
(queue listeners)
Reprocess data,
query additional
services, store
data on MongoDB
Spot Instances
Secondary
work queues –
processed data
MongoDB
cluster
Command and
Control Queue
![Page 37: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/37.jpg)
![Page 38: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/38.jpg)
Architecture
Choosing a region
Building a naming scheme
Considering LISTs
Optimizing PUTs
Multipart upload
Demo
Optimizing GETs
Using CloudFront
Range-based GETs
Demo
Customer Case
BigData Corp
![Page 40: (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014](https://reader033.fdocuments.net/reader033/viewer/2022060119/55908d811a28ab736d8b45e6/html5/thumbnails/40.jpg)
Please give us your feedback on this
presentation