EE 660: Computer Architecture Cloud Architecture: IaaS
Transcript of EE 660: Computer Architecture Cloud Architecture: IaaS
![Page 1: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/1.jpg)
Yao ZhengDepartmentofElectrical Engineering
UniversityofHawaiʻiat Mānoa
BasedontheslidesofProf.RoyCampbell&ProfRezaFarivar
EE660:ComputerArchitectureCloudArchitecture:IaaS
![Page 2: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/2.jpg)
Agenda
• Infrastructureasa Service• IaaS:OpenStack• IaaSProviders:Amazon• Serverless Architecture• AmazonS3• Dropbox
![Page 3: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/3.jpg)
Spacereservedfor videoDonotputanything here
XenandtheLinux Kernel
• Xen was initially a university research project• Invasive changes to the kernel to run Linux as a paravirtualized
guest• Even more changes to run Linux as dom0• Xen support in the Linux kernel; no upstream• Great maintenance effort on distributions• Risk of distributions dropping Xen support• Xen hard to use
![Page 4: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/4.jpg)
Spacereservedfor videoDonotputanything here
Xen Concepts
• Control Domain a.k.a. Dom0• Dom0 kernel with drivers• Xen management toolstack• Trusted computing base
• Guest Domains• Your apps• For example, your cloud management stack
• Driver/Stub/Service Domain(s)• A "driver, device” model or “control service in a box"• De-privileged and isolated• Lifetime: start, stop, kill
![Page 5: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/5.jpg)
Spacereservedfor videoDonotputanything here
Mirantis FuelandOpenStack Architecture
Fuelcontroller Node Node
OpenStackcompute
Boot serverImages
Dashboard
Images =controller,compute, bootserver,networkOS,storage OS
![Page 6: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/6.jpg)
Spacereservedfor videoDonotputanything here
OpenStack Architecture
Shared resources
OpenStackcontroller Compute Storage
User applications
Dashboard
Network
![Page 7: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/7.jpg)
Spacereservedfor videoDonotputanything here
Mirantis Example
• Mirantis brings up a fuel controller
• The fuel controller allows you to bring up OpenStack controller, networks, compute servers, and storage
• https://training.mirantis.com/online-resources/fuel-video
![Page 8: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/8.jpg)
Agenda
• Infrastructureasa Service• IaaS:OpenStack• IaaSProviders:Amazon• Serverless Architecture• AmazonS3• Dropbox
![Page 9: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/9.jpg)
Spacereservedfor videoDonotputanything here
OpenStack
“The OpenStack project has been created with the audaciousgoal of being the ubiquitous software choice for building cloudinfrastructures.”— Ken Pepple
“OpenStack is a collection of open source software projects that enterprises/service providers can use to set up and run their cloud compute and storage infrastructure.”— docs.openstack.org
![Page 10: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/10.jpg)
Spacereservedfor videoDonotputanything here
OpenStack Architecture
Shared resources
OpenStackcontroller Compute Storage
User applications
Dashboard
Network
![Page 11: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/11.jpg)
Spacereservedfor videoDonotputanything here
OpenStack'sCore Components
• Compute (“Nova”)• Orchestrates large network for virtual machines (VMs)• Responsible for VM instance lifecycle, network management, and user
access control
• Object Storage (“Swift”)• Provides scalable, redundant, long-term storage for things like VM
images, data archives, and multimedia
![Page 12: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/12.jpg)
Spacereservedfor videoDonotputanything here
OpenStack'sCore Components
• Image Service (“Glance”)• Manages VM disk images• Can be a stand-alone service• Supports private/public permissions, and can handle a variety of disk
image formats
![Page 13: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/13.jpg)
Agenda
• Infrastructureasa Service• IaaS:OpenStack• IaaSProviders:Amazon• Serverless Architecture• AmazonS3• Dropbox
![Page 14: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/14.jpg)
Spacereservedfor videoDonotputanything here
AmazonWebServices (AWS)
• AWS provides a collection of services for building Cloud applications
• Services for• Storage: S3, EBS• Computation: Elastic Cloud Computing (EC2), scaling/load
balancer, Elastic MapReduce, Elastic Beanstalk• Database: RDS, DynamoDB, ElastiCache• Coordination: Simple notification service, simple workflow
framework
• All services are paid depending on use
![Page 15: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/15.jpg)
Spacereservedfor videoDonotputanything here
AmazonWeb Services
![Page 16: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/16.jpg)
Spacereservedfor videoDonotputanything here
Nine Regions
• US East (N. Virginia) (135)• US West (Oregon) (135)• US West (N. California) (135)• EU (Frankfurt) (60)• EU (Ireland) (135)• Asia Pacific (Singapore) (134)• Asia Pacific (Tokyo) (134)• Asia Pacific (Sydney) (133)• South America (São Paulo) (130)
![Page 17: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/17.jpg)
Spacereservedfor videoDonotputanything here
SixTypesof Instances
• Micro instances (free tier)• General purpose• Memory optimized• Storage optimized• Compute optimized• GPU instances
![Page 18: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/18.jpg)
Spacereservedfor videoDonotputanything here
Amazonversus Competition
• Wall Street estimates of AWS revenue are typically in the $4 billion to $5 billion range for 2014, representing a doubling of revenue over a 2-year period, despite continuous decreases in prices
• 10× capacity of its nearest 145 competitors combined (Gartner)
![Page 19: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/19.jpg)
Spacereservedfor videoDonotputanything here
Amazon Technology
• Xen Hypervisor – proprietary• Micro instances are oversubscribed, but others are not• Storage and local area networks are shared – multi-
tenant• Billed by hour• Provisional Amazon machine image instances –
licenses are a user problem• Operating system is a customer problem• Can use third-party consoles to control Cloud VMWare,
Microsoft
![Page 20: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/20.jpg)
Spacereservedfor videoDonotputanything here
Storage
• Transient, instance-specific storage• Persistent, instance-independent Elastic Block
Store (EBS) storage (SSD and encryption options)
• Object-based Simple Storage Service (S3)• Data restricted to region
![Page 21: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/21.jpg)
Spacereservedfor videoDonotputanything here
Networking
• Virtual Private Cloud (=VPN)• Private routing between VPCs• VPN tunnels can connect your enterprise to
Amazon• DirectConnect allows a customer to use a carrier
for private WAN services
![Page 22: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/22.jpg)
Spacereservedfor videoDonotputanything here
![Page 23: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/23.jpg)
Spacereservedfor videoDonotputanything here
![Page 24: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/24.jpg)
Spacereservedfor videoDonotputanything here
Cloud Computing Applications - Roy Campbell 12
![Page 25: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/25.jpg)
Spacereservedfor videoDonotputanything here
Cloud Computing Applications - Roy Campbell 13
![Page 26: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/26.jpg)
Spacereservedfor videoDonotputanything here
Cloud Computing Applications - Roy Campbell 14
![Page 27: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/27.jpg)
Spacereservedfor videoDonotputanything here
Cloud Computing Applications - Roy Campbell 15
![Page 28: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/28.jpg)
Spacereservedfor videoDonotputanything here
Cloud Computing Applications - Roy Campbell 16
![Page 29: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/29.jpg)
Spacereservedfor videoDonotputanything here
Cloud Computing Applications - Roy Campbell 17
![Page 30: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/30.jpg)
Spacereservedfor videoDonotputanything here
Cloud Computing Applications - Roy Campbell 18
![Page 31: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/31.jpg)
Spacereservedfor videoDonotputanything here
Serverless Architecture
![Page 32: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/32.jpg)
Spacereservedfor videoDonotputanything here
IntroductiontoServerless Architecture
• “Applications where some amount of server-side logic is still written by the application developer but unlike traditional architectures is run in stateless compute containers that are event-triggered, ephemeral (may only last for one invocation), and fully managed by a 3rd party”
• ‘Functions as a service / FaaS’• AWS Lambda is one of the most popular implementations
of FaaS at present, but there are others
![Page 33: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/33.jpg)
Spacereservedfor videoDonotputanything here
IntroductiontoServerless Architecture
![Page 34: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/34.jpg)
Spacereservedfor videoDonotputanything here
Hardware
HAL
Operating System
Dell
Microsoft
Desktop Platform
![Page 35: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/35.jpg)
Spacereservedfor videoDonotputanything here
Hardware
HAL
Runtime
Dell Microsoft
QoS Events SDKs
Operating System
Applications
Provision&Deployment
System Administrators
Software
Developers
Desktop Platform
![Page 36: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/36.jpg)
Spacereservedfor videoDonotputanything here
Hardware(S3,EC2,ELB etc.)CAL
CloudOperatingSystem(Elasticity,VPN,DNS etc.)
AWS (IaaS)
Provision&DeploymentSystem Administrators
AWSCloudPlatform, 2010
![Page 37: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/37.jpg)
Spacereservedfor videoDonotputanything here
Hardware(S3,EC2, ELB)CAL
AWS (IaaS)
Provision&Deployment
SystemAdministrators=> “DevOps”
Runtime
QoS SDKs
CloudOperatingSystem(Elasticity,VPNs etc)
Applications(ElasticBeanstalk,custom etc)
Software
Developers
AWS (PaaS)
AWSCloudplatform, 2014
![Page 38: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/38.jpg)
Spacereservedfor videoDonotputanything here
Hardware(S3,EC2, ELB)CAL CloudOperatingSystem(Elasticity,
VPNs etc)
AWS (IaaS)
Provision&Deployment
SystemAdministrators=> “DevOps”
Runtime
QoS Lambda SDKs
Software
Developers
AWS (TCaaS)
Kinesis EMR RedShift
*LambdaisaGame Changer
AWSCloudPlatform 2016
![Page 39: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/39.jpg)
Spacereservedfor videoDonotputanything here
• Deploy and scale web applications easily
• Languages: Java, .NET, PHP, Node.js, Python, Ruby, Docker
• Servers: Apache, Nginx, Phusion Passenger, IIS
• Simply upload your code; AWS handles:Deployment
Capacity Provisioning
Load balancing
Autoscaling
HealthMonitoring
AWSElastic BeanStalk
![Page 40: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/40.jpg)
Spacereservedfor videoDonotputanything here
• Runs stateless, request-driven code calledLambda functions in Java, NodeJS & Python
• Triggered by events (state transitions) in other AWS services
• Pay only for the requests served and the computetime
• Focus on business logic, not infrastructure.• Just upload your code; AWS Lambda handles:
Capacity
Scaling
Deployment
Monitoring
Logging
Webservicefront end
AWSLambdaEvent-driven Compute
FaultTolerance
Security Patching
![Page 41: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/41.jpg)
Spacereservedfor videoDonotputanything here
• Amazon S3
• Amazon DynamoDB
• Amazon Kinesis Streams
• AWS CloudFormation
• Amazon CloudWatch Logs
AWSLambdaEvent Sources
• Amazon CloudWatch Events
• Scheduled Events (powered by Amazon CloudWatch Events)
• AWS Config• Amazon Simple Notification Service
• Amazon Echo• Amazon Simple Email Service
• Amazon API Gateway• Amazon Cognito
• Other Event Sources: Invoking a LambdaFunction On Demand
• Sample Events Published by Event Sources
![Page 42: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/42.jpg)
Spacereservedfor videoDonotputanything here
• State-less functions
• You can use multi-threading, etc.
• 500 MB of /tmp storage space
• You set how much memory you need:• From 128 MB to 1.5GB
• 64GB increments
• CPU scales accordingly
• Function should finish in a certain time• Default 3 seconds, up to 300 seconds
AWSLambdaExecution Environment
![Page 43: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/43.jpg)
Spacereservedfor videoDonotputanything here
• You pay per use of your function
• $0.20 per 1 million function call
• Also, $0.00001667 for every GB-second used
AWSLambda Pricing
![Page 44: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/44.jpg)
Agenda
• Infrastructureasa Service• IaaS:OpenStack• IaaSProviders:Amazon• Serverless Architecture• AmazonS3• Dropbox
![Page 45: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/45.jpg)
Spacereservedfor videoDonotputanything here
Definition
A Binary Large OBject (BLOB) is a collection of binarydata stored as a single entity in a database management system.
(The blob was a science fiction movie featuring Steve McQueen.)
https://en.wikipedia.org/wiki/Binary_large_object
Cloud Computing Applications - Roy Campbell 2
![Page 46: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/46.jpg)
Spacereservedfor videoDonotputanything here
Use case
• Store unstructured object data like text or binary data• Images• Movies• Audio, Signal data• Large queue of messages• Example is LinkedIn data in a user page (Uses Ambry)• Usually accessible over the web
Cloud Computing Applications - Roy Campbell 3
![Page 47: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/47.jpg)
Spacereservedfor videoDonotputanything here
Examples
• Windows Azure Blob Storage• Ambry – LinkedIn• Facebook’s Warm BLOB Storage System• Amazon Simple Storage Service (S3)*• Apache Open Stack Blob Service (SWIFT)
Cloud Computing Applications - Roy Campbell 4
![Page 48: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/48.jpg)
Spacereservedfor videoDonotputanything here
Goals
• Data growth ~ 50% a year• 50%-70% data is unstructured or archival• RESTful API (HTTP)• High availability (no single point of failure)• Agile data centers• Open Source• Multi-region, geographic distribution of data• Storage policies• Erasure Coding
Cloud Computing Applications - Roy Campbell 5
![Page 49: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/49.jpg)
Spacereservedfor videoDonotputanything here
Swift API
https://swift.illinois.edu/version2/auth_account/container/object
PUT /version2/roy/myblobs/classvideo1
GET /version2/reza/hisblobs/yesterdaysdataforhadoop
Cloud Computing Applications - Roy Campbell 6
![Page 50: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/50.jpg)
Spacereservedfor videoDonotputanything here
Swift Components
Cloud Computing Applications - Roy Campbell 7
Proxy
Account|Container| Object
Erasure Encoding
Web requests
![Page 51: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/51.jpg)
Spacereservedfor videoDonotputanything here
WriteRequests- LoadBalancerandProxy
Cloud Computing Applications - Roy Campbell 8
LoadBalancer
Proxy
ObjectNodes
Client Requests
Use Quorum
![Page 52: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/52.jpg)
Spacereservedfor videoDonotputanything here
ReadRequests- LoadBalancerandProxy
Cloud Computing Applications - Roy Campbell 9
LoadBalancer
Proxy
ObjectNodes
Client Requests
![Page 53: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/53.jpg)
Spacereservedfor videoDonotputanything here
Details
• MD5 Checksums with each object• Auditing and active replication• Any sized disks
Cloud Computing Applications - Roy Campbell 10
![Page 54: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/54.jpg)
Spacereservedfor videoDonotputanything here
Swift Partitions
• 1 Node, 8 Disks, 16 Partitions per disk, 8*16 = 128 partitions• 2 Nodes, 8 Disks each, 8 Partitions per disk, 8*16 = 128
partitions• Use Hash into Ring to map objects into storage partitions
Cloud Computing Applications - Roy Campbell 11
A BC
DE
F
…
Disk 5,Partition 7
Disk 1 Disk 2Disk
3
![Page 55: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/55.jpg)
Agenda
• Infrastructureasa Service• IaaS:OpenStack• IaaSProviders:Amazon• Serverless Architecture• AmazonS3• Dropbox
![Page 56: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/56.jpg)
Spacereservedfor videoDonotputanything here
Definition
Online file storage web service offered by Amazon Web Services. Amazon S3 provides storage through web service interfaces REST, SOAP, BitTorrent. (wikipedia)
https://aws.amazon.com/s3/
Cloud Computing Applications - Roy Campbell 2
![Page 57: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/57.jpg)
Spacereservedfor videoDonotputanything here
Use case
• Scalability, high availability, low latency – 99.99% availability
• Files up to 5 terabytes• Objects stored in buckets owned by users• User assigned keys refer to objects
• Amazon Machine Images (exported as a bundle of objects)
• SmugMug, Hadoop file store, Netflix, reddit, Dropbox, Tumbler
Cloud Computing Applications - Roy Campbell 3
![Page 58: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/58.jpg)
Spacereservedfor videoDonotputanything here
SimpleStorageService (S3)
• A bucket is a container for objects and describes location, logging, accounting, and access control.
• A bucket has a name that must be globally unique.• http://bucket.s3.amazonaws.com• http://bucket.s3-aws-region.amazonaws.com.
• A bucket can hold any number of objects, which are files of up to 5TB.
• http://bucket.s3.amazonaws.com/object• http://johnsmith.s3.amazonaws.com/photos/puppy.jpg
Cloud Computing Applications - Roy Campbell 4
![Page 59: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/59.jpg)
Spacereservedfor videoDonotputanything here
FundamentaloperationscorrespondingtoHTTPactions:
http://bucket.s3.amazonaws.com/object
• POST a new object or update an existing object.• GET an existing object from a bucket.• DELETE an object from the bucket• LIST keys present in a bucket, with a filter.
A bucket has a flat directory structure
Cloud Computing Applications - Roy Campbell 5
![Page 60: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/60.jpg)
Spacereservedfor videoDonotputanything here
S3WeakConsistency Model
• “Updatestoasinglekeyare atomic….”• “AmazonS3achieveshighavailabilitybyreplicatingdataacrossmultipleserverswithinAmazon'sdatacenters.IfaPUTrequestissuccessful,yourdataissafelystored. However:
• AprocesswritesanewobjecttoAmazonS3andimmediatelyattemptstoreadit.Untilthechangeisfullypropagated,AmazonS3mightreport"keydoesnotexist."
• AprocesswritesanewobjecttoAmazonS3andimmediatelylistskeyswithinitsbucket.Untilthechangeisfullypropagated,theobjectmightnotappearinthe list.
• Aprocessreplacesanexistingobjectandimmediatelyattemptstoreadit.Until thechangeisfullypropagated,AmazonS3mightreturnthepriordata.
• Aprocessdeletesanexistingobjectandimmediatelyattemptstoreadit.Until thedeletionisfullypropagated,AmazonS3mightreturnthedeleted data.”
Cloud Computing Applications - Roy Campbell 6
![Page 61: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/61.jpg)
Spacereservedfor videoDonotputanything here
S3CommandLineInterface
localfile s3://bucket/keys3://bucket/newname
aws s3 mb s3://bucket. . . cp
mv ls rm rb
s3://bucket/key s3://bucket s3://bucket/key s3://bucket
aws s3 help aws s3 ls help
Cloud Computing Applications - Roy Campbell 7
![Page 62: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/62.jpg)
Agenda
• Infrastructureasa Service• IaaS:OpenStack• IaaSProviders:Amazon• Serverless Architecture• AmazonS3• Dropbox
![Page 63: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/63.jpg)
Spacereservedfor videoDonotputanything here
Cloud Storage
• One interesting case study is Dropbox• Dropbox offers cloud file storage
• Easily synced across multiple devices• Accessible through web interface, mobile
apps, and directly integrated with the file system on PCs
• Dropbox itself uses clouds!• Metadata stored in Dropbox servers• Actual files stored in Amazon S3• Amazon EC2 instances run the logic
![Page 64: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/64.jpg)
Spacereservedfor videoDonotputanything here
Dropbox Architecture
![Page 65: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/65.jpg)
Spacereservedfor videoDonotputanything here
Dropbox API
• Two levels of API access to Dropbox• Drop-ins
• Cross-platform UI components that can be integrated in minutes
• Chooser allows instant access to files in Dropbox• Saver makes saving files to Dropbox easy
• CoreAPI• Support for advanced functionality like search,
revisions, and restoring file• Better fit for deeper integration
![Page 66: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/66.jpg)
Spacereservedfor videoDonotputanything here
Drop-In API
• Simple objects• Chooser available for JavaScript, Android and
iOS• Saver on web and mobile web
• Handles all the authentication (OAuth), file browsing
• Chooser object returns the following:• Link: URL to access the file• File name• File Size• Icon• Thumbnails
• Saver• Pass in URL, filename and options
![Page 67: EE 660: Computer Architecture Cloud Architecture: IaaS](https://reader033.fdocuments.net/reader033/viewer/2022060415/62954e71deb4e44e934ab376/html5/thumbnails/67.jpg)
Spacereservedfor videoDonotputanything here
Core API
• Many languages and environments• Python, Ruby, PHP, Java, Android, iOS, OS X, HTTP
• Based on HTTP and OAuth• OAuth v1, OAuth v2
• Low-level calls to access and manipulate a user's Dropbox account
• Create URL schemes• Upload files• Download files• List files and folders• Delta• Metadata access• Create and manage file sharing