Cloud Storage Simplified

Adrian Cole / Cloudsoft

Tuesday, November 1, 11

Adrian Cole (@jclouds)founded jclouds march 2009chief evangelist at Cloudsoft

Tuesday, November 1, 11

Agenda

• intro• concepts• jclouds• code• awkward silence (or Q/A)

Tuesday, November 1, 11

Blob Storage

4

Set<String> containers = namespacesInMyAccount;

Map<String, InputStream> keyValues = contentsOfContainer;

Tuesday, November 1, 11

adrian@googlestorage

Love Letters

TronMovies

Goonies The Blob

ShrekThe One

putBlob

3d = trueurl = http://disney.go.com/tron

Blob Storage

5

Tuesday, November 1, 11

6

Can I use BlobStores portably?

Tuesday, November 1, 11

are they close enough?

7

Tuesday, November 1, 11

who builds and runs your service?

8

Tuesday, November 1, 11

9

Tuesday, November 1, 11

tenancy

10

Tuesday, November 1, 11

size and resumability

11

Tuesday, November 1, 11

How many copies?

12

Tuesday, November 1, 11

Consistency model?

13

Tuesday, November 1, 11

Are you experienced? Is your product?

14

Tuesday, November 1, 11

15

Tuesday, November 1, 11

PUT /sushi.avi HTTP/1.1 Host: adriansmovies.s3.amazonaws.comContent-Length: 734859264Date: Wed, 01 Mar 2006 12:00:00 GMTAuthorization: signature x-amz-meta-Chef: Kawasaki

PUT /<api version>/<account>/adriansmovies/sushi.avi HTTP/1.1 Host: storage.clouddrive.com Transfer-Encoding: chunkedX-Auth-Token: session-token X-Object-Meta-Chef: Kawasaki

PUT /adriansmovies/sushi.avi HTTP/1.1 Host: <account>.blob.core.windows.netContent-Length: 734859264Date: Wed, 01 Mar 2006 12:00:00 GMTAuthorization: SharedKey <app>:signaturex-ms-meta-Chef: Kawasaki

POST /namespace/adriansmovies/sushi.avi HTTP/1.1 Content-Length: 734859264Date: Wed, 01 Mar 2006 12:00:00 GMTx-emc-uid: <uid> x-emc-signature: signature x-emc-meta: Chef=Kawasaki

CODE AND SIGN THE HTTP REQUEST

16

Tuesday, November 1, 11

GET /ws/IMFS/GetStorageNodeExtended.ashx?&fileOverwrite=true&ipRestricted=true&destFolderPath= adriansmovies&sizeBytes= 734859264&firstByteExpiration=6000&lastByteExpiration=259200&sessionToken=session-token HTTP/1.1

POST /Upload.ashx?uploadToken=from_above&destFolderPath=adriansmovies HTTP/1.1Host: from_above Content-Length: 734859382Content-Type=multipart/form-data; boundary=--jclouds--Authorization=Basic GpjbG9=----jclouds--Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"Content-Type: application/octetstring...

PUT /ws/Metadata/SetMetadata.ashx?&path=Folders/adriansmovies/sushi.avi&sessionToken=session-token&metadata=Chef:Kawasaki HTTP/1.1

CODE AND SIGN THE HTTP REQUEST

17

Tuesday, November 1, 11

POST /<api version>/containers/id_of_ adriansmovies/contents HTTP/1.1 Content-Length: 734859382Content-Type=multipart/form-data; boundary=--jclouds--Authorization=Basic GpjbG9=----jclouds--Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"Content-Type: application/octetstring...

PUT /<api version>/files/from_above/metadata/Chef HTTP/1.1 Content-Length: 8Content-Type: text/plainAuthorization: Basic GpjbG9=Kawasaki

CODE AND SIGN THE HTTP REQUEST

18

Tuesday, November 1, 11

19

GET https://api.gogrid.com/api/grid/server/add?v=1.5&name=serverName&server.ram=memory&image=img55&ip=127.0.0.1&isSandbox=true&description=fooy&sig=3f446f171455fbb5574aecff4997b273&api_key=foo HTTP/1.1

POST http://serverManagementUrl/servers?format=json HTTP/1.1Accept: application/jsonContent-Type: application/json{"server":{"name":"ralphie","imageId":2,"flavorId":1,"sharedIpGroupId":2,"addresses":{"public":["127.0.0.1"]}}}

POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1Content-Type: application/x-www-form-urlencodedVersion=2010-06-15&Action=RunInstances&ImageId=ami-voo&MinCount=1&MaxCount=5&KernelId=kernelId&Monitoring.Enabled=true&SecurityGroup.1=group1&SecurityGroup.2=group2&Placement.AvailabilityZone=eu-west-1a

POST https://vcloud.safesecureweb.com/api/v0.8/vdc/1/action/instantiateVAppTemplate HTTP/1.1

Accept: application/vnd.vmware.vcloud.vApp+xmlContent-Type: application/vnd.vmware.vcloud.instantiateVAppTemplateParams+xml

<InstantiateVAppTemplateParams xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

name="my-vapp" xsi:schemaLocation="http://www.vmware.com/vcloud/v0.8 http://vcloud.safesecureweb.com/ns/vcloud.xsd">

<VAppTemplate href="https://vcloud.safesecureweb.com/api/v0.8/vAppTemplate/3" />

<InstantiationParams> <VirtualHardwareSection xmlns:q1="http://www.vmware.com/vcloud/v0.8">

<Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID

xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</InstanceID> <ResourceType

xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">3</ResourceType> <VirtualQuantity

xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</VirtualQuantity> </Item>

<Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID

xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">2</InstanceID> <ResourceType

xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">4</ResourceType> <VirtualQuantity

xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">512</VirtualQuantity> </Item>

<Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID

xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">9</InstanceID> <ResourceType

xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">17</ResourceType> <VirtualQuantity

xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1024</VirtualQuantity> </Item>

</VirtualHardwareSection> <NetworkConfigSection>

<NetworkConfig name="my-vapp"> <Features>

<FenceMode>allowInOut</FenceMode> </Features>

<NetworkAssociation href="https://vcloud.safesecureweb.com/network/1990" /> </NetworkConfig>

</NetworkConfigSection> </InstantiationParams>

</InstantiateVAppTemplateParams>

POST https://www-180.ibm.com/cloud/enterprise/beta/api/rest/20090403/instances HTTP/1.1Accept: Accept: application/jsonContent-Type: application/x-www-form-urlencodedlocation=location&imageID=22&name=name&instanceType=instanceType&ip=1&publicKey=MOO&volumeID=2&oss.storage.id.0.mnt-%2Fmnt&insight_admin_password=myPassword1&db2_admin_password=myPassword2&report_user_password=myPassword3

POST https://rimuhosting.com/r/orders/new-vps HTTP/1.1Accept: Accept: application/jsonContent-Type: application/json{"request":{"instantiation_options":{"distro":"lenny","domain_name":"test.ivan.api.com"},"pricing_plan_code":"MIRO1B","meta_data":[]}}

Deal with ErrorsDeal with ConcurrencyDeal with Complexity

do you want to

Tuesday, November 1, 11

open source

feels like java (and clojure)

portability between clouds

deal with web complexity

unit testability

thread-safe and scalable

20

Tuesday, November 1, 11

BlobStore LoadBalancer

Compute Table

Portable APIs

Embeddable

Provider-Specific Hooks

Over 30 Tested Providers!

21

Tuesday, November 1, 11

jclouds locations

Location helps normalize placement across resource types All top-level resources have a locationLocation metadata is extensible

listAssignableLocations

IE

US

SG

22

Tuesday, November 1, 11

jclouds modularity

APIs are software focused Providers are offering focusedAPI + location + defaults = Provider

jclouds-blobstore

s3

aws-s3

walrus

eucaluptus-partnercloud-s3

googlestorage

scality-ring

scaleup-storage

23

Tuesday, November 1, 11

Alternatives to jclouds

Roll-your-own• RESTEasy, Jersey

Cloud-specific SDKs• typica, jets3t, Amazon SDK, vCloud SDK

Dasein Cloud API

24

Tuesday, November 1, 11

BlobStore

25

Tuesday, November 1, 11

java overview github jclouds/jclouds

// initcontext = new BlobStoreContextFactory().createContext("s3", accesskeyid, secret);blobStore = context.getBlobStore();

// create containerblobStore.createContainerInLocation(null, “adriansmovies”);

// add blobblob = blobStore.blobBuilder("sushi.avi").payload(file).build();blobStore.putBlob(“adriansmovies”, blob);

26

Tuesday, November 1, 11

clojure overview github jclouds/jclouds

(use 'org.jclouds.blobstore2)

(def *blobstore* (blobstore “azureblob” account key))

(create-container *blobstore* “movies”)(put-blob *blobstore* “movies” (blob “tron.mp4“ :payload tron-file))

27

Tuesday, November 1, 11

Who’s coding BlobStore?

28

Tuesday, November 1, 11

Filesystem github jclouds/jclouds

props.setProperty(FilesystemConstants.PROPERTY_BASEDIR, “basedir”);

blobStore = new BlobStoreContextFactory(). createContext("filesystem", props);

(blobStore "filesystem" “” “” FilesystemConstants/PROPERTY_BASEDIR “basedir”);

29

Tuesday, November 1, 11

BlobRequestSigner github jclouds/jclouds

HttpRequest request = context.getSigner(). signGetBlob(“adriansmovies”, "sushi.avi");

(let [request (sign-blob-request "adriansmovies" "sushi.avi" {:method :get})])

30

Tuesday, November 1, 11

What’s next?

• ACL support• OutputStream returns• Public/Private pathing

31

Tuesday, November 1, 11

32

github jclouds-examples

@jcloudsadrian@cloudsoftcorp.com

Questions?

Tuesday, November 1, 11