Windows Azure Blob Storage A Deep Dive
Transcript of Windows Azure Blob Storage A Deep Dive
Windows Azure Blob Storage – A Deep Dive
Paul Bouwer@pbouwer
http://blog.paulbouwer.com
A service for storing large amounts of
unstructured data that can be accessed from
anywhere in the world via HTTP or HTTPS.
Windows Azure Blob Storage
• Serving images or documents directly to a browser
• Storing files for distributed access
• Streaming video and audio
• Performing secure backup and disaster recovery
• Storing analysis data by an on-premises or Windows Azure
hosted service
3 copies per data centre
> 500 miles
Continuous geo-replication
• Start with letter or number
• Contain only letters, numbers, and dashes (-)
• Dashes (-) must be immediately preceded and followed by a
letter or number. Consecutive dashes not permitted
• All letters must be lowercase.
• Between 3 and 63 characters in length
• Case sensitive
• Any combination of characters
• Reserved URL characters must be escaped
• Between 1 and 1024 characters in length
• Targeted at streaming workloads
• Comprise of blocks
• Block sizes may differ but max size 4MB
• No more than 50,000 blocks
• Max size 200GB
• Targeted at random read/write workloads
• Comprise of 512 byte pages
• Max size 1TB
• Private (Default)
Container only accessible by account owner
• Public Container
Full public read access to container and blobs
• Public Blob
Public read access to blobs
The term Representation State Transfer (REST) was coined by Roy
Fielding in his Ph.D. dissertation “Architectural Styles and the
Design of Network-based Software Architectures.”
Roy Fielding - one of the principal authors of the HTTP specification and co-founder of the
Apache HTTP Server project.
The REST architectural style is commonly applied to the design of
APIs for modern web services. A Web API conforming to the REST
architectural style is a REST API.
• Microsoft.WindowsAzure.Storage.dll
• NuGet – WindowsAzure.Storage
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
<?xml version="1.0" encoding="utf-8"?><configuration><appSettings>
<add key="StorageConnectionString" value="DefaultEndpointsProtocol=https;AccountName=[AccountName];AccountKey=[AccountKey]" /></appSettings>
</configuration>
App.config | Web.config
C#
HEAD http://myaccount.blob.core.windows.net/mycontainer?restype=container
PUT http://myaccount.blob.core.windows.net/mycontainer?restype=container
CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");container.CreateIfNotExists();
C#
REST API http://msdn.microsoft.com/en-us/library/dd179468.aspx
PUT http://myaccount.blob.core.windows.net/mycontainer?restype=container&comp=acl
x-ms-blob-public-access: blobx-ms-blob-public-access: container
CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");
// Public Containercontainer.SetPermissions(new BlobContainerPermissions{ PublicAccess = BlobContainerPublicAccessType.Container });
// Public Blobcontainer.SetPermissions(new BlobContainerPermissions{ PublicAccess = BlobContainerPublicAccessType.Blob });
C#
REST API http://msdn.microsoft.com/en-us/library/dd179391.aspx
PUT http://myaccount.blob.core.windows.net/mycontainer/myblob
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob"); using (var fileStream = File.OpenRead(@"path\myfile")) {blockBlob.UploadFromStream(fileStream);
}
C#
REST API http://msdn.microsoft.com/en-us/library/dd179451.aspx
GET http://myaccount.blob.core.windows.net/mycontainer?restype=container&comp=list
GET http://myaccount.blob.core.windows.net/mycontainer?restype=container&comp=list&prefix=&delimiter=
foreach (var blockBlob in blobContainer.ListBlobs(prefix: null, useFlatBlobListing: true)) {}
foreach (var blockBlob in blobContainer.ListBlobs(prefix: prefix, useFlatBlobListing: false)) { if (blockBlob is CloudBlockBlob) { ... }if (blockBlob is CloudBlockDirectory) { ... }
}
C#
REST API http://msdn.microsoft.com/en-us/library/dd135734.aspx
GET http://myaccount.blob.core.windows.net/mycontainer/myblob
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob"); using (var fileStream = File.OpenWrite(@"path\myfile")) { blockBlob.DownloadToStream(fileStream);
}
C#
REST API http://msdn.microsoft.com/en-us/library/dd179440.aspx
DELETE http://myaccount.blob.core.windows.net/mycontainer/myblob
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob"); blockBlob.DeleteIfExists();
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");blockBlob.Delete();
C#
REST API http://msdn.microsoft.com/en-us/library/dd179413.aspx
HEAD http://myaccount.blob.core.windows.net/mycontainer/myblob
PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=properties
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob"); blockBlob.FetchAttributes();Console.WriteLine(blockBlob.Properties.ETag);
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");blockBlob.Properties.ContentType = @"text/plain"; blockBlob.SetProperties();
C#
REST APIhttp://msdn.microsoft.com/en-us/library/dd179394.aspx
http://msdn.microsoft.com/en-us/library/ee691966.aspx
HEAD http://myaccount.blob.core.windows.net/mycontainer/myblob
PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=metadata
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");blockBlob.FetchAttributes();Console.WriteLine("{0}", blockBlob.Metadata.ContainsKey("mymeta") ? blockBlob.Metadata["mymeta"] : "-");
var metadata = new Dictionary<string, string>();CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob"); foreach (KeyValuePair<string, string> pair in metadata) { blockBlob.Metadata.Add(pair); } blockBlob.SetMetadata();
C#
REST APIhttp://msdn.microsoft.com/en-us/library/dd179350.aspx
http://msdn.microsoft.com/en-us/library/dd179414.aspx
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");var etag = blockBlob.Properties.ETag; var accessCondition = new AccessCondition {IfMatchETag = etag};blockBlob1.SetMetadata(accessCondition);
C#
PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=lease
x-ms-lease-action: <acquire | renew | change | release | break>
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");
// Leases must be 15 - 60s or infiniteblockBlob.AcquireLease(TimeSpan.FromSeconds(15), leaseId);blockBlob.SetMetadata(new AccessCondition { LeaseId = leaseId }); blockBlob.ReleaseLease(new AccessCondition { LeaseId = leaseId });
C#
REST API http://msdn.microsoft.com/en-us/library/ee691972.aspx
PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=snapshot
GET http://myaccount.blob.core.windows.net/mycontainer/myblob?snapshot=<DateTime>
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");CloudBlockBlob snapshot = blockBlob.CreateSnapshot();
CloudBlockBlob backup = new CloudBlockBlob(blockBlob.Uri, ((CloudBlockBlob)snapshot).SnapshotTime, blobClient.Credentials); blockBlob.StartCopyFromBlob(backupSnapshot);
C#
REST API http://msdn.microsoft.com/en-us/library/ee691971.aspx
PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=block&blockid=id
PUT http://myaccount.blob.core.windows.net/mycontainer/myblob?comp=blocklist
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");blockBlob.PutBlock(blockId1, memStream, null); blockList.Add(blockId1); blockBlob.PutBlock(blockId2, memStream, null); blockList.Add(blockId2); blockBlob.PutBlockList(blockList);
C#
REST APIhttp://msdn.microsoft.com/en-us/library/dd135726.aspx
http://msdn.microsoft.com/en-us/library/dd179467.aspx
• Authentication Schemes
Authentication via Storage Key
• Shared Access Signature
Single use authentication and access
• Shared Access Policy
Revokable authentication and access
• SharedKey
• SharedKeyLite
Authorization="[SharedKey|SharedKeyLite] <AccountName>:<Signature>"
StringToSign = VERB + "\n" +Content-Encoding + "\n"Content-Language + "\n"Content-Length + "\n"Content-MD5 + "\n" +Content-Type + "\n" +Date + "\n" +If-Modified-Since + "\n"If-Match + "\n"If-None-Match + "\n"If-Unmodified-Since + "\n"Range + "\n"CanonicalizedHeaders + CanonicalizedResource;
Signature=Base64(HMAC-SHA256(UTF8(StringToSign)))
http://msdn.microsoft.com/en-us/library/dd179428.aspx
GET\n /*HTTP Verb*/\n /*Content-Encoding*/\n /*Content-Language*/\n /*Content-Length*/\n /*Content-MD5*/\n /*Content-Type*/\n /*Date*/\n /*If-Modified-Since */\n /*If-Match*/\n /*If-None-Match*/\n /*If-Unmodified-Since*/\n /*Range*/x-ms-date:Sun, 11 Oct 2009 21:49:13 GMT\nx-ms-version:2009-09-19\n /*CanonicalizedHeaders*//myaccount/myaccount/mycontainer\ncomp:metadata\nrestype:container\ntimeout:20 /*CanonicalizedResource*/
Authorization: SharedKey myaccount:ctzMq410TV3wS7upTBcunJTDLEJwMAZuFPfr0mrrA08=
StringToSign = VERB + "\n" +Content-MD5 + "\n" +Content-Type + "\n" +Date + "\n" +CanonicalizedHeaders + CanonicalizedResource;
Signature=Base64(HMAC-SHA256(UTF8(StringToSign)))
http://msdn.microsoft.com/en-us/library/dd179428.aspx
PUT\n /*HTTP Verb*/\n /*Content-MD5*/text/plain; charset=UTF-8\n /*Content-Type*/\n /*Date*/x-ms-date:Sun, 20 Sep 2009 20:36:40 GMT\nx-ms-meta-m1:v1\nx-ms-meta-m2:v2\n /*CanonicalizedHeaders*//testaccount1/mycontainer/hello.txt /*CanonicalizedResource*/
Authorization: SharedKeyLite myaccount:ctzMq410TV3wS7upTBcunJTDLEJwMAZuFPfr0mrrA08=
https://myaccount.blob.core.windows.net/mycontainer/myblob?sv=2012-02-12&se=2013-01-16T00%3A45%3A58Z&sr=b&sp=r&sig=NqVpCv9LQGUDbD8hOKz%2FO4ybcZlU9yOB6G42xaAbq%2Bw%3D
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");var signature = blockBlob.GetSharedAccessSignature(new SharedAccessBlobPolicy{ Permissions = SharedAccessBlobPermissions.Read, SharedAccessStartTime = null, SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(1)
}); Console.WriteLine"Shared Access URI:{0}{1}", blockBlob.Uri.AbsoluteUri, signature);
C#
URL http://msdn.microsoft.com/en-us/library/hh508996.aspx
PUT http://myaccount.blob.core.windows.net/mycontainer?restype=container&comp=acl
var blobContainerPermissions = new BlobContainerPermissions(); blobContainerPermissions.SharedAccessPolicies.Add("ReadOnlyPolicy", new SharedAccessBlobPolicy{ Permissions = SharedAccessBlobPermissions.Read, SharedAccessStartTime = null
}); blobContainer.SetPermissions(blobContainerPermissions);
C#
REST API http://msdn.microsoft.com/en-us/library/ee393341.aspx
https://myaccount.blob.core.windows.net/mycontainer/myblob?sv=2012-02-12&se=2013-01-26T00%3A57%3A35Z&sr=b&si=ReadOnlyPolicy&sig=X2mv6QDv%2FeMoNDvCheoJIMC8QzPpJSDoVAUhvEvbzC0%3D
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("myblob");var signature = blockBlob.GetSharedAccessSignature(new SharedAccessBlobPolicy{ SharedAccessExpiryTime = DateTime.UtcNow.AddDays(10)
}, "ReadOnlyPolicy");
Console.WriteLine"Shared Access URI:{0}{1}", blockBlob.Uri.AbsoluteUri, signature);
C#
URL http://msdn.microsoft.com/en-us/library/hh508996.aspx
• Storage Analytic Metrics
Aggregated transaction statistics
Capacity data
• Accessed via Table Storage
$MetricsTransactionsBlob
$MetricsCapacityBlob
http://msdn.microsoft.com/en-us/library/windowsazure/hh343258.aspx
http://msdn.microsoft.com/en-us/library/windowsazure/hh343264.aspx
• Storage Analytic Logging
Successful and failed requests
• Access via Blob Storage
$logs
<service-name>/YYYY/MM/DD/hhmm/<counter>.log
http://msdn.microsoft.com/en-us/library/windowsazure/hh343262.aspx
• How to use the Windows Azure Blob Storage Service in .NET
http://www.windowsazure.com/en-us/develop/net/how-to-guides/blob-storage/
• MSDN - Blob Service REST API
http://msdn.microsoft.com/en-us/library/windowsazure/dd135733.aspx/
• Library Reference - Microsoft.WindowsAzure.Storage.Blob
http://msdn.microsoft.com/en-
us/library/windowsazure/microsoft.windowsazure.storage.blob.aspx
• Azure Storage Samples (REST and .NET Storage Client Library)
• http://azurestoragesamples.codeplex.com/
• Azure Storage Explorer - FREE
http://azurestorageexplorer.codeplex.com/
• Windows Azure Explorer - FREE
http://www.cloudberrylab.com/free-microsoft-azure-explorer.aspx
• CloudXplorer - $35
http://clumsyleaf.com/products/cloudxplorer
• Cloud Storage Studio 2 - $195.00
http://www.cerebrata.com/Products/CloudStorageStudio/