Azure Data StorageKen Cenerelli@KenCenerelli
Microsoft Azure
Agenda
Blobs Files Queues Tables
Agenda
SQL
Database
SQL Server
VMDocumentDB Search HDInsight
Microsoft Azure
Storage Blob
Two Types of Blobs Under the Hood
Two Types of Blobs Under the Hood
Block Blob:
Two Types of Blobs Under the Hood
Page Blob:
Blob Storage Concepts
Blob Details –
Shared Access Signatures
Policy based:
Stored Access PolicyUse for longer dated revocable permission sets
Points to a Container level policy
Can only revoke by deleting the blob or waiting for expiry
Use case: Providing revocable permissions to certain users/groups
Shared Access Signatures – Two broad approaches
Ad-hoc:
Shared Access SignatureUse for very short dated single use scenarios
Can only revoke by deleting the blob or waiting for expiry
Use case: Single use URLs
E.g. Provide URL to mobile client to upload to container
Always endeavour to use least permission set possible
Microsoft Azure
Storage Files
Azure Files
Shared Network File Storage for Azure
Availability, durability, scalability are managed automatically
Supports two interfaces: SMB and REST
Azure Files – Usage
Share data across VMs and applications
Share settings throughout services
Share tools amongst developers in a group
Dev/Test/Debug
Azure Files vs Blobs
Description Azure Blobs Azure Files
Durability
Options
LRS, ZRS, GRS (and RA-GRS for higher
availability)LRS, GRS
Accessibility REST APIsSMB 2.1 (standard file system APIs)
REST APIs
Connectivity REST – WorldwideSMB 2.1 - Within region
REST – Worldwide
Endpointshttp://myaccount.blob.core.windows.net/
mycontainer/myblob
\\myaccount.file.core.windows.net\myshare\myfile.txt
http://myaccount.file.core.windows.net/myshare/myfile.txt
DirectoriesFlat namespace however prefix listing
can simulate virtual directoriesTrue directory objects
Case Sensitivity of Names Case sensitive Case insensitive, but case preserving
Capacity Up to 500TB containers 5TB file shares
Throughput Up to 60 MB/s per blob Up to 60 MB/s per share
Object size Up to 1 TB/blob Up to 1 TB/file
Billed capacity Based on bytes written Based on file size
Microsoft Azure
Storage Queue
Queue length reflects how well the backend processing nodes are doing.
Decouples the application and makes scaling easier.
Flexibility of efficient resource usage within an application.
Absorb traffic bursts and reduce the impact of individual component failures.
Why use a Queue?
Storage Account: All access to Azure Storage is done through
a storage account. It can store millions of messages until the
storage account’s maximum capacity of 500 TB is reached.
Queue: A queue contains a set of messages.
Message: A message, in any format, of up to 64KB.
Queue Components
Queues are addressable using the following URL format:
http://{storage-account}.queue.core.windows.net/{queue}
Queue URL format
Faster Web Applications with queues
using asynchronous workloads
Demo
Messages are ordered but not guaranteed FIFO.
Message will be processed at least once.
Message may be processed more than once.
.DequeueCount increases every time.
Messages are stored up to 7 days.
Queue Considerations
Microsoft Azure
Storage Table
Table Storage Concepts
Table Storage Details
Table Storage Details
Table Storage Details – Entity Properties
Table Storage Details
Table Storage Details – Purpose of the PartitionKey
Microsoft Azure SQL Database
From private to public Cloud
Microsoft Relational Storage Options
A Continuous offering
A Server is not a machine
SQL Server database technology “as a Service”
Fully Managed
Enterprise-ready with automatic support for HA, DR, Backups, replication and more
SQL Database – The Basics
Scale out with ElasticScale
Built-in regional database replicas for additional protection
Uptime SLA of 99.99%
SQL Database – The Basics
The Service head contains databases
Connect via automatically generated FQDN: {name}.database.windows.net
Initially contains only a master database
SQL Database – Server Definition
Auto backups, transactional logs every 5 min
Backups in Azure Storage and geo-replicated
Creates a side-by-side copy, non-disruptive
Backups retention policy: 7, 14 or 35 days
Automated export of logical backups for long-term backup protection
Point-in-time restore - “oops recovery”
Available in all tiers: Basic, Standard and Premium
Built on geo-redundant Azure Storage
Recover to any Azure region
Geo-restore – Emergency data recovery when you need it most
Focus on logical vs. physical administration
Database and log files automatically placed
Three high-availability replicas maintained for every database
Tables require a clustered index
Maximum database size is 500 GB
Azure SQL Database vs. SQL Server
Transact-SQL
Use Familiar Technologies
(obviously)
.NET Framework (C#, Visual Basic, F#): ADO.NET
C / C++: ODBC
Java: Microsoft JDBC provider
PHP: Microsoft PHP provider
Use Familiar Technologies - Languages
OData
Entity Framework
WCF Data Services
NHibernate (etc.)
Use Familiar Technologies - Frameworks
SQL Server Management Studio (>=2008 R2)
SQL Server command-line utilities (SQLCMD, BCP)
Visual Studio IDE for database development
Use Familiar Technologies - Tools
Use command, distributed transactions, distributed views
Service Broker
Common Language Runtime (CLR)
SQL Agent
SQL Profiler
Native Encryption
Unsupported SQL Server Features
Elastic Scale
Classic 3-tier enterprise architecture:
Scale out the front ends to multiple instances is easy
Scale the data-tier is more challenging
Web
Role
L
SQL
Worker
Role
Canonical cloud app architecture
SQL Database Considerations and Best Practices
Elastic Scale across thousands of databases via custom sharding
Scale out via .NET Client libraries consumed by customer applications to support sharded database pattern
Enables developer and manageability functions
Supports split, merge, and move operations on data
Vertical: Scale-up or scale-down
Horizontal: Scale-out or scale-in
Basic
Standard
Premium
Basic Basic Basic Basic Basic Basic
Premium
Standard
Scale out/in
Sca
le u
p/d
ow
n
Scalability options in Azure SQL DB
Web
Role
J
SQL
Worker
Role
J
J
Elastic Scale architecture
SQL Server VM
Run SQL on VM
Run any SQL product on cloud VM
Support for SQL Server, Oracle, MySQL
Ready to go VM images available in the Gallery
Persistent storage using attached disk in blob storage
Has all the benefits and powers of VMs combined with the full features of a SQL Engine
SQL Database SQL IaaS
Why (at least) two offerings of SQL in Azure?
SQL Database SQL IaaS
Other features SQL IaaS supports that SQL Database doesn’t (yet)
Microsoft Azure DocumentDB
Fully managed, scalable JSON document database service
Microsoft Azure Data Services
fully managed, scalable, queryable, schemafree JSON
document database service for modern applications
transactional processing
rich query
managed as a service
elastic scale
internet accessible http/rest
schema-free data model
arbitrary data formats
DocumentDB – Reasons to use
Ways to work with data
RESTful access methodsFor Create/Read/Update/Delete (CRUD) operations
DocumentDB SQLA query language with SQL-derived syntax
Executing logic in the database
Stored procedures
Triggers
User-defined functions (UDFs)
- Allow extending DocumentDB SQL
All written in JavaScript
Azure Search
Azure Search
Embed a sophisticated search experience into web and mobile applications without having to worry about the complexities of full-text search and without having to deploy, maintain or manage any infrastructure.
Azure Search
Perfect for enterprise cloud developers, cloud software vendors, cloud architects who need a fully-managed search solution.
Simple HTTP/JSON API for creating indexes, pushing documents, searching
Keyword search with user-friendly operators (+, -, *, “”, etc.)
Hit highlighting
Faceting (histograms over ranges, typically used in catalog browsing)
Search Functionality
Suggestions (auto-complete)
Rich structured queries (filter, select, sort) that combines with search
Scoring profiles to model search result relevance
Geo-spatial support integrated in filtering, sorting and ranking (such as finding all restaurants within 5 KM of your current location)
Search Functionality
What are customers doing with Azure Search?
high
high heelshigh topshigh arch
Azure Search doesn’t
provide any UI components
Users expect suggestions, so
Azure Search supports this
HDInsight
Data size
Access
Updates
Structure
Integrity
Scaling
Relational DB vs. Hadoop
Distributed Storage
(HDFS)
Query
(Hive)
Distributed Processing
(MapReduce)
OD
BC
Legend■ Core Hadoop
■ Data processing
■ Data Movement
■ Packages
Hadoop ecosystem
Hadoop Core +
Hive, Pig, HBase
C#, F#, .NET
Azure Storage (WASB)
Office 365 Power BI
(Excel, PowerQuery,
PowerView,
BI Sites)
World's Data (Azure Data
Marketplace)
HDInsight and Hadoop
ODBCSqoop for SQL
ServerPowerShell
Microsoft Azure
The Big Picture
The Big Picture
The traditional world:
The Big Picture
A modern view:
The Big Picture
Microsoft Azure technologies:
The Big Picture
Why Use a NoSQL Technology on Azure?
The Big Picture
Choosing a Data Technology
Resources
Resources
Microsoft Azure Essentials: Fundamentals of Azure
Michael S. Collier and Robin E. Shahan
February 2015
246 pages
http://bit.ly/1zRa1UW
Resources
• Azure Data Services series by Adarsha Datta
• http://bit.ly/1aTDSBA
• Get Ready for Global Azure Bootcamp
• http://bit.ly/1BkD0Lf
• Azure Documentation Center
• http://azure.microsoft.com/en-us/documentation/
• http://www.hadooponazure.com/
Top Related