Implementing Scale-Out Solutions with Microsoft SQL Server 2005
description
Transcript of Implementing Scale-Out Solutions with Microsoft SQL Server 2005
DAT309
Implementing Scale-Out Solutions with Microsoft SQL Server 2005
Don VilenProgram ManagerSQL ServerMicrosoft Corporation – Redmond
Who Am I?
Program Manager on SQL Server’s Storage EngineWorking on SQL Server at Microsoft for 12+ yearsResponsible for Database Mirroring and High AvailabilityPreviously:
PM for Filestream, other features
Training for PSS Engineers on SQL Server around the world
Technologies to Implement Scale-out
Which Technology to Use?
Classes of Data and Scale-out
Deployment Scenarios
Use Cases
Agenda
Scaling Database Systems
This presentation is about scalingSecond 10,000 users as fast as first 10,000With twice as much horsepower – or less
Two ways to scaleScale up: bigger hardware, softwareScale out: more small boxes
Scale-out has potential savingsScale-up: redundant big serverScale-out: 100% redundancy not required
Scale-Out For Everyone?
Scale-out is not the answer to every problemSimilar to dividing processing among threadsIf scale is not required, solution will just be:
Unnecessarily more complex
Slower
Some scaling technologies require changingPrograms
Procedures
Best to plan this into the design ahead of time
We’ll assume you’ve addressed basic tuningAppropriate indexes, stored procs, etc.
The Technologies
Linked Servers and Distributed Queries
DPV Distributed Partitioned View
SSD Scalable Shared Database
Replication
P2P Peer-to-Peer Replication
SSB Service Broker – Queuing
QN Query Notifications – Caching
DDR Data Dependent Routing
Linked Servers and Distributed Queries
Define Linked Servers and connection attributesUse 4-part names to reference <server>.<database>.<schema>.<object>Query optimizer will split query or DML into appropriate queries to be sent to each linked serverEach linked server processes its query and returns result setResult sets are combined / joined Distributed transactions for DML
Linked Servers and Distributed QueriesExecuting a Query
1. User issues query / DML with DPV
2. Sends appropriate query / DML to linked servers, eliminating those with no relevant data
3. Queries run asynchronously on involved linked servers
4. Data is returned by linked servers and combined
Distributed Partitioned View – DPV
Define sub-tables with appropriate constraints on the various Linked ServersCreate a UNION-ALL view SELECTing from all the sub-tablesQuery optimizer will split query or DML into appropriate queries to be sent to each linked server, with dynamic elimination of uninvolved partitionsEach linked server processes its query and returns result setResult sets are combined / joined Distributed transactions for DML
Distributed Partitioned View – DPV Setting it up
1 - 999 1000 - 1999 2000 - 2999
1. Create tables with range constraints
2. Create linked servers
3. Create view with UNION ALL
Distributed Partitioned ViewsExecuting a Query
1 - 999 1000 - 1999 2000 - 2999
1. User issues query / DML with DPV
2. Sends appropriate query / DML to linked servers, eliminating those with no relevant data
3. Queries run asynchronously on involved linked servers
4. Data is returned by linked servers and combined
Distributed Partitioned View
Performs the work across involved nodes asynchronouslyAdd additional nodes as needed
How it Provides Scale-out
1 - 999 2000 - 29991000 - 1999 3000 - 3999
Distributed Partitioned Views
Available since SQL Server 7.0SQL Server 2005 adds the asynchronous processing
Usable for read or writeRequires MSDTC for distributed transactionsConsiderations for scalability
All queries and updates are affectedResponse is determined by the slowest partLocks are held for whole transactionNot feasible across security/organization boundaries
Scalable Shared Database – SSD
One copy of database attached to multiple instancesNew in SQL Server 2005 – Enterprise Edition onlyRequires a SANDatabase itself is read-only
Refresh on a schedule
Rolling refresh possible
Sweet spot is connection by 2 to 8 instances
Scalable Shared Database – SSD Setting it up
1. Create database, fill with data, and detach
2. Set volumes Read Only
3. Attach the same database to multiple instances
4. Periodically attach refreshed database to instances, perhaps via SAN operations
Scalable Shared Database – SSD Executing a Query
User queries on multiple instances use same reference data from single copy
Scalable Shared Database – SSD Refreshing the Data
1. Create a Clone of hot data, with recoveryHot Data
2. Set Clone volume to Read Only
3. For each node: Detach Instance from Vn, and attach Vn+1
Clone
Vn+1 Vn 4. Refresh Vn, and create Vn+2
Scalable Shared Database
Multiple instances access the same database filesAdd additional nodes as needed
How it Provides Scale-out
Setting Up SSD
Build processUnmask SAN unit number to make volume accessibleMount volume as read-write with DISKPARTBuild or refresh one copy of the database
Attach processDetach the database from the SQL Server instance that built it, using the DETACH command
Mark each volume as read-only by using DISKPART and dismount the disk volume from the production server
Unmask the SAN logical unit number to make the database available to each reporting server
Mount the disk volume on each reporting server by using DISKPART
Attach to one or more instances of SQL Server
Replication
Provides for distribution of all or subset of dataAvailable with various capabilities since SQL Server 7.0All use Publisher – Distributor – Subscriber modelAll incur latency; data changes appear some time, perhaps seconds, minutes, hours, later
Snapshot ReplicationMerge ReplicationTransactional Replication
Snapshot Replication
Periodic refreshing of whole sets of dataEquivalent to BCP-out from Publisher and BCP-in to Subscriber(s)Perhaps some rows haven’t changed at all; others have changed perhaps millions of times
Publisher / Distributor
Subscriber(s)
Mechanism: BCP to copy whole sets of data
Merge Replication
Periodic exchange of changed rows in both directionsChanged rows are tracked; net changes are sent
No guarantee of transactional consistency
Can update in multiple placesBuilt-in conflict detection
Publisher / Distributor
Subscriber(s)
Mechanism: Track changed rows, then exchange net changes
Transactional ReplicationPeriodically sends individual changes, maintaining transactional integrityChanges are read from log, converted into individual T-SQL statements, and sent preserving order and transaction grouping
Guarantees transactional consistency
Publisher / Distributor
Subscriber(s)
Mechanism: Translate each log record to T-SQL, send in same order, and with similar transactions
Peer-to-Peer Replication – P2P
Variety of transactional replicationNew in SQL Server 2005 – Enterprise Edition only
All Publish their ‘owned’ data and subscribe to others’ data
Peer-to-Peer Replication – P2P
Variety of transactional replicationNew in SQL Server 2005 – Enterprise Edition only
Allows ring-type replication configurationMore flexible than hierarchical topologies
Allows Read and Write operations
More replication paths than bus-typeNo built-in conflict detection
Each row must have an "owner"
Replication Topologies
P2P and Connectivity
Peer-to-Peer Replication – P2P
Clients work with a server with a ‘local’ copy Add additional nodes to increase capacityAlso increases availability
How it Provides Scale-out
All Publish their ‘owned’ data and subscribe to others’ data
SQL Server Service Broker – SSB
Transactional messaging, asynchronous processing, and disconnected systems
Separates input from executionRouting among instances is built inProvides service-oriented infrastructure
Available in all SQL Server 2005 editionsEven SQL Server Express
Requires application changes / programming
Service Broker – SSB Setting it up
1. Enable Service Broker
2. CREATE QUEUE
3. Rather than performing all the processing, it SENDs a message
4. Message is routed to receiver who RECEIVEs it from the queue and processes it
Service Broker – SSB
Distributes the workProcessing is done elsewhere and/or laterCan specify an Activation procedure and number of instancesCan further divide the work across multiple servicesAdd more resources as needed
How it Provides Scale-out
Query Notifications – QN Notify my client / cache when row is changed
Notification only; client must refresh cached data
Implemented in-server using:Indexed View-based notification technologyService Broker for communications
Requires SQL Server 2005 – any editionRequires "new" client stackUsed in ASP.NET "Cache Sync"
Client cache
1. Client caches data and registers for notification
3. SSB message notifies client of a change to result set
2. Data is changed on server
Query Notifications – QN
Intelligent cache notifications avoid continuous polling to check for changesEffectively “scales-out” to the clients
How it Provides Scale-out
Client cache
Data Dependent Routing – DDR More architectural concept than featureNot built in to SQL Server 2005Can be implemented:
By application
With SSB using built-in routing service with custom code
With middle-tier component
Can be very helpful with: Transactional replication
Distributed Partitioned Views
Which Technology?
DPV SSD P2P SSB QN
Special Hardware No Yes No No No
Program Changes No No No Yes Maybe
OpsChanges Maybe Yes Yes No No
Read-Write Yes No Per-Row Yes Yes
IncreaseRobustness No Maybe Yes Yes No
DDRUseful Yes Maybe Yes Maybe Maybe
SQL ServerEditions No Express EE only EE only All* All
Technologies to Implement Scale-out
Which Technology to Use?
Classes of Data and Scale-out
Deployment Scenarios
Use Cases
Agenda
Data Usage-Based Scale-Out
Scaling choice is based on data usageClassify the dataDetermine the database deployment topologyDetermine the application topologyChoose the appropriate technology
Classifying Data
Four main data classificationsResource data: customers
Traditional "data processing" data
Reference data: catalogs, price listsLookup tables
Service interaction data Original order forms
Activity data Pick lists
Shopping carts
Traditional Deployment
Single-tier or client-serverAll data together in databaseAll data treated the sameAll data updated directlyService interaction is part of transaction
Deployment in Tiers
Multi-tiersReference Data can be cached and replicatedActivity and Service Interaction coupled
Reside in application servers
Direct interaction between activity, service, and resourceMulti-operation transaction
Deployment with Services
Some data separated into servicesActivity data internal but asynchronous
Not tightly coupled to resource data
Managed through messaging
Service interactions can be disconnectedManaged through messaging
Some data replicatedTraditional transactional replicationSmart, active cache
Only resources need synchronous processing
Scaling and Data Types
Techniques differ with dataReference data
Scalable Shared Database
Snapshot replication
Query notifications
Activity data Dynamic partitioning
Data dependent routing
Service interaction data Inherently asynchronous
Resource data Static partitioning
Peer-to-peer or bi-directional replication
Storage and Data Types
Techniques differ with dataReference data
Master copy in central database
Replicas online
Activity dataKeep copies for activity tracking, trending, mining
Service interaction dataKeep EXACT copies of request and response
For retry and recovery
Sometimes a legal requirement
Resource dataDivide processes
Updates can be async with other categories
Use CasesDistributed Partitioned Views
“SQL Server Megaservers: Scalability, Availability, Manageability” (Jim Gray and Richard Waymire)http://www.microsoft.com/technet/prodtechnol/sql/2000/plan/ssmsam.mspx
Scalable Shared DatabaseCorporate Phone Book, Product Catalog
Peer-to-Peer Replication“Inside Microsoft.com: Managing SQL Server 2005 Peer-to-Peer Replication” (David Lindquist)http://www.microsoft.com/technet/technetmag/issues/2006/07/InsideMSFT/default.aspx
Service Broker“Inside MSDN: Building the MSDN Content Aggregation System”(John Mollman)http://msdn.microsoft.com/msdnmag/issues/06/07/InsideMSDN/default.aspx
Query NotificationsCache information about server status, Windows mobile server update
Summary
Scale-out more achievable with SQL Server 2005Multiple features geared toward scale-outService-oriented, replicated, or distributed
Which technologies to use depends on:Deployment scenariosWhether app or process can be changedData classes used
Scale-out technologies can be combined Data Dependent Routing useful adjunct to all
Resources“Scaling Out SQL Server 2005” Roger Wolterhttp://msdn2.microsoft.com/en-us/library/aa479364.aspx
“Planning, Implementing, and Administering Scaleout Solutions with SQL Server 2005” (Bob Beauchemin)http://www.microsoft.com/technet/prodtechnol/sql/2005/implementingscaleout.mspx
Bob Beauchemin's Blog: http://www.SQLskills.com/blogs/bobb
“Architecting Service Broker Applications” (Roger Wolter)http://msdn2.microsoft.com/en-us/library/aa964144.aspx
“Achieving Scalability and Availability with Peer-to-Peer Transactional Replication” (Michael Blythe)http://www.microsoft.com/technet/prodtechnol/sql/2005/p2ptranrepl.mspx
Scalable Shared Databases Are Supported by SQL Server 2005http://support.microsoft.com/default.aspx?scid=kb;en-us;910378
“Service-Oriented Data Architecture: App-Server:Lite?” (David Campbell)http://research.microsoft.com/research/pubs/view.aspx?tr_id=983
“Data on the Outside vs. Data on the Inside: An Examination of the Impact of Service Oriented Architectures on Data” (Pat Helland)http://msdn2.microsoft.com/en-us/library/ms954587.aspx
Q&A
Resources
Technical Communities, Webcasts, Blogs, Chats & User Groupshttp://www.microsoft.com/communities/default.mspx
Microsoft Developer Network (MSDN) & TechNet http://microsoft.com/msdn http://microsoft.com/technet
Trial Software and Virtual Labshttp://www.microsoft.com/technet/downloads/trials/default.mspx
Microsoft Learning and Certificationhttp://www.microsoft.com/learning/default.mspx
SQL Down Under Code Camp
Biggest SQL Server community event in Australia14 SQL Server sessions over two daysSpecial guest presentation from Kevin Kline
President of PASS, Director of Technology – Quest Software
When?Weekend of the 13th and 14th of October
Where?Charles Sturt University, Wagga Wagga
What does it cost?This event is FREE
http://www.sqldownunder.com
SQL Server User Groups
Best place to learn about SQL Server for FREEUser Groups in most capital citiesGreat networking opportunityhttp://www.sqlserver.org.au
Call to Action
Visit the TLC Area to talk to SQL Server experts
Download the latest SQL Server “Katmai” CTP from: http://www.microsoft.com/sql/prodinfo/futureversion/default.mspx
SQL Server TechCenter: http://www.microsoft.com/technet/prodtechnol/sql/default.mspx
SQL Server DevCenter:http://msdn.microsoft.com/sql
SQL Server Best Practices:http://www.microsoft.com/technet/prodtechnol/sql/bestpractice/default.mspx
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.