DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program...

51
DAT411 DAT411 Lessons Learned: Lessons Learned: Programming and Deploying Programming and Deploying SQL Server 2000 SQL Server 2000 Replication Replication Bren Newman Bren Newman Program Manager Program Manager SQL Server Development SQL Server Development Microsoft Corporation Microsoft Corporation

Transcript of DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program...

Page 1: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

DAT411DAT411

Lessons Learned: Lessons Learned: Programming and Deploying Programming and Deploying SQL Server 2000 ReplicationSQL Server 2000 Replication

Bren NewmanBren NewmanProgram ManagerProgram ManagerSQL Server DevelopmentSQL Server DevelopmentMicrosoft CorporationMicrosoft Corporation

Page 2: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

AgendaAgenda Overview of replicationOverview of replication Replication programming modelReplication programming model Deployment scenarios and examplesDeployment scenarios and examples

Optimized reporting servers using TransactionalOptimized reporting servers using Transactional Multi-site update using TransactionalMulti-site update using Transactional Multi-site update using MergeMulti-site update using Merge DTS integrationDTS integration Snapshot generation and deliverySnapshot generation and delivery

SummarySummary Questions and AnswersQuestions and Answers

Page 3: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Overview of ReplicationOverview of Replication Keeps multiple sites in-syncKeeps multiple sites in-sync Provides increased data autonomyProvides increased data autonomy Can increase aggregate read performanceCan increase aggregate read performance Types:Types:

SnapshotSnapshot Total refresh of dataTotal refresh of data

TransactionalTransactional Log basedLog based Allows updates, but very limited conflict handlingAllows updates, but very limited conflict handling

MergeMerge Trigger basedTrigger based Focused on data convergenceFocused on data convergence Offers highest degree of autonomy and partitioningOffers highest degree of autonomy and partitioning

Page 4: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

AgendaAgenda Overview of replicationOverview of replication Replication programming modelReplication programming model Deployment scenarios and examplesDeployment scenarios and examples

Optimized reporting servers using TransactionalOptimized reporting servers using Transactional Multi-site update using TransactionalMulti-site update using Transactional Multi-site update using MergeMulti-site update using Merge DTS integrationDTS integration Snapshot generation and deliverySnapshot generation and delivery

SummarySummary Questions and AnswersQuestions and Answers

Page 5: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Replication Programming ModelReplication Programming ModelDMODMO

SQL Distributed Management SQL Distributed Management Objects (SQL-DMO) for ReplicationObjects (SQL-DMO) for Replication Visual Basic, C++Visual Basic, C++ Subset of SQL-DMO objects for SQL Subset of SQL-DMO objects for SQL

Server managementServer management Implement, Administer, Monitor Implement, Administer, Monitor

ReplicationReplication Configure Distribution, Add/Drop/Modify Configure Distribution, Add/Drop/Modify

Publication, Articles, Subscriptions. Lots Publication, Articles, Subscriptions. Lots more…more…

Page 6: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Replication Programming ModelReplication Programming ModelDMODMO

Dim oRes As QueryResultsDim oRes As QueryResults

Set oSvr = CreateObject("SQLDMO.SQLServer")Set oSvr = CreateObject("SQLDMO.SQLServer")

oSvr.Connect PubBren.Server_Nm, PubBren.User_Nm, PubBren.PswoSvr.Connect PubBren.Server_Nm, PubBren.User_Nm, PubBren.Psw

Set oRes = Set oRes =

oSvr.Replication.Publisher.EnumPublications(oSvr.Replication.Publisher.EnumPublications(

SQLDMORepType_Merge, Pub_Bren.Database, "", False)SQLDMORepType_Merge, Pub_Bren.Database, "", False)

Hierarchy of objects e.g.Hierarchy of objects e.g.

ReplicationReplication ReplicationReplicationDatabasesDatabases

ReplicationReplicationDatabaseDatabase

MergeMergePublicationsPublications

MergeMergePublicationPublication

MergeMergeArticlesArticles

MergeMergeArticleArticle

MergeMergeSubSetFiltersSubSetFilters

MergeMergeSubSetFilterSubSetFilter

Example: Enumerating Merge PublicationsExample: Enumerating Merge Publications

Page 7: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Replication Programming ModelReplication Programming ModelStored ProceduresStored Procedures

osql -S%1\%2 -E -n -dTESTDB –Qosql -S%1\%2 -E -n -dTESTDB –Q““exec exec sp_addmergepullsubscriptionsp_addmergepullsubscription @publication = @publication =N'TESTDB_%3_MPUB', @publisher = N'%1\%3', @publisher_dbN'TESTDB_%3_MPUB', @publisher = N'%1\%3', @publisher_db= N'TESTDB', @subscriber_type = N'global',= N'TESTDB', @subscriber_type = N'global',@subscription_priority = %MPRIORITY%.000000,@subscription_priority = %MPRIORITY%.000000,@sync_type = N'automatic',@sync_type = N'automatic',@description = N'Merge publication of TESTDB database from @description = N'Merge publication of TESTDB database from Publisher %1\%3.'"Publisher %1\%3.'"

Example: Adding a merge pull subscriberExample: Adding a merge pull subscriber

• Transact SQL system stored proceduresTransact SQL system stored procedures Configures replication on multiple serversConfigures replication on multiple servers Automate Replication TasksAutomate Replication Tasks SQL-DMO calls into these stored proceduresSQL-DMO calls into these stored procedures

Page 8: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Replication Programming ModelReplication Programming ModelCOM objectsCOM objects

• Replication Active X controlsReplication Active X controls Visual Basic and C/C++Visual Basic and C/C++ Snapshot, Merge and Distribution agentsSnapshot, Merge and Distribution agents Lighter weight than SQL-DMOLighter weight than SQL-DMO Add / Copy / Reinitialize SubscriptionAdd / Copy / Reinitialize Subscription Generate Initial SnapshotGenerate Initial Snapshot ““Synchronize Now”Synchronize Now” Use SQL-DMO for other Replication Use SQL-DMO for other Replication

AdministrationAdministration

Page 9: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Replication Programming ModelReplication Programming ModelCOM objectsCOM objects

• PropertiesProperties Publisher, Subscriber, Publication, PublisherLogin, Publisher, Subscriber, Publication, PublisherLogin,

SubscriberLogin, QueryTimeout, etcSubscriberLogin, QueryTimeout, etc MethodsMethods

Initialize, Run, Terminate, AddSubscription, CopySubscription, Initialize, Run, Terminate, AddSubscription, CopySubscription, ReinitializeSubscriptionReinitializeSubscription

Status CallbacksStatus Callbacks User's code is invoked to report progress messages, progress User's code is invoked to report progress messages, progress

statistics, and errorsstatistics, and errors SQLReplErrors CollectionSQLReplErrors Collection

Private Private Function SQLMerge_StatusFunction SQLMerge_Status( ByVal Message As String,( ByVal Message As String,ByVal Percent As Long) As ByVal Percent As Long) As SQLMERGXLib.STATUS_RETURN_CODESQLMERGXLib.STATUS_RETURN_CODE

AddLBMsg MessageAddLBMsg MessageSQLMerge_Status = SUCCESSSQLMerge_Status = SUCCESS

End FunctionEnd Function

Example: Status CallbackExample: Status Callback

Page 10: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Replication Programming ModelReplication Programming ModelMerge Conflict ManagementMerge Conflict Management

• Custom Conflict Resolvers (Merge)Custom Conflict Resolvers (Merge) Default is site priority based resolutionDefault is site priority based resolution Can implement custom resolver as:Can implement custom resolver as:

A COM DLL A COM DLL TSQL stored procedureTSQL stored procedure

Resolvers specified at article (i.e. table) levelResolvers specified at article (i.e. table) level User code can examine both row & metadata User code can examine both row & metadata

on pub & sub for desired resolutionon pub & sub for desired resolution Custom resolvers included with SQL Server Custom resolvers included with SQL Server

20002000 Maximum value, Minimum value, Earlier wins, Maximum value, Minimum value, Earlier wins,

Later wins, Subscriber wins, Additive, Averaging, Later wins, Subscriber wins, Additive, Averaging, Text appendText append

Page 11: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Replication Programming ModelReplication Programming ModelDistribution InterfaceDistribution Interface

• Distribution Interface (Transactional)Distribution Interface (Transactional) Allows heterogeneous data source (e.g. Allows heterogeneous data source (e.g.

Oracle, Microsoft Access) to act as the Oracle, Microsoft Access) to act as the PublisherPublisher

User implements code to read changes from User implements code to read changes from the heterogeneous data sourcethe heterogeneous data source

Call methods on Replication Distributor Call methods on Replication Distributor Interface to store replicated commands in Interface to store replicated commands in SQL Server distribution databaseSQL Server distribution database

Interface provides methods to log history Interface provides methods to log history and error informationand error information

Page 12: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Replication Programming ModelReplication Programming Model

Code ExamplesCode Examples

Page 13: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

AgendaAgenda Overview of replicationOverview of replication Replication programming modelReplication programming model Deployment scenarios and examplesDeployment scenarios and examples

Optimized reporting servers using TransactionalOptimized reporting servers using Transactional Multi-site update using TransactionalMulti-site update using Transactional Multi-site update using MergeMulti-site update using Merge DTS integrationDTS integration Snapshot generation and deliverySnapshot generation and delivery

SummarySummary Questions and AnswersQuestions and Answers

Page 14: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

AgendaAgenda Overview of replicationOverview of replication Replication programming modelReplication programming model Deployment scenarios and examplesDeployment scenarios and examples

Optimized reporting servers using TransactionalOptimized reporting servers using Transactional Multi-site update using TransactionalMulti-site update using Transactional Multi-site update using MergeMulti-site update using Merge DTS integrationDTS integration Snapshot generation and deliverySnapshot generation and delivery

SummarySummary Questions and AnswersQuestions and Answers

Page 15: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Transactional Deployment OptionsTransactional Deployment OptionsOptimized Reporting Servers Using TransactionalOptimized Reporting Servers Using Transactional

InternetInternet

https:NLB NLB NLB NLB NLB NLB

Page 16: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Transactional Deployment OptionsTransactional Deployment OptionsOptimized Reporting Servers Using TransactionalOptimized Reporting Servers Using Transactional

ExecutiveExecutiveSummarySummaryReportsReports

West CoastWest CoastLive ReportingLive ReportingDataData

East CoastEast CoastLive ReportingLive ReportingDataData

Indexed Views

Indexed Views

Filter

ed

Filter

edFiltered

Filtered

OLTP WorkloadOLTP Workload

Page 17: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Transactional ReplicationTransactional ReplicationOOptimized Reporting Servers Using Transactionalptimized Reporting Servers Using Transactional

Benefits of using new dynamic mapped Benefits of using new dynamic mapped update stored procedure for heavily update stored procedure for heavily indexed subscriber tablesindexed subscriber tables

0

200

400

600

800

1,000

1,200

1,400

1,600

1,800

2,000

cm

d/s

ec

1 index 2 indexes 4 indexes 8 indexes

Static Mapped Dynamic Update

Page 18: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

AgendaAgenda Overview of replicationOverview of replication Replication programming modelReplication programming model Deployment scenarios and examplesDeployment scenarios and examples

Optimized reporting servers using TransactionalOptimized reporting servers using Transactional Multi-site update using TransactionalMulti-site update using Transactional Multi-site update using MergeMulti-site update using Merge DTS integrationDTS integration Snapshot generation and deliverySnapshot generation and delivery

SummarySummary Questions and AnswersQuestions and Answers

Page 19: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Multi-site update using TransactionalMulti-site update using TransactionalBi-directional ReplicationBi-directional Replication

Publisher/Publisher/SubscriberSubscriber

Publisher/Publisher/SubscriberSubscriber

LocalLocal DistributorDistributor

LoopbackLoopback detection to avoid sending back detection to avoid sending back transaction to transaction to originatororiginator

Require Require conflictconflict handling if data not partitioned handling if data not partitioned If only two servers and no conflict handling – no If only two servers and no conflict handling – no

extra triggers or columns requiredextra triggers or columns required

Publisher/Publisher/SubscriberSubscriber

Publisher/Publisher/SubscriberSubscriber

or

LocalLocal DistributorDistributor

RemoteRemote DistributorDistributor

Page 20: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Publisher / Subscriber Subscriber / Publisher

Reports&

Queries

Reports&

Queries

Client reporting

Reports&

Queries

Reports&

Queries

Client reporting

Multiple High Volume Data Feeds

Multi-site replication – TransactionalMulti-site replication – TransactionalNon-partitioned bi-directionalNon-partitioned bi-directional

New

YorkSeattle

Page 21: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Multi-site replication – TransactionalMulti-site replication – TransactionalNon-partitioned bi-directionalNon-partitioned bi-directional

DEMODEMO

Page 22: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Multi-site update using TransactionalMulti-site update using TransactionalPartitioned bi-directionalPartitioned bi-directional

Publish: Publish: Site_ID = Site_ID = ‘LON’‘LON’

Subscriber to:Subscriber to:Site_ID = Site_ID = ‘NYC’‘NYC’ and and ‘PAR’‘PAR’

Publish:Publish:Site_ID = Site_ID = ‘NYC’‘NYC’

Subscriber to:Subscriber to:Site_ID = Site_ID = ‘LON’‘LON’ and and ‘PAR’‘PAR’

Publish:Publish:Site_ID = Site_ID = ‘PAR’‘PAR’

Subscriber to:Subscriber to:Site_ID = Site_ID = ‘NYC’‘NYC’ and and ‘LON’‘LON’

London

New York Paris

New YorkNew York ParisParisLondonLondon

Page 23: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Multi-site update using TransactionalMulti-site update using TransactionalImmediate Updating SubscribersImmediate Updating Subscribers Uses auto-generated Uses auto-generated triggerstriggers at subscriber at subscriber Requires additional Requires additional uniqueidentifieruniqueidentifier column column ApplicationApplication doesn’t need to understand its doesn’t need to understand its

working in a distributed environment working in a distributed environment Be aware of error code for update failure (like security, etc.)Be aware of error code for update failure (like security, etc.)

Update at a subscriber only IF accepted at Update at a subscriber only IF accepted at publisher with publisher with 2-phase commit2-phase commit If cause a conflict, it is not accepted (If cause a conflict, it is not accepted (rollbackrollback)) The The conflictconflict is is preventedprevented – not resolved. – not resolved.

If publisher is not If publisher is not availableavailable, then subscriber , then subscriber cannot update - read activity allowedcannot update - read activity allowed Not appropriate for normally disconnected usage.Not appropriate for normally disconnected usage.

Page 24: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

BuildsBuilds upon immediate updating subscribers upon immediate updating subscribers Uses Uses auto-generated triggersauto-generated triggers on subscriber on subscriber

Requires uniqueidentifier column on Pub & SubRequires uniqueidentifier column on Pub & Sub

Offline updates with Offline updates with transactional semanticstransactional semantics Updates at subscriber are posted to Updates at subscriber are posted to QueueQueue

Queue = local SQL Server Queue = local SQL Server tabletable or or MSMQMSMQ

New replication agent - New replication agent - QueueReaderQueueReader Immediate updating subscriber can Immediate updating subscriber can “fail-over”“fail-over”

to queued updatingto queued updating SimpleSimple and and Restricted Restricted ConflictConflict Handling Handling

Publisher or Subscriber wins (or re-init)Publisher or Subscriber wins (or re-init)

Multi-site update using TransactionalMulti-site update using TransactionalQueued Updating SubscribersQueued Updating Subscribers

Page 25: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Multi-site update using TransactionalMulti-site update using TransactionalImmediate & Queued Updating SubscribersImmediate & Queued Updating Subscribers

Immediate Updating Subscriber

2PC, RPC

Publisher

Distributor

Read-OnlySubscriber

Queued Updating Subscriber

NETWORK

Queue

Queue

Queue Reader Agent

LogReader

Page 26: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Multi-site replication – TransactionalMulti-site replication – TransactionalDeployment considerationsDeployment considerations Transactional Scaling using pull subscriptionsTransactional Scaling using pull subscriptions

Departmental server can service HIGH concurrent number of Departmental server can service HIGH concurrent number of subscribers (poss. 1000’s) with low CPU usagesubscribers (poss. 1000’s) with low CPU usage

Low latency can be attained (<= 6 seconds)Low latency can be attained (<= 6 seconds) Will become I/O bound prior to CPU boundWill become I/O bound prior to CPU bound Push vs. Pull Distribution AgentsPush vs. Pull Distribution Agents

Push does not scale well, but easier to manage (consider remote)Push does not scale well, but easier to manage (consider remote)

Page 27: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Scaling Queued Updating subscribersScaling Queued Updating subscribers Max of 22 concurrent threads (using SQL tables)Max of 22 concurrent threads (using SQL tables) MSMQ uses DTC for transaction boundaries (cost)MSMQ uses DTC for transaction boundaries (cost) Conflict model can affect performanceConflict model can affect performance All changes are sent down to all subscribersAll changes are sent down to all subscribers

Scaling can depend on network speed & num cmdsScaling can depend on network speed & num cmds Must customize client side stored procs for partitioningMust customize client side stored procs for partitioning

Subscribers 500 500 500 500Changes per user 100 200 100 100Total cmds p/d 50000 100000 50000 50000Rows per second 150 150 75 300Sync time (secs) 333.3 666.7 666.7 166.7

No of publications 2 10 2 2Subscribers 500 500 500 500Subs per Pub 250 50 250 250Changes per user 100 100 50 200Total cmds p/d 25000 5000 12500 50000Rows per second 150 150 150 150Sync time (secs) 166.7 33.3 83.3 333.3

Multi-site replication – TransactionalMulti-site replication – TransactionalDeployment considerationsDeployment considerations

Page 28: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Use horizontal partitioning carefullyUse horizontal partitioning carefully Each row is evaluatedEach row is evaluated Only one logreader per published databaseOnly one logreader per published database

Replication of stored procedure executionReplication of stored procedure execution Within serializable boundary or any executionWithin serializable boundary or any execution Beware script changes etcBeware script changes etc

If possible, replace user triggersIf possible, replace user triggers Add logic to generated replication SP’sAdd logic to generated replication SP’s Mark user triggers as Not For ReplicationMark user triggers as Not For Replication

If many indexes on subscriber, use new SP’sIf many indexes on subscriber, use new SP’s Use sp_scriptdynamicupdproc (SP1)Use sp_scriptdynamicupdproc (SP1)

Can use concurrent snapshotCan use concurrent snapshot Text update not supportedText update not supported

Use sp_browsereplcmds @results_tableUse sp_browsereplcmds @results_table

Multi-site replication – TransactionalMulti-site replication – TransactionalDeployment considerationsDeployment considerations

Page 29: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

AgendaAgenda Overview of replicationOverview of replication Replication programming modelReplication programming model Deployment scenarios and examplesDeployment scenarios and examples

Optimized reporting servers using TransactionalOptimized reporting servers using Transactional Multi-site update using TransactionalMulti-site update using Transactional Multi-site update using MergeMulti-site update using Merge DTS integrationDTS integration Snapshot generation and deliverySnapshot generation and delivery

SummarySummary Questions and AnswersQuestions and Answers

Page 30: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Multi-site update using MergeMulti-site update using MergeSales Force Automation TopologySales Force Automation Topology

West Coast

East CoastHQ Regional Managers

Mobile Users

Mobile UsersMobile Users Sales Office

Regional Server

Regional Server

Regional Server

Sales Office

Central Publisher

Page 31: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Multi-site update using MergeMulti-site update using MergeUpdating Online Worldwide Search ServersUpdating Online Worldwide Search Servers

The Americas

Europe

Africa, India, Middle East

EastRegional Servers• Local Edits• Local Data Loads• Offline capability

Central Server• Conflict Management• Final Edits• Central Admin of

Dynamic Subscriptions

Page 32: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Daily merge download

Driver allocated assignments

Deliversshipments

Capturedelivery details

Deliversshipments

Receives change notifications & new

instructions

Changes mergedMerge when

cradled

Multi-site update using MergeMulti-site update using MergeUpdating Online Worldwide Search ServersUpdating Online Worldwide Search Servers

Page 33: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Multi-site update using MergeMulti-site update using MergeDeployment considerationsDeployment considerations

Think “Merge” during Think “Merge” during db designdb design phase phase Investigate use of partitioned tables (updated via Investigate use of partitioned tables (updated via

triggers)triggers)

Avoid complex Avoid complex FILTERFILTER clauses clauses UDF’s and sub-selectsUDF’s and sub-selects Ensure filter columns are indexedEnsure filter columns are indexed Validate use of @join_unique_key = true/falseValidate use of @join_unique_key = true/false

Do periodic meta-data Do periodic meta-data cleanupcleanup New retention-based cleanup task in SP1New retention-based cleanup task in SP1

Beware partition Beware partition re-alignmentsre-alignments Use Publication property KeepPartitionChangesUse Publication property KeepPartitionChanges

Page 34: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Multi-site update using MergeMulti-site update using MergeDeployment considerations (filtered)Deployment considerations (filtered)

Merge

• Multiple dual processor servers with new 2.2Ghz – with large RAID etc.

• Poss. multiple publications per server

• Poss. multiple SQL instances per server

• But, more data partitioning required

• But, more complex to administrate

• 120 concurrent, filtered merge agents per server = +-600 (5 servers each servicing 120 concurrent users, synch time <= 5 minutes)

• New and impressive, but relatively inexpensive rackable hardware e.g. (5*$20,000)+(1*$70,000)= +-$170,000)

n=5

Scale-out approachScale-out approach

nc = 600 (15%)

n = 4,000

Page 35: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Merge Merge meta-datameta-data and scaling and scaling Affects merge times and disk space usageAffects merge times and disk space usage Keep retention period low for meta-data cleanupKeep retention period low for meta-data cleanup Periodic DBCC DBREINDEX/INDEXDEFRAG on meta-dataPeriodic DBCC DBREINDEX/INDEXDEFRAG on meta-data Plan ahead for extra space requirementsPlan ahead for extra space requirements

Merge Agent run times (sec)

18

25

3234

30

20

0

5

10

15

20

25

30

35

40

1st 100th 200th 300th 301st runafterDBCC

302nd runafter

Cleanup

Seco

nd

s

Example:Example: 100 merge subscribers100 merge subscribers 10 share same partition10 share same partition 30,000 merges (100 * 300)30,000 merges (100 * 300) 3,000,000 data changes 3,000,000 data changes

(100 * 100 * 300)(100 * 100 * 300) 16,000 conflicts16,000 conflicts

Multi-site update using MergeMulti-site update using MergeDeployment considerationsDeployment considerations

Page 36: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

DiskDisk considerations considerations TempDBTempDB is frequently used – use dedicated, many & fast spindles is frequently used – use dedicated, many & fast spindles Plan for extra Plan for extra meta-datameta-data storage requirements (pub &sub) storage requirements (pub &sub) Beware of Beware of RAID5RAID5 configuration - double write configuration - double write

data datalog logPub 1 Tempdb

data datalog logPub 2 Tempdb

data logDistribution Snapshot

area

db2db1SQL Instance SQL Instance

Multi-site update using MergeMulti-site update using MergeDeployment considerationsDeployment considerations

Page 37: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Clustered geographically dispersed sitesClustered geographically dispersed sites Active / Active mode - Reporting and Replication ServerActive / Active mode - Reporting and Replication Server

Transactional Stored Procedure ExecutionTransactional Stored Procedure Execution For targeted data modificationsFor targeted data modifications Stored procedures published on each serverStored procedures published on each server

Merge Replication on all tablesMerge Replication on all tables Traps ALL data modificationsTraps ALL data modifications Alternate publisher in case of connection failure etc.Alternate publisher in case of connection failure etc.

Reporting serverReporting server Transactional subscription of all tablesTransactional subscription of all tables Highly indexed for efficient reportingHighly indexed for efficient reporting

Log-shippingLog-shipping For disaster recovery to offsite locationsFor disaster recovery to offsite locations

Mixing Transactional and MergeMixing Transactional and MergeA real-world example (Sybase migration)A real-world example (Sybase migration)

Visio Diagram: TRAN_AND_MERGE_MIX.vsd

Page 38: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

DEMODEMO

Mixing Transactional and MergeMixing Transactional and MergeA real-world example (Sybase migration)A real-world example (Sybase migration)

Page 39: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

AgendaAgenda Overview of replicationOverview of replication Replication programming modelReplication programming model Deployment scenarios and examplesDeployment scenarios and examples

Optimized reporting servers using TransactionalOptimized reporting servers using Transactional Multi-site update using TransactionalMulti-site update using Transactional Multi-site update using MergeMulti-site update using Merge DTS integrationDTS integration Snapshot generation and deliverySnapshot generation and delivery

SummarySummary Questions and AnswersQuestions and Answers

Page 40: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

TransformTransform or Reshape data or Reshape data Datatype mappingDatatype mapping Column renaming/reformatting/concat Column renaming/reformatting/concat Useful for heterogeneous OLE-DB subscribersUseful for heterogeneous OLE-DB subscribers

Dynamic filtersDynamic filters: subscriber specific : subscriber specific vertical and/or horizontal partitionsvertical and/or horizontal partitions Each subscriber defines custom transformation on a Each subscriber defines custom transformation on a

single publicationsingle publication Reduces number of publications - dynamicReduces number of publications - dynamic Transform performed at distributor or subscriber – Transform performed at distributor or subscriber –

network speed vs. distributor resourcesnetwork speed vs. distributor resources

Transactional – DTS IntegrationTransactional – DTS Integration

Page 41: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Transactional – DTS IntegrationTransactional – DTS Integration

Publisher

Distributor

Subscriber

Log Reader Agent

Snapshot Agent

DTS Pump Task

Distribution Agent

DTS Package

Page 42: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

AgendaAgenda Overview of replicationOverview of replication Replication programming modelReplication programming model Deployment scenarios and examplesDeployment scenarios and examples

Optimized reporting servers using TransactionalOptimized reporting servers using Transactional Multi-site update using TransactionalMulti-site update using Transactional Multi-site update using MergeMulti-site update using Merge DTS integrationDTS integration Snapshot generation and deliverySnapshot generation and delivery

SummarySummary Questions and AnswersQuestions and Answers

Page 43: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Investigate using -MaxBCPThreadsInvestigate using -MaxBCPThreads Merge – Powerful Dynamic SnapshotMerge – Powerful Dynamic Snapshot

xp_enumntusers and xp_enumntusers and sp_addntgroupmergedynsnapshotjobsp_addntgroupmergedynsnapshotjob Generate snapshots based off GroupsGenerate snapshots based off Groups

Can use multiple concurrent jobsCan use multiple concurrent jobs

Compressed SnapshotCompressed Snapshot FTP supportFTP support Alternative media supportAlternative media support Concurrent Snapshot – TransactionalConcurrent Snapshot – Transactional Pre/Post Snapshot scriptsPre/Post Snapshot scripts

Snapshot Generation and ApplicationSnapshot Generation and ApplicationDeployment considerationsDeployment considerations

Page 44: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

AgendaAgenda Overview of replicationOverview of replication Replication programming modelReplication programming model Deployment scenarios and examplesDeployment scenarios and examples

Optimized reporting servers using TransactionalOptimized reporting servers using Transactional Multi-site update using TransactionalMulti-site update using Transactional Multi-site update using MergeMulti-site update using Merge DTS integrationDTS integration Snapshot generation and deliverySnapshot generation and delivery

SummarySummary Questions and AnswersQuestions and Answers

Page 45: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

SummarySummaryOffline usageOffline usage

Merge replicationMerge replication Emphasis on convergenceEmphasis on convergence Excellent conflict handlingExcellent conflict handling Exceptionally feature rich Exceptionally feature rich

for offline usagefor offline usage Powerful partitioning Powerful partitioning

functionalityfunctionality Requires thorough Requires thorough

understanding of merge understanding of merge usage/design and usage/design and application integrationapplication integration

Can be difficult to scale Can be difficult to scale performance when high performance when high concurrent usage of concurrent usage of partitioned publicationspartitioned publications

Transactional (Queued)Transactional (Queued) Can be used with immediate Can be used with immediate

updating (failover)updating (failover) Poss. integration with MSMQPoss. integration with MSMQ Very limited conflict Very limited conflict

handlinghandling No text/image supportNo text/image support Transactional (bi-di)Transactional (bi-di) Very low latency achievableVery low latency achievable Can solve geographically Can solve geographically

dispersed data duplication dispersed data duplication needsneeds

Conflict handling must be Conflict handling must be programmed/codedprogrammed/coded

Difficult to scale (manage)Difficult to scale (manage)

Page 46: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

SummarySummaryKey SQL 2000 Replication SP fixesKey SQL 2000 Replication SP fixes

MergeMerge Automatic retention-based meta-data cleanupAutomatic retention-based meta-data cleanup Reduced subscriber GenHistory meta-dataReduced subscriber GenHistory meta-data Reduced deadlocks/contention and expansionReduced deadlocks/contention and expansion Many others….Many others….

TransactionalTransactional Dynamic Update SP Dynamic Update SP (sp_scriptdynamicupdproc)(sp_scriptdynamicupdproc)

LogReader supports LogReader supports (–MaxCmdsInTran)(–MaxCmdsInTran)

Bounded singleton updates Bounded singleton updates (-T8207)(-T8207)

Subscriber custom SP script Subscriber custom SP script (sp_scriptpublicationcustomprocs)(sp_scriptpublicationcustomprocs)

Page 47: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

SummarySummaryResourcesResources SQL User Group and event (SQL User Group and event (www.sqlpass.orgwww.sqlpass.org)) SQL Server Magazine (SQL Server Magazine (www.sqlmag.comwww.sqlmag.com)) Replication docs on Replication docs on www.microsoft.com/sql/techinfo/www.microsoft.com/sql/techinfo/

Transactional Performance and TuningTransactional Performance and Tuning .../.../administration/2000/replication_performance_tuning.docadministration/2000/replication_performance_tuning.doc

Diagnosing and Troubleshooting Slow Partitioned Diagnosing and Troubleshooting Slow Partitioned MergeMerge …/development/2000/MergeProcesses.doc…/development/2000/MergeProcesses.doc

Middle Tier Application Data Caching …/development/2000/…/development/2000/SqlCacheWp.docSqlCacheWp.doc

Non-Microsoft-Sponsored NewsgroupsNon-Microsoft-Sponsored Newsgroups microsoft.public.sqlserver.replicationmicrosoft.public.sqlserver.replication

Page 48: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Resources from Microsoft PressResources from Microsoft Press

For more information please visit the TechEd Bookshop.For more information please visit the TechEd Bookshop.www.microsoft.com/mspresswww.microsoft.com/mspress

SQL SERVER 2000

Page 49: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

CONNECTCONNECT at Europe’s Premier Conference for at Europe’s Premier Conference for Planning, Deploying and Managing a Connected Planning, Deploying and Managing a Connected InfrastructureInfrastructure

…….register now & SAVE.register now & SAVE www.microsoft.com/europe/itforum/www.microsoft.com/europe/itforum/

CHOOSECHOOSE from over 140 technical breakout sessionsfrom over 140 technical breakout sessions HEARHEAR from Microsoft executives and technical expertsfrom Microsoft executives and technical experts GAINGAIN best practices for managing your systemsbest practices for managing your systems ATTENDATTEND extensive hands-on training labsextensive hands-on training labs EXPLOREEXPLORE the hottest new tools and technologiesthe hottest new tools and technologies

Page 50: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.

Don’t forget to complete the Don’t forget to complete the on-line Session Feedback form on-line Session Feedback form on the Attendee Web siteon the Attendee Web site

https://web.mseventseurope.com/teched/https://web.mseventseurope.com/teched/

Page 51: DAT411 Lessons Learned: Programming and Deploying SQL Server 2000 Replication Bren Newman Program Manager SQL Server Development Microsoft Corporation.