Microsoft SQL Server Compact Edition Ultimate Performance Tuning Darren Shaffer

Microsoft MVPHandheld LogicWMB403

Agenda

SQL Compact Edition ArchitectureSQL Compact Edition Sweet SpotMeasuring & Understanding PerformanceOptimal Query & DML PerformanceHigh-Performance DevelopmentOptimal Data Synchronization

Managed StackNative Stack

Storage Engine / Replication Tracking

Query Processor

SQL Server CE Data Provider

OLEDB CE

VS 2005/2008 (C++)

OLEDB Provider

SQL Compact Edition Architecture

CLR / .NET CF

ADO.NET

VB.NET & C#

SQL CE Query Processor

A Heuristic, Optimizing QP in under 1MB!Heuristic

Rewrites query into semantically equivalent form that leads to a better execution planBased on syntaxAdjacent inner joins are merged together so alternative join orders may be considered

... FROM (Table_1 INNER JOIN Table_2 ON Table_1.Col = Table_2.Col) INNER JOIN Table_3 ON Table_1.Col = Table_3.Col ...... FROM Table_1, Table_2, Table_3 WHERE Table_1.Col = Table_2.Col AND Table_1.Col = Table_3.Col ...

Cost-Based OptimizationDetermines

Base table scan type (File scan / Index scan)What indexes, if any, are usedJoin order, join algorithmSort / filter placement

How it worksEnumerates a selected subset of all possible execution plansFinds out the plan with lowest estimated costGenerates executable data structures that implement the plan

SQL CE Storage Engine

Completely New in v3.1 4KB Page SizeACID Transaction SupportSmart Device AwarenessRow-Level LockingAutomatic Reuse of Empty PagesImproved Tools for Database Health and Maintenance

ManagementTools

Communicationsand Messaging

Device Update AgentSoftware Update Services

Live Communications ServerExchange ServerInternet Security and Acceleration Server

Speech Server

Image Update

Location Services

Multimedia

MapPoint

DirectXWindows Media

Visual Studio 2005/2008Development Tools

MFC 8.0, ATL 8.0Win32Native

Managed

Server Side

Lightweight

Relational

SQL Server 2005 Express EditionEDB

Dat

aPr

ogra

mm

ing

Mod

el

Device Building Tools

Hardware/Drivers

Windows XP DDKWindows Embedded Studio

Platform Builder

OEM/IHV Supplied BSP(ARM, SH4, MIPS)

OEM Hardware and Standard Drivers

Standard PC Hardware and Drivers

SQL Server Compact Edition

ASP.NET Mobile Controls ASP.NET

.NET Compact Framework .NET Framework

Microsoft Operations ManagerSystems Management Server

SELECT IMEI, ProductCode, Quantity FROM (SELECT NULL AS IMEI, product ASProductCode, (physicalqty - allocatedqty) AS Quantity FROM importstockWHERE (NOT mpstype IN(N'U', N'C', N'M', N'X', N'Y', N'P')) AND product IN(SELECT ProductCode FROM (SELECT importstock.product AS ProductCode FROMStockCountSchedule INNER JOIN StockCountProductCategories ON(StockCountSchedule.ID = StockCountProductCategories.ID) INNER JOINimportstock ON (StockCountProductCategories.Product_Type =importstock.product_type) WHERE (StockCountSchedule.IsRecount = 0) AND(StockCountSchedule.ID = 121231) UNION SELECT ProductCode FROMStockCountSchedule INNER JOIN CrossDevice_ProductsToRecount ON(StockCountSchedule.ID = CrossDevice_ProductsToRecount.StockCountID) WHERE(StockCountSchedule.IsRecount = 1) AND (StockCountSchedule.ID = 121231)) ASStockCountProducts) UNION SELECT IMEI.imei AS IMEI, NULL AS ProductCode,NULL AS Quantity FROM importstock INNER JOIN IMEI ON importstock.product =IMEI.product WHERE (mpstype IN(N'U', N'C', N'M', N'X', N'Y', N'P')) ANDimportstock.product IN (SELECT ProductCode FROM (SELECTStockCountSchedule.ID AS StockCountID, importstock. product AS ProductCodeFROM StockCountSchedule INNER JOIN StockCountProductCategories ON(StockCountSchedule.ID = StockCountProductCategories.ID) INNER JOINimportstock ON (StockCountProductCategories.Product_Type =importstock.product_type) WHERE (StockCountSchedule.IsRecount = 0) UNIONSELECT StockCountSchedule.ID AS StockCountID, ProductCode FROMStockCountSchedule INNER JOIN CrossDevice_ProductsToRecount ON(StockCountSchedule.ID = CrossDevice_ProductsToRecount.StockCountID) WHERE(StockCountSchedule.IsRecount = 1)) AS StockCountProducts)) ASStockCountItems

“Runs fine on SQL Server 2000, but not under SQL CE…”

Actual SQL CE Newsgroup Post:

SQL CE vs. SQL ExpressThe real story…

Footprint vs. capabilitySchema complexityQuery complexityOff-line experienceData synchronizationSecurityEase of deploymentMindset

Measuring Performance

Code instrumentationSystem.Diagnostics.StopWatch (new in NET CF 3.5)System.Environment.TickCount (.1 - .5 sec resolution)System.DateTime.Now (1 sec or worse resolution)

.NET CF profiling toolsSystem Information on SOTI’s PocketControllerHKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETCompactFramework\PerfMonitor\ set (DWORD) Counters = 1 Performance Countershttp://blogs.msdn.com/davidklinems/archive/2005/12/09/502125.aspx

Remote Performance Monitor (.NET CF 3.5 PowerToys)

http://www.microsoft.com/downloads/details.aspx?FamilyID=c8174c14-a27d-4148-bf01-86c2e0953eab&displaylang=en

SQL Server Management StudioDisplay Query Execution Plan

Discard the first measurementTake sufficient number of samplesRestart the application between testsUse realistic data, devices, and storage

Measuring PerformanceMeasurement tips

Measuring SQL CE PerformanceDarren ShafferMicrosoft MVPHandheld Logic

demo

Match schema to SQL CE’s capabilitiesBase Table Cursors/TableDirectSqlCeResultSet versus SqlCeDataAdapterLeverage useful indexes

Optimal Query PerformanceBig hitters

Minimize column count on tablesUse variable length columns, narrow as possibleAvoid Using: Max Footprintntext 536M 2n bytesnvarchar/nchar(big #) 4000 2n bytesvarbinary/binary(big #) 8000 1n bytes

Consider de-normalizing schemaKeep JOINs to no more than 3 or 4 if possible

Pre-calculate, pre-aggregate when possible

Schema Tips

seconds

8.94

3.97

On The Fly Pre-Computed

SELECT OrderID, SUM(UnitPrice * Quantity * (1.0 - Discount)) AS Total FROM OrderDetails GROUP BY OrderID

SELECT OrderID, OrderTotal AS Total FROM Orders

Pre-Computing Impact

Base Table Cursors/TableDirect

Example:

// create and execute SqlCeCommandSqlCeCommand cmd = new SqlCeCommand(“Authors",cnn);cmd.CommandType = CommandType.TableDirect;SqlCeDataReader dr = cmd.ExecuteReader();

// process results as usual while(dr.Read()){ MessageBox.Show("Name = " + dr["au_lname"]);}dr.Close();dr.Dispose();

Bypasses the query processorReturns all columns in a rowFastest way to read from a table when you need all columns

Seek/SetRange

Example:

cmd.CommandType = CommandType.TableDirect;cmd.CommandText = "Orders";cmd.IndexName = "idxDateTime"; object[] start = new object[1];object[] end = new object[1];start[0] = new SqlDateTime(2007, 1, 1); end[0] = new SqlDateTime(2008, 2, 3;

cmd.SetRange(DbRangeOptions.Match, start, end); SqlCeDataReader dr = cmd.ExecuteReader(); dr.Seek(DbSeekOptions.FirstEqual, new SqlDateTime(2007,3,4));

while(dr.Read()) {// process results as usual }

Bypasses the query processorOpen a base table indexFastest way to select a range of values

10 Iterations (sec) 100 Iterations (sec) 1000 Iterations (sec)2.3 0.289

2.5920.642000000000003

5.069

47.904

Base Table Seek versus Query

Base Table Seek SELECT Query

Base Table Cursors/TableDirect

private SqlCeResultSet resultSet = null;private ResultSetView view1 = null;

private void _bindData(){ this.command.CommandText = “SELECT * FROM Orders”; ResultSetOptions options = ResultSetOptions.Scrollable |

ResultSetOptions.Updatable;

this.resultSet = this.command.ExecuteResultSet(options); this.view1 = this.resultSet.ResultSetView; int[] ordinals = new int[] { 1,3,5,8 }; this.view1.Ordinals = ordinals; this.dataGrid.DataSource = view1;}

Query Performance Similar to SqlCeDataReaderExcellent DML PerformanceBi-Directional Scrolling and Update in Place

SqlCeResultSet

seconds

1.759

1.079

SqlCeDataAdapter SqlCeResultSet

SELECT * FROM Orders SqlCeDataAdapter.Fill(DataSet)

SELECT * FROM Orders cmd.ExecuteResultSet(ResultSetOptions)

ResultSet Vs. DataAdapter1,078 Orders

Scrollability

Example:

// Set the index range cmd.SetRange(DbRangeOptions.InclusiveStart, start, end); SqlCeDataReader dr = cmd.ExecuteReader();

// Seek to a value (customer name) dr.Seek(DbSeekOptions.FirstEqual,”Shaffer”);dr.Read();// Process the row // Seek to another value (customer name) dr.Seek(DbSeekOptions.LastEqual,”Snerdley”);dr.Read();// Process the row

SqlCeDataReader is forward-onlySqlCeDataAdapter SELECT does not leverage SetRange/SeekSqlCeResultSet is scrollable & fast. This is even faster:

Useful = Selective and Chosen by the Query ProcessorSelectivity is ratio of qualifying rows to total rows (low is good)Index on Orders.OrderID is selectiveIndex on Orders.ShipVia is not selectiveUse sp_show_statistics_steps ‘table’, ‘index’

SQL CE uses only one index per table in an execution planIndexes increase database sizeAvoid indexing small tables; table scan is more efficientHeavy DML, use fewer indexesHeavy querying, use more indexesMax of 249 indexes per table, 16 columns per index

Leverage Useful IndexesAren't they all useful?

seconds

.265

0.065

No Index Indexed

SELECT OrderID, ProductID FROM OrderDetails WHERE OrderID = 10900

ADD INDEX on OrderID

Impact of Indexes2,820 OrderDetails records

Limit Requested Columns (70% better)

SELECT * FROM SELECT ColumnName FROM

Write SARGABLE Clauses (55% better)

SELECT OrderID FROM Orders WHERE DATEPART(YEAR, OrderDate) = 1992 AND DATEPART(MONTH, OrderDate) = 4

SELECT Order ID FROM Orders WHERE OrderDate >= '04/01/1992' AND OrderDate < '05/01/1992‘

NON-SARGABLE Clauses: IS NULL, <>, !=, !>, !<, NOT, NOT EXISTS, NOT IN, NOT LIKE, LIKE %ABCD

Query PerformanceMiscellaneous recommendations

JOINs versus SUBQUERIES (88% better)

SELECT OrderID FROM Orders O WHERE EXISTS (SELECT OrderID FROM OrderDetails OD WHERE O.OrderID = OD.Order ID AND Discount >= 0.25)

SELECT DISTINCT O.OrderID FROM Orders O INNER JOIN OrderDetails OD ON O.OrderID = OD.OrderID WHERE Discount >= 0.25

Query PerformanceMiscellaneous recommendations

Avoid Redundant DISTINCT (55% better)

SELECT DISTINCT C.CustomerID, O.OrderID FROM Customers C INNER JOIN Orders O ON C.CustomerID = O.CustomerID

SELECT C.CustomerID, O.OrderID FROM Customers C INNER JOIN Orders O ON C.CustomerID = O.CustomerID

Use GetValues() versus GetXXX() (17% better)

Index WHERE, ORDER BY, GROUP BY Columns (39% better)

Query PerformanceMiscellaneous recommendations

Optimum Query PerformanceDarren ShafferMicrosoft MVPHandheld Logic

demo

Pre-Load reference tablesParameterized DML Queries (specify precision)SqlCeResultSet update-in-placeConsider removing/re-adding indexesIncrease MaxBufferSizeUse faster storage

Optimal DML PerformanceBig hitters

Optimum DML PerformanceDarren ShafferMicrosoft MVP, Chief Software ArchitectHandheld Logic, LLC

demo

Version of SQL Compact EditionDeployment PlatformSQL CE Connection StringConnection cachingEncryptionOptions for deploying the “Starter Database”Database maintenanceRecovery planning

High-Performance DevelopmentConsiderations

Put SQL in the UI layerConcatenate a bunch of strings to form SQLUse string. Replace for parameter valuesUse SqlCeDataAdapter (period)Forget to close and dispose of SqlCeDataReadersForget to dispose of SqlCeCommandsForget to use SqlCeTransactions for DMLShow users SqlCeExceptionsOne connection, > 1 Thread

High-Performance DevelopmentWhat not to do…

High-Performance DevelopmentClient Application

Presentation

Logic

Data Access

DB Manager

SQL Compact

High-Performance DevelopmentDarren ShafferMicrosoft MVP, Chief Software ArchitectHandheld Logic, LLC

demo

Batch-Mode SDF File ExchangeRemote data accessMerge replicationCustom web servicesSync services for ADO.NET (Devices)

Optimal Data Synchronization Options

New!

The Secret to Data Sync SuccessMake a plan!

Database Table Purpose Table Needed in Local Cache?

All Columns Required?

Primary Key

Indexes Required?

Filtering Possible?

Lookup/Reference only?

Add/Change/Remove Endpoints

Conflict Potential

Business Logic?

CreditCardTypes Enumerate valid credit card types

Yes Yes GUID No No Yes server none N/A

NewSubscriptions New subscriptions are stored here

Yes Yes GUID No Yes No device only none Yes - authorize credit card on delivery to server

PostalCodes Reverse Lookup City and State based on entering Zip Code

Yes No (can omit Country)

GUID Yes (PK is sufficient)

No Yes server none N/A

Products Newspapers and periodicals which can be subscribed to

Yes No (can omit PublicationTypeFk)

GUID No No Yes server none N/A

ProductsPromotions Join table Yes Yes GUID No No Yes server none N/A

Promotions A PremiumPack offered for a specific time period

Yes Yes GUID No No Yes server none N/A

PublicationTypes Enumerate valid publication types

No N/A N/A N/A N/A N/A N/A N/A N/A

Signatures Subscriber's digital signature

Yes Yes GUID No Yes No device only none N/A

Users Authenticate the mobile user

Yes No (can omit Name)

GUID No Yes Yes server none N/A

Vendors Each user works for a Sales Vendor

Yes Yes GUID No Yes Yes server none N/A

VendorsPromotions Join Table Yes Yes GUID No Yes Yes server none N/A

CRITERIA BEST CHOICE VIABLE CHALLENGING

Connectivity - Firewall Friendly WS/SS RDA/Merge

Connectivity - WWAN/Dialup RDA WS/SS/Merge

Connectivity - WLAN RDA WS/SS/Merge

Conflict Resolution Merge SS WS/RDA

Code to Implement RDA Merge/SS WS

Setup/Deployment Effort RDA WS Merge

On-Going Administration RDA WS/SS Merge

Enterprise Management Merge WS/SS RDA

Large Data Volumes Merge/RDA WS/SS

Server DBMS Independence WS/SS

Auto SQL CE DBMS Creation Merge/SS

Ability to Secure Data Sync Merge RDA/WS/SS

Overall Complexity RDA/SS WS Merge

Choosing a Data Sync Strategy

V1 available now, new version coming Sweet-spotAtomic control over performance tuningGreat way to create starter SDF filesConcerns

Sync Services for ADO.NET (Devices)Schedule some time to evaluate this!

Take only what you needAvoid implicit column type conversionsSave a column, eliminate identity range pain

Use uniqueidentifier PKsSet IsRowGuid to True

Download-only vs. Bi-Directional articlesDefrag indexes on the distributor

DBCC INDEXDEFRAG (Driver, MSMerge_Contents, 1-4)Defrag all indexes on MSmerge_*, MSrepl_*, Mssnapshot_*Daily is not too often…

Merge Replication TuningImportant considerations

Row-Level Tracking vs. Column-Level TrackingMaxBuffer Size on Subscriber DatabaseInvestigate lowering retention period from default of 14 days Set Merge Agent profile to match networkInvestigate mixing Merge replication with other techniques (RDA, Sync Services)Replication Monitor does not tell the whole story – instrument your subscriber code

Merge Replication Tuning (cont'd)

Hardware mattersSeparate distributor from publisherData and log files on separate, fast spindlesIncrease SQL Server minimum memory Tune IIS – the ISAPI DLL is the bottleneckMore information:

Merge Replication TuningHundreds of subscribers

www.microsoft.com/teched

Sessions On-Demand & Community

http://microsoft.com/technet

Resources for IT Professionals

http://microsoft.com/msdn

Resources for Developers

www.microsoft.com/learning

Microsoft Certification & Training Resources

Resources

Windows Mobile® ResourcesTechNet TechCenter – System Center Mobile Device Manager 2008 http://technet.microsoft.com/scmdm

TechNet TechCenter – Windows Mobile http://technet.microsoft.com/windowsmobile

MSDN Center – Windows Mobilehttp://msdn.microsoft.com/windowsmobile

Webcasts and Podcasts for IT – Windows Mobilehttp://www.microsoft.com/events/series/msecmobility.aspx

General Information – Windows Mobilehttp://www.windowsmobile.com

General Information – System Center Mobile Device Manager 2008http://www.windowsmobile.com/mobiledevicemanager

Windows Marketplace Developer Portalhttp://developer.windowsmobile.com

Windows Mobile® is giving away Blackjack IIs !

Stop by the Windows Mobile Technical Learning Center to learn how to enter

Complete an evaluation on CommNet and enter to win!

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,

IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.