SQL Server and SharePointBest Frienemies

Lisa GardnerPremier Field EngineerLisaG@microsoft.comhttp://blogs.msdn.com/sqlgardner@sqlgardner

2

Lisa Gardner aka SQLGardnerCentral Ohio nativeWorking with SQL since 6.5http://blogs.msdn.com/sqlgardner@SQLGardner

Premier Field EngineeringReactive and Proactive support for Premier customersArchitecture/Project GuidanceTeam MentoringDeliver workshopsTroubleshooting

Who am I ? What is PFE?

Table of ContentsSharePoint OverviewSharePoint DatabasesConfiguration, Setup, and MaintenanceWhat to Look Out For

SharePoint OverviewUnderstanding the Application.

5

WFEWeb ApplicationService ApplicationSite CollectionULS LogsTimer Jobs

SharePoint Glossary

6

SharePoint Web Architecture

Farm

Web Applicati

on

Site Collectio

n

Site

List

Item

7

Web Application

• Different IIS Site

• Created on each WFE

• Isolates Content

• Provides authentication mechanism

Site Collection• Container of

Sites• Quotas• Decentralized

Content Administration

• Also serves as a site

Site• Permission

Inheritance• Can Share

layout and data with other sites

• Can provide unique feature set from other sites

Key Attributes

8

Provides granular pieces of functionality Some can be tied to a specific serverOffers scalability, load balancing, fault tolerance for most servicesMany to many relationship with web applications and service applications

Each web application can have a unique set of service applications

Service Applications

9

SharePoint equivalent to SQL AgentOWSTimer - Windows service for SharePoint 2010 at a predefined scheduleUses same logging infrastructure as web tier

Includes Correlation IDsJobs can be nested

Timer Jobs

10

SharePoint Internal Data

Logging Database

ULS Logs

Correlation IDs

11

Stores all SharePoint usage and health dataULS trace log dataEvent log dataBlocking SQL QueriesCrawl and Query StatisticsFeature UsagePage Requests+More

Logging Database

12

Logging Database

Table partitioning for ease of use and custom views

Tables partitioned for performance

Views for usability

Custom reports through Excel or Excel web parts

Some out-of-the-box reports use this data

13

ULS Logs

Over 260 categories of events

Must be consistent location on all farm servers

Also referred to as trace logs

Nine columns including: Timestamp, Process, Category, EventID, Level, Message, and Correlation

14

Correlation IDs

Web Front-End Server

Application Server

~~~~~~~~~~~~~

~~~~~~~~~~~~~

ULS

7d25d051-ca73-43…

ULS7d25d051-ca73-43 …

7d25d051-ca73-43 … 7d25d051-ca73-43 …

Generated for every requestLogged from the start of a request through to the endUseful for troubleshooting and tracingOn error pages, ULS logs, Windows Logs, SQL Traces

SharePoint DatabasesSo Many Databases, So Little Time

16

Farm Configuration StoreObjects Table – Serialized ObjectsBinaries Table – Farm Solution StoreSiteMap Table – Links a site into the configuration

Content Database for Central Admin is a Content DB with very specific templates - considered to be an extension of the configuration databaseBackup and Recovery

It is Supported to back up this databaseIt is Not Supported to restore unless the farm is fully stopped when the backup is taken

Configuration and Admin Content Databases

17

General Recommendations Default recovery model is Full but in most cases this database should be run in simple recovery modeInitial Data File Size: 2GB is appropriate for most situationsConfig databases are typically smaller and do not get much load

MirroringSupported to mirror within the farm (partner on same network as primary)Not Supported to mirror asynchronously or to log ship over WAN

Configuration and Admin Content Databases

18

Stores all site data in a site collectionSite MetadataWeb Part PagesFiles uploaded to document librariesList ItemsSecuritySolutions

It is supported to Mirror in Farm for High AvailabilityIt is supported to Mirror Asynchronously or Log Ship over WAN for disaster recoveryGeneral Recommendations

Run in Full recovery mode only if the site data requires point in time restores

Content Database

19

Content Database SchemaWhy SharePoint seems so crazy.

Userdata table

Namespace Table

Container TablesId Quota Other Metadata

Sites

SiteId Id Url Title ScopeId MetadataWebs

WebId Id Title ItemCount ScopeId Fields Metadata

AllLists

IdSit

eIdDirN

ame

LeafNam

e

WebId

ListId

DoclibR

owId

Other Metadata

ntext

sql_va

riant

int float

bitOther

metadata

nvarch

ar

dateti

me 1…64 1...32 1..8 1..16 1..12 1..8 1…16 ~35

Url

20

Can contain 1-2000 site collectionsScale out at the db level and the instance level.

Sizing Guidance <200GB – Maintenance tasks stay manageable– Makes db movement and DR easier– Plan for 2 IOPs per GB dataCan have 200GB-4TB if .25 IOPs per GB Size and load depends on the sites they containSeparate very active sites into different site collections/content dbsCan have 32,767 dbs per instance, but recommend 200 per instance as manageability can be an issue300 DBs per Web Application

Content Database Layout

21

Search• Admin• Crawl• Property

Profile• Profile• Syncronizatio

n• Social

Tagging

Web Analytics• Reporting• Staging

Service Application DBs

Logging BDC State Secure Store

Reporting Services

Power Pivot

Project Server

Performance Point

22

The different service application dbs have a wide variety of performance/sizing considerations.Write-Intensive dbs

Usage and Health Data Collection database (Logging)Web Analytics Reporting database (during load)Search service application Crawl database (during crawls)Search service application Property databaseUser Profile service application Synchronization database

Service Application DatabasesPerformance Considerations

23

Read Intensive DB’sWeb Analytics Reporting databaseSearch service application Crawl databaseUser Profile service application Profile databaseUser Profile service application Synchronization databaseUser Profile service application Social Tagging databaseReporting database (Project Server)

Service Application DatabasesPerformance Considerations Cont’d

24

Database Scale Out Guidance

Search

Content

Logging

Web Analyti

cs

Other

ContentContent

Admin/ Content

ContentContentContent

Configuration, Setup, and Maintenance

26

Allow the SharePoint installer to create databasesModify file sizes and growth settingsRename dbs to remove GUIDs

SharePoint setup and admin accounts required roles:

DB CreatorSecurity Admin

Can be removed for the setup account but will need to be added again for any further installs – not recommended

Patching/Service packsAdding a new Service Application

Add Service Application account logins Requires db_owner role in DB

Planning for SharePoint Setup

27

Follow general Best Practices for SQL ConfigurationUse Latin1_General_CI_AS_KS_WS collationConfigure for heavy TempDB usage

Multiple data filesData and log files separated/isolatedPre-size data files

Set max degree of parallelism to 1SharePoint overrides with MAXDOP

Set max server memory and use Lock Pages In MemConsider setting fill factor (%) to 80

Instance Configuration

28

Do not use Auto ShrinkSet Auto Create Statistics OFF Set Page Verify to ChecksumSet Auto Grow sizes to MB not Percent

Pre-size for growthMonitor utilization and grow manually!

Database Configuration

29

Index Maintenance is extremely important in SharePoint

DMV Sys.dm_db_index_physical_stats can be used to report index fragmentation

SharePoint 2007 by default would rebuild every index via a Timer Job

SharePoint 2010 does a much better job at keeping index fragmentation in check

It only rebuilds indexes that are fragmentedUpdates statistics

Index Maintenance

30

Index defragmentation and statistics maintenance address the following databases:

Configuration databasesContent databasesUser Profile: Profile databasesUser Profile: Social databasesWeb Analytics Reporting databasesWeb Analytics Staging databasesWord Automation Services databasesSearch Property/Crawl databases

These databases contain proc_DefragmentIndicesRun daily

Health Analyzer Rules

31

SearchProperty database– Proc_MSS_DefragSearchIndexes– Run weeklyCrawl database– Proc_MSS_DefragGathererIndexes– Manual– Always report as fragmented– Execute this rule after the first full crawl

Health Analyzer Rules Cont’d

32

Health Analyzer rules rebuild indexes and update statisticsAutoUpdate – off in SP 2010 by defaultUpdate manually when:

Query execution times are slowAfter maintenance operations such as table truncation or a large batch insert/update/delete

Statistics

33

GUIDs are used as clustered primary keysRandom values = unpredictable insert pattern16 bytes each

Heavy insert/update activity

These properties lead to rapid index fragmentation due to many page splits

Fillfactor helps delay the inevitable but increases space usageSharePoint rebuilds indexes with fillfactor of 80

Why is Index/Stats Maintenance So Important?

What to Look Out ForCommon issues

35

Use DBA created content databases!SharePoint hard codes small file size and growth settingsAutomation Options:

Powershell is a great option to allow SP Administrators to create dbs!Have a number of empty DBs already created

Must Do’sUse Latin1_General_CI_AS_KS_WS collationSet appropriate recovery model for your recovery needsAdd SP farm setup account and service account with db_owner role

New Content Databases

36

When a full crawl is running – it is a very intensive operation that can have an impact on other dbs hosted on that instance – if asked about overall performance slowdown, ask if a crawl is running

It is common to see deadlocking in the Crawl database during this time.

If size rapidly grows: ask about the depth of crawling links in documents

Full Crawl Impact

37

Health Analyzer Rule DefinitionDatabases used by SharePoint have fragmented indicesDatabases used by SharePoint have outdated index statistics

Health Analysis Job in Logging DBDetails in ULS logs

Ensure Index Maintenance is Running

38

Common scenario: “The SQL Server is slow”Ask for ULS Log infoBlocking/Deadlocks can be common in content DBs

Try a manual update stats

Inquire about large lists, dbs over threshold, and other capacity limitations being exceeded

Ask about list throttling and “happy hour”

Read Committed Snapshot Isolation is not supported

Excessive Blocking

39

ASYNC_NETWORK_IO_WAITSDisk IOTempDB BottleneckVery Large QueriesLogging is the ONLY DB to be queried directly

Others

40

Shredded storage to minimize storage needs with versioningSparse Columns to support wide listsWeb Analytics redesigned – more robust/scalableProfile Sync – tests of a 3 week import for 300k users now only takes 7 hoursStretch farms no longer supported – all databases must now reside in the same data center

SP2013 Changes

41

Know the Limits!http://technet.microsoft.com/en-us/library/cc262787.aspx

More info on SharePoint DBshttp://technet.microsoft.com/en-us/library/cc678868.aspxhttp://www.microsoft.com/en-us/download/details.aspx?id=3408

Helpful Links

Questions?