Welcome To The 2016 Query Store!

Welcome To The 2016 Query Store!

Janis Griffin

Senior DBA / Performance Evangelist

2

• Senior DBA / Performance Evangelist for Solarwinds• [email protected]• Twitter® - @DoBoutAnything • Current – 25+ Years in Oracle®, SQL Server®, ASE, and now MySQL®• DBA and Developer

• Specialize in Performance Tuning• Review Database Performance for Customers and Prospects• Common Question – How do I tune it?

Who am I

© 2017 SolarWinds Worldwide, LLC. All rights reserved.

mailto:[email protected]

3

• What is Query Store?• In the Beginning: Tracing, DMVs & Extended Events • Overview of Query Store

• Why use Query Store?• Benefits & Gotchas

• How Query Store Works• Enabling, configuring & managing it• Best practice considerations – configuration & storage• Performance Impact

• What’s New In 2017?• Reporting & Several Use Cases

• Demos - using both scripts & SSMS

Agenda


4

• User complains query is slow• DBA says nothing has changed

• No new releases• No additional load

• What changed? • Changes in data causing out-of-date statistics• Changes in execution parameters

• Parameter sniffing and/or data skew

• System changes

• Troubleshooting plan changes difficult in previous versions• Had to trace • Had to know xQuery to decipher the plan cache XML

In the Beginning…


5

• <2005 had tracing • High overhead • Difficult to use (had to start & stop)

• 2005+ DMVs: dm_exec_cached_plans & query_stats• Real time only or since last restart or clear• Can’t see history of plan changes or regression overtime

• 2012+ Extended Events• Easier to use than trace • System Health session on by default• No history of plan changes / regression

In the Beginning…


6

• New feature in 2016, Improved in 2017• Provides insight on Query Plans

• Useful in troubleshooting performance degradation • Quickly find differences when query plan changes • Can see history of plans and runtime statistics• Data is persisted in database

• Not just stored in memory

• Shows workload and database usage patterns• Makes upgrades easier

• Between versions of SQL Server• Put in compatibility mode 100 (lowest) or keep at prior level• Turn on query store• Collect representative workload• Force plans where needed

What is Query Store?


7

• Simplifies performance tuning• Can see plan changes & quickly compare their differences• Can aggregate runtime statistics over different time intervals• Can enable at database level

• You get to control when, where and how you use it

• Tracks both runtime and compile time metrics

• Has graphical interface in SSMS• Supported on all SQL Server editions

Benefits Of Using Query Store


2016 2017

8

• Allows forcing or pinning of a better plan for a query• Can be a quick temporary fix• However, can be easily forgotten

• When data and/or schema changes

• Easy T-SQL for scripting• Can quickly find queries:

• Which frequently fail• Which use literals • Which are resource intensive and/or most regressed over time

• Many SPs, QSD Wait types, & Extended Events• For management purposes

Benefits – cont.


9

How Query Store Works

https://msdn.microsoft.com/en-us/library/mt631173.aspx© 2017 SolarWinds Worldwide, LLC. All rights reserved.

https://msdn.microsoft.com/en-us/library/mt631173.aspx

10

How Query Store Works – cont.

https://msdn.microsoft.com/en-us/library/mt631173.aspx© 2017 SolarWinds Worldwide, LLC. All rights reserved.

https://msdn.microsoft.com/en-us/library/mt631173.aspx

11

• Query Store is local for each database• Is disabled by default• To enable in SSMS

Enabling the Query Store


12

• SSMS

• T-SQL

Changing Query Store Settings

ALTER DATABASE AdventureWorks

SET QUERY_STORE = ON

( OPERATION_MODE = READ_WRITE,

CLEANUP_POLICY = ( STALE_QUERY_THRESHOLD_DAYS = 30 ),

DATA_FLUSH_INTERVAL_SECONDS = 900,

MAX_STORAGE_SIZE_MB = 100,

INTERVAL_LENGTH_MINUTES = 10);


13

• https://msdn.microsoft.com/en-us/library/dn818146.aspx

View Query Store Settings

select * from sys.database_query_store_options;

Must watch size!

If full or state

changes, errors

aren’t written to log.

Monitor actual_state.


https://msdn.microsoft.com/en-us/library/dn818146.aspx

14

• Operation Mode (Requested) / operation_mode• Default value: Off• Valid values are Off, Read Write, Read Only

• T-SQL: off, read_write, read_only

• Actual operation mode can change to “read only” with full• Consider monitoring – change not written to logs

• Data Flush Interval (Minutes)• When data is written from memory to disk • Default is 15 minutes

• T-SQL: DATA_FLUSH_INTERVAL_SECONDS • 900 secs = 15 min

Query Store Settings


15

• Statistics Collection Interval• Defaults to 1 hour

• Consider changing to 10 minutes to get granular aggregation• Higher frequency will require more storage• Lowest level – 1 minute

• T-SQL: INTERVAL_LENGTH_MINUTES

• Max Size (MB)• Maximum size allocated to the Query Store

• Defaults to 100 MB• Stored in primary file of database• When full, changes to read-only

• T-SQL: MAX_STORGE_SIZE_MB

Query Store Settings – cont.

ALTER DATABASE AdventureWorks

SET QUERY_STORE CLEAR;


16

• Query Store Capture Mode • ‘All’ collects all plans (default)• ‘Auto’ captures queries based on resource consumption• ‘None’ stops the capture process • T-SQL: QUERY_CAPTURE_MODE

• Size Based Cleanup Mode• Activates data cleanup when it reaches maximum size• 2016 - defaults to ‘Off’, 2017 – defaults to ‘Auto’• Consider changing to ‘Auto’

• Flushes older queries and trivial plans as new data comes in• Will not change to ‘read only’

• T-SQL: SIZE_BASED_CLEANUP_MODE

Query Store Settings – cont.


17

• Stale Query Threshold (Days)• Duration to retain query store statistics• Default 30 days• T-SQL: CLEANUP_POLICY (STALE_QUERY_THRESHOLD_DAYS)

• MAX_PLANS_PER_QUERY • Default is 200 plans

• Have you ever seen 200 plans to 1 query?• Consider changing to something less

• Must change via T-SQL not in SSMS

Query Store Settings– cont.


18

• Consider enabling in the MODEL database• In SSMS - no Query Store property in MODEL db

• Need to turn on via T-SQL• ALTER DATABASE MODEL SET QUERY_STORE=ON;

• Consider proper sizing of Query Store• Default is 100mb• May be too small for busy database• By default, it converts to read-only when full

select max_storage_size_mb - current_storage_size_mb qs_free_space_mbfrom sys.database_query_store_options;

Query Store Considerations


19

• Disabled by default• Need to enable at each individual database level

• Can modify Model so it’s on by default• Consider sizing / performance implications

• Can’t enable for master or tempdb• Can’t enable for Read-Only databases

• E.g. AG replication for read-only

• No global way to control configuration• Without manual scripting or setting up Model DB• DBAs can easily change in SSMS so hard to track

• Changes to QS settings are logged in ERRORLOG

Gotchas Of Using Query Store


20

• Query Store (QS) requires care & feeding• Default sizes may be too small for a busy database• If QS storage is full, switches to READ-Only mode

• May miss important performance info

• Query Store data is stored in primary file • May conflict or compete for resource with user data

• Can add 3-5% load to the database load• May take longer database restores

• Plans may contain literal values (HIPAA)• No additional context – Who ran it, etc…

Gotchas – cont.


21

• Don’t blindly turn it on / plan, plan, plan• Consider what you will use it for

• Which databases are enabled / what’s the retention period

• How will you manage it?• Must monitor QS status, performance & storage• Create standard setup & configuration scripts• Consider backups / restores

• Restore into development / test or NOT• Replication – sensitive data, forced plans, etc…

Query Store Management


22

• Microsoft states 3% to 5% impact on average• Depends on

• # of databases monitored• Transaction / Compile rates• Query Store Configurations

• Can monitor Query Store via • Perfmon Counters• QDS Wait Types• Extended Events

Query Store Performance Impact


23

Query Store Performance Impact

QS Events

2016 – 58

2017 – 90+

Not very well

documented

- 17 in 2016

- 20 in 2017


24

• sp_query_store_flush_db;• Flushes in-memory portion of data to disk

• sp_query_store_reset_exec_stats plan_id;• Clears runtime stats for a specific query plan

• sp_query_store_force_plan query_id, plan_id;• Enables forcing a particular plan, if forcing fails a Xevent is fired to optimize normally

• sp_query_store_unforce_plan query_id, plan_id;• Removes forcing a particular plan for a particular query

• sp_query_store_remove_plan plan_id;• Removes a single plan from the Query Store

• sp_query_store_remove_query query_id;• Removes a single query, all associated plans and statistics from Query Store

Management – Stored Procedures



…

…

……

…

25

• query_store_query_text - text entered by user, includes whites space, hints, etc... msdn-dn818159.aspx• query_context_settings - unique combinations of plan affecting settings under which queries run msdn-dn818148.aspx• query_store_plan - execution plan information for queries msdn-dn818155.aspx• query_store_query - query information and overall aggregated runtime statistics msdn-dn818156.aspx• query_store_runtime_stats - runtime execution statistics for queries (avg,min,max,std deviation) msdn-dn818158.aspx• query_store_runtime_stats_interval - start and end times intervals for statistics collected msdn-dn818147.aspx2

Query Store DMVs







26

• SSMS Reports• Force Plans

• View Forced Plans

• Compare Plans• Find queries with high variation• See overall resource consumption

• T-SQL • Useful Queries

• Find Multiple Plans• Find Long Running Queries• Find Queries Doubling in time • Find AD-Hoc Queries

• Using literals

• Plan Guides versus Forced Plans

Query Store Reporting & Use Cases


• New Wait Stats Store• Plus plan store• And runtime stats store• On = 1 (default)• T-SQL: WAIT_STATS_CAPTURE_MODE

• OFF = 0

• New 23 Wait Categories• Summarizes wait types• Common resolution

• Using-the-query-store


What’s New In 2017 Query Store?

Join to query_store_plan and

query_store_runtime_stats_interval

https://docs.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store


2017 Query Store – Cont.

• Find Top 10 Queries Waiting The Most

Over 1 hour waiting on Parallelism

SELECT TOP 10

qt.query_sql_text,

q.query_id,

p.plan_id,

ws.wait_category_desc,

sum(total_query_wait_time_ms) AS sum_total_wait_ms

FROM sys.query_store_wait_stats ws

JOIN sys.query_store_plan p ON ws.plan_id = p.plan_id

JOIN sys.query_store_query q ON p.query_id = q.query_id

JOIN sys.query_store_query_text qt ON q.query_text_id = qt.query_text_id

WHERE ws.wait_category_desc != 'Idle'

GROUP BY qt.query_sql_text, q.query_id, p.plan_id, ws.wait_category_desc

ORDER BY sum_total_wait_ms DESC


2017 Query Store – Cont.

• Query Store and the New Adaptive Query Processing• Captures and can force batch mode adaptive join plan• Can force plans with interleaved execution (MSTVFs)

• Command-line Tools• sqlcmd & bcp

• SSMS• Connect remotely from Windows• Includes PowerShell

• Visual Studio Code• How-to-develop-use-vscode

• SQL Server Data Tools (SSDT)• For Visual Studio • Build project on Windows

• Deploy to Linux

• SQL Operations Studio • Installs on Linux/Windows/MacOS• GUI interface – sqlops

https://docs.microsoft.com/en-us/sql/sql-operations-studio/download

Client Connectivity & Tools for 2017


https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-develop-use-vscode

https://docs.microsoft.com/en-us/sql/sql-operations-studio/download

31

• Review Properties• Database defaults• Turn on QS in Model• Database with QS on• Show full QS

• Look at T-SQL Code• Multiplan Queries• Long running – doubled in duration (microseconds)• Parameter Canidates

• Review SSMS Reports• Force a Plan

• Show Forced Plan Failures

• Extended Events• Q & A

Demo Time


32

SSMS - Regressed Queries Report


33

Overall Resource Consumption

34

Top Resource Consuming Queries

Canceled

Failed


35

Tracked Queries


36

Compare Query Plans


37

Forcing Plans

SELECT p.plan_id, p.query_id, q.object_id as containing_object_id, force_failure_count, last_force_failure_reason_desc

FROM sys.query_store_plan p

JOIN sys.query_store_query q on p.query_id = q.query_id

WHERE is_forced_plan = 1;

To remove force plan via TSQL:

sp_query_store_unforce_plan 615,665;


38

Failed Forced Plans

Reasons why plan forcing failed

0: no failure

8637: ONLINE_INDEX_BUILD

8683: INVALID_STARJOIN

8684: TIME_OUT

8689: NO_DB

8690: HINT_CONFLICT

8691: SETOPT_CONFLICT

8694: DQ_NO_FORCING_SUPPORTED

8698: NO_PLAN

8712: NO_INDEX

8713: VIEW_COMPILE_FAILED

<other value>: GENERAL_FAILURE


39

• https://msdn.microsoft.com/en-us/library/dn817826.aspx

;WITH Query_MultPlans

AS

(

SELECT COUNT(*) AS cnt, q.query_id

FROM sys.query_store_query_text AS qt

JOIN sys.query_store_query AS q

ON qt.query_text_id = q.query_text_id

JOIN sys.query_store_plan AS p

ON p.query_id = q.query_id

GROUP BY q.query_id

HAVING COUNT(distinct plan_id) > 1

)

Useful Query Store Queries

SELECT q.query_id, object_name(object_id) AS ContainingObject,

query_sql_text, plan_id, convert(xml,p.query_plan) AS plan_xml,

p.last_compile_start_time, p.last_execution_time

FROM Query_MultPlans AS qm


ON qm.query_id = q.query_id

JOIN sys.query_store_plan AS p

ON q.query_id = p.query_id

JOIN sys.query_store_query_text qt


ORDER BY query_id, plan_id;

GO



40

Multi-plan Queries


41

;WITH AggregatedDurationLastHour

AS

(

SELECT q.query_id, SUM(count_executions * avg_duration) AS total_duration,

COUNT (distinct p.plan_id) AS number_of_plans

FROM sys.query_store_query_text AS qt JOIN sys.query_store_query AS q


JOIN sys.query_store_plan AS p ON q.query_id = p.query_id

JOIN sys.query_store_runtime_stats AS rs ON rs.plan_id = p.plan_id

JOIN sys.query_store_runtime_stats_interval AS rsi

ON rsi.runtime_stats_interval_id = rs.runtime_stats_interval_id

WHERE rsi.start_time >= DATEADD(hour, -1, GETUTCDATE())

AND rs.execution_type_desc = 'Regular'

GROUP BY q.query_id

) ,OrderedDuration

AS

(

SELECT query_id, total_duration, number_of_plans,

ROW_NUMBER () OVER (ORDER BY total_duration DESC, query_id) AS RN

FROM AggregatedDurationLastHour

)

Long Running – Last Hour

SELECT qt.query_sql_text, object_name(q.object_id) AS

containing_object, q.query_id,

p.plan_id,rsi.start_time as interval_start, rs.avg_duration,

CONVERT(xml, p.query_plan) AS query_plan_xml

FROM OrderedDuration od JOIN sys.query_store_query

AS q ON q.query_id = od.query_id

JOIN sys.query_store_query_text AS qt ON

q.query_text_id = qt.query_text_id

JOIN sys.query_store_plan AS p ON q.query_id =

p.query_id

JOIN sys.query_store_runtime_stats AS rs ON rs.plan_id

= p.plan_id

JOIN sys.query_store_runtime_stats_interval AS rsi ON

rsi.runtime_stats_interval_id = rs.runtime_stats_interval_id

WHERE rsi.start_time >= DATEADD(hour, -1,

GETUTCDATE())

AND number_of_plans > 1

ORDER BY total_duration DESC, query_id,

rsi.runtime_stats_interval_id, p.plan_id


42

Long Running – Last Hour


© 2017 SolarWinds Worldwide, LLC. All rights reserved. 43

SELECT

qt.query_sql_text,

q.query_id,

qt.query_text_id,

rs1.runtime_stats_id AS runtime_stats_id_1,

rsi1.start_time AS interval_1,

p1.plan_id AS plan_1,

rs1.avg_duration AS avg_duration_1,

rs2.avg_duration AS avg_duration_2,

p2.plan_id AS plan_2,

rsi2.start_time AS interval_2,

rs2.runtime_stats_id AS runtime_stats_id_2

FROM sys.query_store_query_text AS qt



JOIN sys.query_store_plan AS p1

ON q.query_id = p1.query_id

JOIN sys.query_store_runtime_stats AS rs1

ON p1.plan_id = rs1.plan_id

Execution Time - Doubled in Last 48hrs

JOIN sys.query_store_runtime_stats_interval AS rsi1

ON rsi1.runtime_stats_interval_id = rs1.runtime_stats_interval_id

JOIN sys.query_store_plan AS p2

ON q.query_id = p2.query_id

JOIN sys.query_store_runtime_stats AS rs2

ON p2.plan_id = rs2.plan_id

JOIN sys.query_store_runtime_stats_interval AS rsi2

ON rsi2.runtime_stats_interval_id = rs2.runtime_stats_interval_id

WHERE rsi1.start_time > DATEADD(hour, -48, GETUTCDATE())

AND rsi2.start_time > rsi1.start_time

AND p1.plan_id <> p2.plan_id

AND rs2.avg_duration > 2*rs1.avg_duration

ORDER BY q.query_id, rsi1.start_time, rsi2.start_time;

44

Execution Time - Doubled in Last 48hrs

Plan_1 = 370ms Plan_2 = 3.7s


45

• Query Store helps you manage query performance• See plan changes & performance regression• Find bad parameter sniffing & stale statistics• See resource consumption at query & database level

• Info is only kept in database where query runs• If you run them out of master or tempdb, they won’t be kept

• Need to plan & maintain the Query Store• Size appropriately for workload• Consider how long to keep the data• Consider backups & restores

Summary


The SolarWinds, SolarWinds & Design, Orion, and THWACK trademarks are the exclusive

property of SolarWinds Worldwide, LLC or its affiliates, are registered with the U.S.

Patent and Trademark Office, and may be registered or pending registration in other

countries. All other SolarWinds trademarks, service marks, and logos may be common

law marks or are registered or pending registration. All other trademarks mentioned

herein are used for identification purposes only and are trademarks of (and may be

registered trademarks) of their respective companies.

Thank You!

Welcome To The 2016 Query Store!

Technology

Transcript of Welcome To The 2016 Query Store!