Knowing the Internals - Who Needs SQL Server Anyway - Mark Rasmussen

8/17/2019 Knowing the Internals - Who Needs SQL Server Anyway - Mark Rasmussen

1/98

Knowing the Internals,Who Needs SQL ServerAnyway?

Mark S. Rasmussen – improve.dk


2/98

2

Whoami

Tech Lead @ iPaper

Developer/DBA/Ssadmin/Pro!ec"

mana#er/$

%omp.Sci @ Aarhus &niversi"

Blo##in# a" improve.dk@improvedk

Au"hor o' "he (rcaMD) pro!ec"

https://github.com/improvedk/OrcaMDFhttps://github.com/improvedk/OrcaMDF


3/98

*

Level +,- – mean" "o inspire no"

"each

Based on 2--0 R2

1 have no idea...

Mos" o' ha" 1 sa is incorrec"

Disclaimer


4/98

3

Presen"a"ion a" Miracle (pen 4orld

)ormall s"ar"ed (rcaMD)

Backgrond


5/98

,

!ld School Qerying

private static void oldschool(){

using (var conn = new SqlConnection("Data Source= .;Initial Catalog= QD ;! "){

conn. pen();

var c# d = new SqlCo# # and("S$%$C& ' *ersons"+ conn);var reader= c# d.$,ecuteeader();

w hile(reader.ead())Console.- rite%ine(reader"ID"/0 "1 " 0 reader"2a# e"/0 " (" 0 reader"3

44


6/98

+

!rca"D# Qerying

using (var # d5= new d5ile(# d5*ath)){

var scanner= new DataScanner(# d5);var row s = scanner.Scan&a6le("*ersons");

$ntit7*rinter.*rint(row s);4

using (var # d5= new d5ile(# d5*ath))

{var scanner= new DataScanner(# d5);var row s = scanner.Scan&a6le("*ersons")

.- here(, = 8 ,.ield9 short8 (:3ge:) 9


7/98

$ages The 'ounda"ion o' S5L Server s"ora#e


8/980

0672 8"es

%verything s"ored

as pa#es

What Is a $age?


9/987

DB%% 19D

DB%% PA:;

DB%% TRA%;(9


10/986-

A8solu"el no documen"a"ion

A8solu"el necessar 'or parsin#

$age )eader


11/9866

D;M(

(rcaMD) Pa#eeader

*everse %ngineering the)eader

http://var/www/apps/conversion/tmp/scratch_5/Demos/Reverse%20engineering%20the%20header%20format.sqlhttp://var/www/apps/conversion/tmp/scratch_5/Demos/PageHeader.cshttp://var/www/apps/conversion/tmp/scratch_5/Demos/PageHeader.cshttp://var/www/apps/conversion/tmp/scratch_5/Demos/Reverse%20engineering%20the%20header%20format.sql


12/9862

Poin"s "o 8e#innin#

o' records in 8od

DeCnes lo#ical

order o' records

Slot Array


13/986*

Da"a records S"ores "a8le da"a

1nde records S"ores nonclus"ered inde da"a as ell as

nonElea' level clus"ered inde da"a

S"ored in "he FixedVar 'orma"

*ecords


14/9863

#i+edar *ecord #ormat


15/98


16/98

6+

Stats Bits B


17/98

6?

Bi"map "rackin# he"her columns are

9&LL

%;1L


18/98

60

aria.le Length !/setArray


19/98

67

%+am-le *ecord

CREATE TABLERecordTest(

A int,B int,C char(5),D varchar(10),E varchar(20)

)

INSERT INT RecordTest !AL"ES (25, #$, %ABCD%, %&ar'%,%Denar'%)


20/98

2-

)ied len#"h da"a alas presen" ;ven i' null

▪ Thou#h no" necessaril "ail columns

Haria8le len#"h da"a onl presen" hen no"

null

Addin# nulla8le columns is a me"ada"a op Denali de'aul" value columns is me"ada"a "oo

When Is Data $resent?


21/98

Data 0y-eso are da"a "pes s"ored i"hin a recordI


22/98

22

)ied len#"h da"a "pes 8i" char in" decimal da"e da"e"ime

Joa" e"c.Haria8le len#"h da"a "pes


23/98

2*

SL(Bs varchar


24/98

23

In2row 3n4varchar3+4Storage

C$3&$ &3%$ >archar&est(

3 varchar())

I2S$& I2& >archar&est >3%?$S@ @


25/98

2,

D;M(

1den"iCed usin# "he si#n 8i"


26/98

2+

Harchar nvarchar var8inar

D;M(

!/2row SL!B Storage

http://var/www/apps/conversion/tmp/scratch_5/Demos/Off-row%20varchar(x)%20storage.sqlhttp://var/www/apps/conversion/tmp/scratch_5/Demos/Off-row%20varchar(x)%20storage.sql


27/98

2?

%olumn da"a moved "o ne pa#e

poin"er le'" 8ehind

!/2row SL!B Storage


28/98

20

!/2row SL!B Storage


29/98

27

!/2row SL!B Storage

= % Inline oot/ Slot < Colu#n G 55set


30/98

*-

S"ored on shared


31/98

*6

Allas s"ored inEro i' N 23 8"es

23 8"e OBL(B 1nline Roo" poin"er

Da"a s"ored in BL(B)RA:M;9T on

Te"Mi pa#e

Times"amp QQ Blo8 1D Per'ormance predic"ion is "ou#h

!/2row SL!B Storage


32/98

*2

varchar


33/98

**

&sed hen da"a C"s in record

9o" an ocial L(B s"ruc"ure

8BL!B Inline Data8

Slot < Colu#n B 55set


34/98

*3

%an re'erence up "o , pa#es – da"a roo"s "rees

e"c.

62 8"e header

Arra o' 62 8"e re'erences

(nl used 8 SL(Bs >


35/98

*,

4rapped in a sin#leEcolumn me"a

record

L!B Strctre *ecords


36/98

*+

Tpe *

4here da"a is ac"uall s"ored

Sie alas +3 8"es


37/98

*?

o much da"a can e s"ore in a

DATA recordI 0-7+ – Pa#e 8od sie

0-0-


38/98

*0

Tpe 2

%urLinks Q num8er o' re'erences

MaLinks Q I

Level Q "ree level

Sie Q compu"ed

IN0%*NAL

lo6 row at1 *age (B1FF) Slot < %ength1 G &7pe1 G (I2&$23%)

lo6 Id1 BGB


39/98


40/98


41/98

36

1n "heor


42/98

32

'onnecting the Dots


43/98


44/98

33

0wo Levels Is All It 0akes

0-3------- 8"es / ?30 :B


45/98

3,

sp"a8leop"ion VMTa8leG V(p"ionG

V(9/())G

;ven more permu"a"ions

W"e" in ro 23E?--- de'aul" 2,+

Large ale 0y-es !t o1*ow


46/98


47/98

3?

Xou "hou#h"


48/98


49/98

37

Tpe Q ,

Min sie Q 03

Par" o' L(B "ree

LA*6%5*!!059&K!N


50/98

,-

'onnecting the Dots


51/98

,6

'onnecting the Dots


52/98

,2

Data archar374

archar3"A74

0e+t

N&LL - - -

:2;< -E+3 -E+3 6-- ::: +,E0--- +,E0--- $ k. 9/A 23 U K 6-- U K

L!B Storage !verview

;"reme impac" on small da"a

The more da"a "he less o' a diY

Per'ormance diYerences h""p//sl8lo#.com/8lo#s/paulhi"e/archive/2-66/-2/2*/AdvancedETS5LETunin#E4hE1n"ernals

EZnoled#eEMa""ers.asp

http://sqlblog.com/blogs/paul_white/archive/2011/02/23/Advanced-TSQL-Tuning-Why-Internals-Knowledge-Matters.aspxhttp://sqlblog.com/blogs/paul_white/archive/2011/02/23/Advanced-TSQL-Tuning-Why-Internals-Knowledge-Matters.aspxhttp://sqlblog.com/blogs/paul_white/archive/2011/02/23/Advanced-TSQL-Tuning-Why-Internals-Knowledge-Matters.aspxhttp://sqlblog.com/blogs/paul_white/archive/2011/02/23/Advanced-TSQL-Tuning-Why-Internals-Knowledge-Matters.aspx


53/98

,*

0y-e Name

- SMALLR((T

6 I

2 19T;R9AL

* DATA

3 I

, LAR:;R((TX&Z(9

+ I

? I0 I

7U I

"ind the 6a-

Bending the Will o1 DB''


54/98

,3

Bending the Will o1 DB''$age@ Again@


55/98

,,

Archeology :

0y-e Name

- SMALLR((T

LA*6%5*!!0

2 19T;R9AL

* DATA

< LA*6%5*!!05S)IL!)

, LAR:;R((TX&Z(9

; S&$%*5LA*6%5*!!0

> N&LL

C INALID


56/98

,+

N 0--- Q


57/98

Indices ( )ea-so pa#es are or#anied


58/98

,0

DeCnes ho da"a is $phsicall$ s"ored

%lus"ered inde

:uaran"ees phsical order o' da"a Ro iden"iCed 8 clus"ered ke

eap

Da"a s"ored hereever S5L Server an"s "o Ro iden"iCed 8 R1D

'lstered Inde+ vs )ea-


59/98

,7

B2tree Scanning


60/98

+-

Relies on 1AM pa#es

Lea' pa#es no" linked

;cep"...

)ea-s

$ ( % t t


61/98

$age ( %+tent

Allocation;"en"s pa#es > o8!ec"s


62/98

+2

All pa#es alloca"ed as par" o' an

e"en"

Mied e"en"s

&ni'orm e"en"s

)irs" 0 Q mied res" uni'orm

%+tents


63/98

+*

6lo8al Alloca"ion "ap

6 Q )ree


64/98

+3

Shared 6lo8al Alloca"ion "ap

6 Q Mied >


65/98

+,

Inde Alloca"ion "ap

6 Q &ni'orml alloca"ed "o 1AM chain /

alloca"ion uni"


66/98

++

IA" $age )eaderStorage


67/98

+?

%+tent Allocation Stats

6A" S6A" AnyIA"

Stats

0 0 0 Mied all pa#esalloca"ed

0 1 0 Mied


68/98

+0

$a#e #ree Space

Bytemap

6 / 0-00 ever 0-00 pa#es

P)S in"erval

(nl "racks 'ullness here necessar

$#S $ages


69/98

ll i i


70/98

?-

Allocation &nits

0h B $


71/98

?6

Pa#e 7 in primar da"a Cle

DB%% PA:; QQ DB%% DB19)(

Lo"s o' in"eres"in# in'o Phsical version

Lo# re8uild coun"

Las" (Z %;%ZDB Las" L(: 8ackup

9ame U 1D

)irs"Ss1ndees

0he Boot $age

System iews ( Base


72/98

System iews ( Base

0a.les The source o' our parsin# me"ada"a

Needed "etadata 1or


73/98

?*

Schema ss."a8les U ss.columns / ss.indees U

ss.indecolumns

1ndees Roo" pa#e

eaps 1AM chain roo"

D;M(

Needed "etadata 1or$arsing

Allocation "etadata

http://var/www/apps/conversion/tmp/scratch_5/Demos/Needed%20metadata%20for%20parsing.sqlhttp://var/www/apps/conversion/tmp/scratch_5/Demos/Needed%20metadata%20for%20parsing.sql


74/98

?3

Allocation "etadata!verview

D" A & l


75/98

?,

[us" vies no phsical s"ora#e

%hicken or "he e##

o a8ou" e "ake a look a" "hose

viesI

D;M(

D"s Are &seless

B 0 .l

http://var/www/apps/conversion/tmp/scratch_5/Demos/DMVs%20are%20useless.sqlhttp://var/www/apps/conversion/tmp/scratch_5/Demos/DMVs%20are%20useless.sql


76/98

?+

The 8asis 'or DMH da"a

%an onl 8e ueried "hrou#h "he DA%

ere 8e dra#ons

%on'usin# column names

&"ilies in"ernal 'unc"ions

Base 0a.les


77/98

# ll th * ..it


78/98

?0

Boo" pa#e poin"s "o ssallocuni"s


79/98

?7

!$%N*!WS%030ABL%*S'$*!$4

SELECT C3S$ c.#a,inrowlen *EN < T*EN p.length

ELSE c.#a,inrowlen END AS #a,LinrowLlength+ p.,t7pe AS s7ste#Lt7peLid+

p.length AS #a,Llength+p.prec AS +RECISIN+p.scale AS scale+

R& s7s.s7srscols c MN s7s.s7ste#LinternalsLpartitionLcolu#ns

?&$ 3**%O *$2-S$&(TABLE SC**+ c.ti) p

!$%N*!WS%030ABL%


80/98

0-

!$%N*!WS%030ABL%*S'$*!$4

!$%N*!WS%030ABL%


81/98

06

!$%N*!WS%030ABL%*S'$*!$4

CREATE TABLE &I&est( 3 -inar.(F


82/98

02

! ! S 3*S'$*!$4

SELECT t.nae+ r.ti+ p.scale+ p.3recision+ p.#a,Llength+ p.s7ste#Lt7peLid+ p.#a,LinrowLlengthR& s7s.s7ste#LinternalsLpartitionLcolu#ns pINNER RI2 s7s.s7srscols r N r.rscolid = p.partitionLcolu#nLid 32D

r.rsid = p.partitionLidINNER RI2 s7s.t7pes t N t.s7ste#Lt7peLid = p.s7ste#Lt7peLid 32D t.userLt7peLid = p.s7ste#Lt7peLid*ERE

partitionLid = G


83/98

0*

3*S'$*!$4

!$%N*!WS%030ABL%


84/98

03

BGH =


85/98

0,

BGG =


86/98

0+


87/98

0?

g y-$arser

&est/ 3-/ic void Deci#al() { var parser = ne4 S7srscol&I*arser(


88/98

Data *ecovery4hen everything else 'ails

$lease DonEt *ely on


89/98

07

Xou should always have 8ackups

availa8le

Make sure "o test our 8ackups

Run reglar consistency checks

This is a last resort measure

y0his

What Kind o1 $ro.lems to


90/98

7-

Torn pa#es

%orrup" pa#es

Bad me"ada"a

Acciden"al dele"es > "runca"ions

o does (rcaMD) diYerI

%+-ect

0orn $ages


91/98

76

(ne pa#e Q 6+ disk sec"ors

)irs" and las" sec"or mos" impor"an"

9o header 1den"i' o8!ec" 'rom 1AM linked lis"

9o slo" arra Slo" coun" in header

1den"i' record 'orma"s in 8od

0orn $ages

'orr-t $ages


92/98

72

%hecksum doesnG" ma"ch con"en"

%ould 8e minor issue pro8a8l ma!or

Trea" like "orn pa#e

'orr-t $ages

Bad "etadata


93/98

7*

S5L Server 8u#s

%orrup"/"orn pa#es

Scan pa#es and iden"i' o8!ec" in header

Scan pa#es and look 'or 1AM chain

Deduce schema App

Docs

Record 'orma"

Bad "etadata

Accidental Deletes (


94/98

73

Acciden"al dele"e Records ma 8e #hos"ed

Records removed 'rom slo" arra

ST(P

Acciden"al "runca"ion

Pa#es dealloca"ed phsicall in"ac" Scan pa#es linked lis"

0rncations

Watch !t #or Instant


95/98

7,

:ar8a#e ma 8e mis"aken 'or da"a

4as pa#e alloca"edI

Look 'or clues in salva#a8le alloca"ions"ruc"ures

InitialiFation

!rca"D# #tre $lans


96/98

7+

%orrup"Md' class

Heri'in# "orn8i"s / checksum

&"ili" me"hods Scan 'or pa#es 8elon#in# "o o8!ec"

Scan 'or 1AM pa#es

Bes"EeYor" parsin# o' pa#es

!rca"D# #tre $lans


97/98


98/98

0hank yo

Blo# improve.dkTi""er @improvedk

Knowing the Internals - Who Needs SQL Server Anyway - Mark Rasmussen

Documents

Transcript of Knowing the Internals - Who Needs SQL Server Anyway - Mark Rasmussen