04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols)...
Transcript of 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols)...
![Page 1: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/1.jpg)
Le
ctu
re #
04
Multi-Version Concurrency Control (Protocols)@Andy_Pavlo // 15-721 // Spring 2020
ADVANCEDDATABASE SYSTEMS
![Page 2: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/2.jpg)
15-721 (Spring 2020)
L AST CL ASS
We discussed the four major design decisions for building a MVCC DBMS.→ Concurrency Control Protocol→ Version Storage→ Garbage Collection→ Index Management
2
![Page 3: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/3.jpg)
15-721 (Spring 2020)
TODAY'S AGENDA
Microsoft Hekaton (SQL Server)
TUM HyPer
SAP HANA
CMU Cicada
3
![Page 4: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/4.jpg)
15-721 (Spring 2020)
MICROSOFT HEKATON
Incubator project started in 2008 to create new OLTP engine for MSFT SQL Server (MSSQL).→ Led by DB ballers Paul Larson and Mike Zwilling
Had to integrate with MSSQL ecosystem.
Had to support all possible OLTP workloads with predictable performance.→ Single-threaded partitioning (e.g., H-Store/VoltDB)
works well for some applications but terrible for others.
4
![Page 5: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/5.jpg)
15-721 (Spring 2020)
HEKATON MVCC
Each txn is assigned a timestamp when they begin(BeginTS) and when they commit (CommitTS).
Each tuple contains two timestamps that represents their visibility and current state:→ BEGIN-TS: The BeginTS of the active txn or the
CommitTS of the committed txn that created it.→ END-TS: The BeginTS of the active txn that created the
next version or infinity or the CommitTS of the committed txn that created it.
5
HIGH-PERFORMANCE CONCURRENCY CONTROL MECHANISMS FOR MAIN-MEMORY DATABASESVLDB 2011
![Page 6: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/6.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
6
READ(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200 Ø
Main Data Table
![Page 7: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/7.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
6
READ(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200 Ø
Main Data Table
![Page 8: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/8.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
6
READ(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200 Ø
Main Data Table
![Page 9: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/9.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
6
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200 Ø
Main Data Table
![Page 10: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/10.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
6
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200 Ø
Main Data Table
![Page 11: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/11.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
6
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200 Ø
A3 Txn@25 ∞ $300
Main Data Table
![Page 12: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/12.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
6
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200 Ø
A3 Txn@25 ∞ $300
Main Data Table
10000000...00000000 00011001Txn@25 →
![Page 13: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/13.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
6
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200 Ø
A3 Txn@25 ∞ $300
Main Data Table
![Page 14: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/14.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
6
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200Txn@25
A3 Txn@25 ∞ $300
Main Data Table
![Page 15: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/15.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
6
READ(A)
WRITE(A)
Thread #1Begin @ 25Commit @ 35
BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200Txn@25
A3 Txn@25 ∞ $300
Main Data Table
![Page 16: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/16.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
6
READ(A)
WRITE(A)
Thread #1Begin @ 25Commit @ 35
BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200Txn@25
A3 Txn@25 ∞ $300
Main Data Table
35
35
![Page 17: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/17.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
6
READ(A)
WRITE(A)
Thread #1Begin @ 25Commit @ 35
BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200Txn@25
A3 Txn@25 ∞ $300
Main Data Table
35
35
REWIND
![Page 18: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/18.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
7
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200Txn@25
A3 Txn@25 ∞ $300
Main Data Table
Thread #2Begin @ 30
![Page 19: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/19.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
7
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200Txn@25
A3 Txn@25 ∞ $300
Main Data Table
READ(A)
Thread #2Begin @ 30
![Page 20: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/20.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
7
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200Txn@25
A3 Txn@25 ∞ $300
Main Data Table
READ(A)
Thread #2Begin @ 30
![Page 21: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/21.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
7
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200Txn@25
A3 Txn@25 ∞ $300
Main Data Table
READ(A)
Thread #2Begin @ 30
![Page 22: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/22.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
7
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200Txn@25
A3 Txn@25 ∞ $300
Main Data Table
READ(A)
WRITE(A)
Thread #2Begin @ 30
![Page 23: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/23.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
7
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200Txn@25
A3 Txn@25 ∞ $300
Main Data Table
READ(A)
WRITE(A)
Thread #2Begin @ 30
![Page 24: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/24.jpg)
15-721 (Spring 2020)
HEKATON: OPERATIONS
7
READ(A)
WRITE(A)
Thread #1Begin @ 25 BEGIN-TS END-TS
A1 10 20
A2 20 ∞
POINTERVALUE
$100
$200Txn@25
A3 Txn@25 ∞ $300
Main Data Table
READ(A)
WRITE(A)
Thread #2Begin @ 30
![Page 25: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/25.jpg)
15-721 (Spring 2020)
HEKATON: TRANSACTION STATE MAP
Global map of all txns’ states in the system:→ ACTIVE: The txn is executing read/write operations.→ VALIDATING: The txn has invoked commit and the
DBMS is checking whether it is valid.→ COMMITTED: The txn is finished but may have not
updated its versions’ TS.→ TERMINATED: The txn has updated the TS for all of
the versions that it created.
8
![Page 26: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/26.jpg)
15-721 (Spring 2020)
HEKATON: TRANSACTION LIFECYCLE
9
Get BeginTS, set state to ACTIVEBEGIN
PRECOMMIT
COMMIT
TERMINATE
Normal Processing
Validation
Post-Processing
Txn Events Txn Phases
Track txn's read set, scan set, and write set.
Get CommitTS, set state to VALIDATING
Validate reads and scans→ If validation OK, write new versions to redo log
Set txn state to COMMITTED
Update version timestamps→ BeginTS in new versions, CommitTS in old versions
Set txn state to TERMINATED
Remove from txn mapSource: Paul Larson
![Page 27: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/27.jpg)
15-721 (Spring 2020)
HEKATON: TRANSACTION META -DATA
Read Set→ Pointers to physical versions returned to access method.
Write Set→ Pointers to versions updated (old and new), versions
deleted (old), and version inserted (new).
Scan Set→ Stores enough information needed to perform each scan
operation again to check result.
Commit Dependencies→ List of txns that are waiting for this txn to finish.
10
![Page 28: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/28.jpg)
15-721 (Spring 2020)
HEKATON: OPTIMISTIC VS. PESSIMISTIC
Optimistic Txns:→ Check whether a version read is still visible at the end of
the txn.→ Repeat all index scans to check for phantoms.
Pessimistic Txns:→ Use shared & exclusive locks on records and buckets.→ No validation is needed.→ Separate background thread to detect deadlocks.
12
![Page 29: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/29.jpg)
15-721 (Spring 2020)
HEKATON: OPTIMISTIC VS. PESSIMISTIC
13
0
0.4
0.8
1.2
1.6
0 6 12 18 24
Thr
ough
put (
txn
/sec
)M
illi
on
s
# Threads
Optimistic Pessimistic
Source: Paul Larson
Database: Single table with 1000 tuplesWorkload: 80% read-only txns + 20% update txns
Processor: 2 sockets, 12 cores
![Page 30: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/30.jpg)
15-721 (Spring 2020)
HEKATON: LESSONS
Use only lock-free data structures→ No latches, spin locks, or critical sections→ Indexes, txn map, memory alloc, garbage collector→ We will discuss Bw-Trees + Skip Lists later…
Only one single serialization point in the DBMS to get the txn’s begin and commit timestamp→ Atomic Addition (CAS)
15
![Page 31: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/31.jpg)
15-721 (Spring 2020)
OBSERVATIONS
Read/scan set validations are expensive if the txnsaccess a lot of data.
Appending new versions hurts the performance of OLAP scans due to pointer chasing & branching.
Record-level conflict checks may be too coarse-grained and incur false positives.
16
![Page 32: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/32.jpg)
15-721 (Spring 2020)
HYPER MVCC
Column-store with delta record versioning.→ In-Place updates for non-indexed attributes→ Delete/Insert updates for indexed attributes.→ Newest-to-Oldest Version Chains→ No Predicate Locks / No Scan Checks
Avoids write-write conflicts by aborting txns that try to update an uncommitted object.
17
FAST SERIALIZABLE MULTI-VERSION CONCURRENCY CONTROL FOR MAIN-MEMORY DATABASE SYSTEMSSIGMOD 2015
![Page 33: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/33.jpg)
15-721 (Spring 2020)
HYPER: STORAGE ARCHITECTURE
18
Delta Storage (Per Txn)Main Data Table
ATTR1
Tupac
IceT
B.I.G.
DrDre
ATTR2
$100
$200
$150
$139
VersionVector
Ø (ATTR2→$122)
Txn #2
Ø
(ATTR2→$199)
Txn #1
Ø
Ø
![Page 34: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/34.jpg)
15-721 (Spring 2020)
HYPER: STORAGE ARCHITECTURE
18
Delta Storage (Per Txn)Main Data Table
ATTR1
Tupac
IceT
B.I.G.
DrDre
ATTR2
$100
$200
$150
$139
VersionVector
Ø (ATTR2→$122)
Txn #2
Ø
(ATTR2→$199)
Txn #1
Ø
Txn #3
(ATTR2→$100)
Ø
![Page 35: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/35.jpg)
15-721 (Spring 2020)
HYPER: STORAGE ARCHITECTURE
18
Delta Storage (Per Txn)Main Data Table
ATTR1
Tupac
IceT
B.I.G.
DrDre
ATTR2
$100
$200
$150
$139
VersionVector
Ø (ATTR2→$122)
Txn #2
Ø
(ATTR2→$199)
Txn #1
Ø
Txn #3
(ATTR2→$100)
Ø
$200
![Page 36: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/36.jpg)
15-721 (Spring 2020)
HYPER: STORAGE ARCHITECTURE
18
Delta Storage (Per Txn)Main Data Table
ATTR1
Tupac
IceT
B.I.G.
DrDre
ATTR2
$100
$200
$150
$139
VersionVector
Ø (ATTR2→$122)
Txn #2
Ø
(ATTR2→$199)
Txn #1
Ø
Txn #3
(ATTR2→$100)
(ATTR2→$139) Ø
Ø
$200
![Page 37: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/37.jpg)
15-721 (Spring 2020)
HYPER: STORAGE ARCHITECTURE
18
Delta Storage (Per Txn)Main Data Table
ATTR1
Tupac
IceT
B.I.G.
DrDre
ATTR2
$100
$200
$150
$139
VersionVector
Ø (ATTR2→$122)
Txn #2
Ø
(ATTR2→$199)
Txn #1
Ø
Txn #3
(ATTR2→$100)
(ATTR2→$139) Ø
$200
$200
![Page 38: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/38.jpg)
15-721 (Spring 2020)
HYPER: VALIDATION
First-Writer Wins→ If version vector is not null, then it always points to the
last committed version.→ Do not need to check whether write-sets overlap.
Check the redo buffers of txns that committed after the validating txn started.→ Compare the committed txn's write set for phantoms
using Precision Locking.→ Only need to store the txn's read predicates and not its
entire read set.
19
![Page 39: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/39.jpg)
15-721 (Spring 2020)
HYPER: PRECISION LOCKING
20
Validating Txn
SELECT * FROM fooWHERE attr2 > 20
AND attr2 < 30
SELECT COUNT(attr1)FROM fooWHERE attr2 IN (10,20,30)
Delta Storage (Per Txn)
Txn #1003
(attr1→'IceCube',attr2→199)
SELECT attr1, AVG(attr2)FROM fooWHERE attr1 LIKE '%Ice%'GROUP BY attr1
HAVING AVG(attr2) > 100
Txn #1002
(attr2→122)
Txn #1001
(attr2→99)
(attr2→33)
99>20 AND 99<30
FALSE
![Page 40: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/40.jpg)
15-721 (Spring 2020)
HYPER: PRECISION LOCKING
20
Validating Txn
SELECT * FROM fooWHERE attr2 > 20
AND attr2 < 30
SELECT COUNT(attr1)FROM fooWHERE attr2 IN (10,20,30)
Delta Storage (Per Txn)
Txn #1003
(attr1→'IceCube',attr2→199)
SELECT attr1, AVG(attr2)FROM fooWHERE attr1 LIKE '%Ice%'GROUP BY attr1
HAVING AVG(attr2) > 100
Txn #1002
(attr2→122)
Txn #1001
(attr2→99)
(attr2→33)FALSE
![Page 41: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/41.jpg)
15-721 (Spring 2020)
HYPER: PRECISION LOCKING
20
Validating Txn
SELECT * FROM fooWHERE attr2 > 20
AND attr2 < 30
SELECT COUNT(attr1)FROM fooWHERE attr2 IN (10,20,30)
Delta Storage (Per Txn)
Txn #1003
(attr1→'IceCube',attr2→199)
SELECT attr1, AVG(attr2)FROM fooWHERE attr1 LIKE '%Ice%'GROUP BY attr1
HAVING AVG(attr2) > 100
Txn #1002
(attr2→122)
Txn #1001
(attr2→99)
(attr2→33)
FALSE
![Page 42: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/42.jpg)
15-721 (Spring 2020)
HYPER: PRECISION LOCKING
20
Validating Txn
SELECT * FROM fooWHERE attr2 > 20
AND attr2 < 30
SELECT COUNT(attr1)FROM fooWHERE attr2 IN (10,20,30)
Delta Storage (Per Txn)
Txn #1003
(attr1→'IceCube',attr2→199)
SELECT attr1, AVG(attr2)FROM fooWHERE attr1 LIKE '%Ice%'GROUP BY attr1
HAVING AVG(attr2) > 100
Txn #1002
(attr2→122)
Txn #1001
(attr2→99)
(attr2→33)
'IceCube' LIKE '%Ice%'TRUE
![Page 43: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/43.jpg)
15-721 (Spring 2020)
HYPER: VERSION SYNOPSES
Store a separate column that tracks the position of the first and last versioned tuple in a block of tuples.
When scanning tuples, the DBMS can check for strides of tuples without older versions and execute more efficiently.
21
Main Data Table
ATTR1
Tupac
IceT
B.I.G.
DrDre
ATTR2
$100
$200
$150
$99
VersionVector
Ø
Ø
Ø
RZA
GZA
ODB
$300
$300
$0
Ø
Ø
VersionSynopsis
[2,5)
![Page 44: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/44.jpg)
15-721 (Spring 2020)
HYPER: VERSION SYNOPSES
Store a separate column that tracks the position of the first and last versioned tuple in a block of tuples.
When scanning tuples, the DBMS can check for strides of tuples without older versions and execute more efficiently.
21
Main Data Table
ATTR1
Tupac
IceT
B.I.G.
DrDre
ATTR2
$100
$200
$150
$99
VersionVector
Ø
Ø
Ø
RZA
GZA
ODB
$300
$300
$0
Ø
Ø
VersionSynopsis
[2,5) 0
1
2
3
4
5
6
Offsets
![Page 45: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/45.jpg)
15-721 (Spring 2020)
HYPER: VERSION SYNOPSES
Store a separate column that tracks the position of the first and last versioned tuple in a block of tuples.
When scanning tuples, the DBMS can check for strides of tuples without older versions and execute more efficiently.
21
Main Data Table
ATTR1
Tupac
IceT
B.I.G.
DrDre
ATTR2
$100
$200
$150
$99
VersionVector
Ø
Ø
Ø
RZA
GZA
ODB
$300
$300
$0
Ø
Ø
VersionSynopsis
[2,5)
![Page 46: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/46.jpg)
15-721 (Spring 2020)
HYPER: VERSION SYNOPSES
Store a separate column that tracks the position of the first and last versioned tuple in a block of tuples.
When scanning tuples, the DBMS can check for strides of tuples without older versions and execute more efficiently.
21
Main Data Table
ATTR1
Tupac
IceT
B.I.G.
DrDre
ATTR2
$100
$200
$150
$99
VersionVector
Ø
Ø
Ø
RZA
GZA
ODB
$300
$300
$0
Ø
Ø
VersionSynopsis
[2,5)
![Page 47: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/47.jpg)
15-721 (Spring 2020)
SAP HANA
In-memory HTAP DBMS with time-travel version storage (N2O).→ Supports both optimistic and pessimistic MVCC.→ Latest versions are stored in time-travel space.→ Hybrid storage layout (row + columnar).
Based on P*TIME, TREX, and MaxDB.
First released in 2012.
22
EFFICIENT TRANSACTION PROCESSING IN SAP HANA DATABASE: THE END OF A COLUMN-STORE MYTHSIGMOD 2012
![Page 48: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/48.jpg)
15-721 (Spring 2020)
SAP HANA: VERSION STORAGE
Store the oldest version in the main data table.
Each tuple maintains a flag to denote whether there exists newer versions in the version space.
Maintain a separate hash table that maps record identifiers to the head of version chain.
23
![Page 49: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/49.jpg)
15-721 (Spring 2020)
SAP HANA: VERSION STORAGE
24
Main Data Table Version Storage
Hash Table
A
C
D
RECORDA3 A2
C5 C4 C3
D8 D7
D
A
B
C
True
VERS?
True
False
True
RID
-
DATA
-
-
-
D6
VERSION
A1
B3
C2
![Page 50: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/50.jpg)
15-721 (Spring 2020)
SAP HANA: TRANSACTIONS
Instead of embedding meta-data about the txn that created a version with the data, store a pointer to a context object.→ Reads are slower because you must follow pointers.→ Large updates are faster because it's a single write to
update the status of all tuples.
Store meta-data about whether a txn has committed in a separate object as well.
25
![Page 51: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/51.jpg)
15-721 (Spring 2020)
SAP HANA: VERSION STORAGE
26
Main Data Table Version Storage
Hash Table
A
C
D
RECORDA3 A2
C5 C4 C3
D8 D7
Txn Meta-Data
Txn Contexts Group Commit Context
Tid=1 Tid=2 Tid=3
D
A
B
C
True
VERS?
True
False
True
RID
-
DATA
-
-
-
D6
VERSION
A1
B3
C2
Tid = 3Thread #1
WRITE(C) WRITE(D)Group 1
![Page 52: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/52.jpg)
15-721 (Spring 2020)
MVCC LIMITATIONS
Computation & Storage Overhead→ Most MVCC schemes use indirection to search a tuple's
version chain. This increases CPU cache misses.→ Also requires frequent garbage collection to minimize the
number versions that a thread must evaluate.
Shared Memory Writes→ Most MVCC schemes store versions in "global" memory
in the heap without considering locality.
Timestamp Allocation→ All threads access single shared counter.
27
Source: Hyeontaek Lim
![Page 53: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/53.jpg)
15-721 (Spring 2020)
OCC LIMITATIONS
Frequent Aborts→ Txns will abort too quickly under high contention,
causing high churn.
Extra Reads & Writes→ Each txn must copy tuples into their private workspace to
ensure repeatable reads. It then has to check whether it read consistent data when it commits.
Index Contention→ Txns install "virtual" index entries to ensure unique-key
invariants.
28
Source: Hyeontaek Lim
![Page 54: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/54.jpg)
15-721 (Spring 2020)
CMU CICADA
In-memory OLTP engine based on optimistic MVCC with append-only storage (N2O).→ Best-effort Inlining→ Loosely Synchronized Clocks→ Contention-Aware Validation→ Index Nodes Stored in Tables
Designed to be scalable for both low- and high-contention workloads.
29
CICADA: DEPENDABLY FAST MULTI-CORE IN-MEMORY TRANSACTIONSSIGMOD 2017
![Page 55: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/55.jpg)
15-721 (Spring 2020)
Record Meta-data
CICADA: BEST-EFFORT INLINING
Record meta-data is stored in a fixed location.
Threads will attempt to inline read-mostly version within this meta-data to reduce version chain traversals.
30
POINTER LATEST VERSION
EMPTY
KEY VALUE
XXX $111
POINTER
KEY VALUE
YYY $222
POINTER
![Page 56: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/56.jpg)
15-721 (Spring 2020)
CICADA: FAST VALIDATION
Contention-aware Validation→ Validate access to recently modified records first.
Early Consistency Check→ Pre-validate access set before making global writes.
Incremental Version Search→ Resume from last search location in version list.
31
Source: Hyeontaek Lim
Skip if all recent txnscommitted successfully.
![Page 57: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/57.jpg)
15-721 (Spring 2020)
CICADA: INDEX STORAGE
32
Index Node Table
NODE DATA
A1Keys→[100,200]Pointers→[B,C]
POINTER
B2Keys→[50,70]Pointers→[D,E]
E3Keys→[10,30]Pointers→[RID,RID]
Ø
B1Keys→[52,70]Pointers→[D,E] Ø
Index
A
B C
D E F G
E2Keys→[11,30]Pointers→[RID,RID]
E1Keys→[12,30]Pointers→[RID,RID]
![Page 58: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/58.jpg)
15-721 (Spring 2020)
CICADA: INDEX STORAGE
32
Index Node Table
NODE DATA
A1Keys→[100,200]Pointers→[B,C]
POINTER
B2Keys→[50,70]Pointers→[D,E]
E3Keys→[10,30]Pointers→[RID,RID]
Ø
B1Keys→[52,70]Pointers→[D,E] Ø
Index
A
B C
D E F G
E2Keys→[11,30]Pointers→[RID,RID]
E1Keys→[12,30]Pointers→[RID,RID]
![Page 59: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/59.jpg)
15-721 (Spring 2020)
CICADA: LOW CONTENTION
33
0
10
20
30
40
50
0 6 12 18 24
Thr
ough
put (
txn
/sec
)
Mil
lio
ns
# Threads
2PL Silo Silo' TicTocFOEDUS Hekaton ERMIA Cicada
Workload: YCSB (95% read / 5% write) - 1 op per txn
Source: Hyeontaek Lim
![Page 60: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/60.jpg)
15-721 (Spring 2020)
CICADA: HIGH CONTENTION
34
0
0.11
0.22
0.33
0 6 12 18 24
Thr
ough
put (
txn
/sec
)M
illi
on
s
# Threads
2PL Silo Silo' TicTocFOEDUS Hekaton ERMIA Cicada
Workload: TPC-C (1 Warehouse)
Source: Hyeontaek Lim
![Page 61: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/61.jpg)
15-721 (Spring 2020)
PARTING THOUGHTS
There are several other implementation factors for an MVCC DBMS beyond the four main design decisions that we discussed last class.
Need to balance the trade-offs between indirection and performance.
35
![Page 62: 04 ADVANCED DATABASE SYSTEMS - CMU 15-721 · 04 Multi-Version Concurrency Control (Protocols) @Andy_Pavlo // 15-721 // Spring 2020 ADVANCED DATABASE SYSTEMS](https://reader034.fdocuments.net/reader034/viewer/2022050602/5faa13d28839bc61da5f9f90/html5/thumbnails/62.jpg)
15-721 (Spring 2020)
NEXT CL ASS
MVCC Garbage Collection
Perf Tutorial for Project #1
36