NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document...
Transcript of NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document...
![Page 1: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/1.jpg)
NOSQL FOR
POSTGRESQL
BEST PRACTICES
DMITRY DOLGOV
12-05-2017
![Page 2: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/2.jpg)
: Jsonb internals and performance-related factors
: Tricky queries
: Benchmarks
: How to shoot yourself in the foot
1
![Page 3: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/3.jpg)
: Jsonb internals and performance-related factors
: Tricky queries
: Benchmarks
: How to shoot yourself in the foot
1
![Page 4: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/4.jpg)
: Jsonb internals and performance-related factors
: Tricky queries
: Benchmarks
: How to shoot yourself in the foot
1
![Page 5: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/5.jpg)
: Jsonb internals and performance-related factors
: Tricky queries
: Benchmarks
: How to shoot yourself in the foot
1
![Page 6: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/6.jpg)
: Jsonb internals and performance-related factors
: Tricky queries
: Benchmarks
: How to shoot yourself in the foot
1
![Page 7: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/7.jpg)
2
![Page 8: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/8.jpg)
AWS EC2
m4.large instance
separate instance (database and generator)
16GB memory, 4 core 2.3GHz
Ubuntu 16.04
Same VPC and placement group
AMI that supports HVM virtualization type
at least 4 rounds of benchmark
3
![Page 9: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/9.jpg)
PostgreSQL 9.6.3/10
MySQL 5.7.9/8.0
MongoDB 3.4.4
YCSB 0.13
106 rows and operations
4
![Page 10: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/10.jpg)
Configuration
shared_buffers
effective_cache_size
max_wal_size
innodb_buffer_pool_size
innodb_log_file_size
write concern level (journaled or transaction_sync)
checkpoint
eviction
5
![Page 11: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/11.jpg)
Document types
“simple” document
10 key/value pairs (100 characters)
“large” document
100 key/value pairs (200 characters)
“complex” document
100 keys, 3 nesting levels (100 characters)
6
![Page 12: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/12.jpg)
Performance-related factors
: On-disk representation
: In-memory representation
: Indexing support
7
![Page 13: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/13.jpg)
Performance-related factors
: On-disk representation
: In-memory representation
: Indexing support
7
![Page 14: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/14.jpg)
Performance-related factors
: On-disk representation
: In-memory representation
: Indexing support
7
![Page 15: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/15.jpg)
Performance-related factors
: On-disk representation
: In-memory representation
: Indexing support
7
![Page 16: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/16.jpg)
Indexing support
: Postgresql – single path, multiple paths, entire doc-
ument
: MongoDB – single path, multiple paths
: MySQL – virtual columns, single path, multiple paths
8
![Page 17: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/17.jpg)
PG indexing details
: jsonb_path
: jsonb_path_ops
9
![Page 18: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/18.jpg)
Select, GIN
”simple” document
jsonb_path_ops
10
![Page 19: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/19.jpg)
11
![Page 20: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/20.jpg)
12
![Page 21: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/21.jpg)
13
![Page 22: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/22.jpg)
Select, BTree
”simple” document
btree
14
![Page 23: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/23.jpg)
15
![Page 24: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/24.jpg)
Internals
![Page 25: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/25.jpg)
17
![Page 26: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/26.jpg)
Jsonb
document size
node
node
JEntry
content
...
18
![Page 27: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/27.jpg)
Jsonb
document size
node
node
JEntry
content
...
19
![Page 28: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/28.jpg)
Jsonb Header
type
number of items
JEntry
length or offset?
value type
value length or offset
20
![Page 29: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/29.jpg)
JB_OFFSET_STRIDE
: JEntry may contains a value lenght or offset
: Offset = access speed
: Length = compressibility
: Every JB_OFFSET_STRIDE’th JEntry contains an offset
: Rest of them contain length
21
![Page 30: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/30.jpg)
Bson
document size
node
node
Header
Content
...
22
![Page 31: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/31.jpg)
Bson
document size
node
node
Header
Content
...
23
![Page 32: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/32.jpg)
Bson Header
Value type
Key name
Value size
24
![Page 33: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/33.jpg)
MySQL Json
node
node
Type
Value
...
25
![Page 34: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/34.jpg)
MySQL Json
node
node
Type
Value
...
26
![Page 35: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/35.jpg)
MySQL Json Object
Count of elements
Size
Pointers to keys
Pointers to values
Keys
Values27
![Page 36: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/36.jpg)
Bson
Key
Value
Key
Value
Key
Value
...
Jsonb/MySQL Json
Key
Key
Value
Key
Value
Value
...
28
![Page 37: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/37.jpg)
{”a”: 3, ”b”: ”xyz”}
29
![Page 38: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/38.jpg)
select pg_relation_filepath(oid),relpages from pg_classwhere relname = ’table_name’;
pg_relation_filepath | relpages———————-+———-base/40960/325477 | 0
(1 row)
30
![Page 39: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/39.jpg)
bson.dumps({”a”: 3, ”b”: u”xyz”})
31
![Page 40: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/40.jpg)
$ hexdump -C database/table.ibd
32
![Page 41: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/41.jpg)
Alignment
Variable-length portion is aligned to a 4-byte
insert into testvalues(’{”a”: ”aa”, ”b”: 1}’);
insert into testvalues(’{”a”: 1, ”b”: ”aa”}’);
33
![Page 42: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/42.jpg)
Select, BTree
”complex” document
btree
34
![Page 43: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/43.jpg)
35
![Page 44: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/44.jpg)
TOAST_TUPLE_THRESHOLD
”simple” document
40 threads
different document size
select
36
![Page 45: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/45.jpg)
37
![Page 46: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/46.jpg)
TOAST
Jsonb Compression Chunks Toast table
: TOAST_TUPLE_THRESHOLD bytes (normally 2 kB)
: PostgreSQL and MySQL use LZ variation
: MongoDB uses snappy block compression
38
![Page 47: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/47.jpg)
In-memory representation
: Tree-like representation (JsonbValue, Document, Json_dom)
: Little bit more expensive but more convenient to work with
: Mostly in use to modify data (except MySQL)
: Most of the read operations use on-disk representation
39
![Page 48: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/48.jpg)
Insert
”simple” document
journaled
40
![Page 49: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/49.jpg)
41
![Page 50: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/50.jpg)
42
![Page 51: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/51.jpg)
43
![Page 52: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/52.jpg)
44
![Page 53: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/53.jpg)
45
![Page 54: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/54.jpg)
: Update one field of a document
: DETOAST of a document
(select, constraints, procedures etc.)
: Reindex of an entire document
46
![Page 55: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/55.jpg)
Index update
”simple” document
Update one field
47
![Page 56: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/56.jpg)
48
![Page 57: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/57.jpg)
Update 50%, Select 50%
”simple” document
Update one field
journaled
max wal size 5GB
49
![Page 58: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/58.jpg)
50
![Page 59: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/59.jpg)
Update 50%, Select 50%
”large” document
Update one field
51
![Page 60: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/60.jpg)
52
![Page 61: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/61.jpg)
Journal
{”aaa”: ”aaa”,”bbb”: ”bbb”,”ccc”: ”ccc”
}
{”aaa”: ”ddd”,”bbb”: ”bbb”,”ccc”: ”ccc”
}
53
![Page 62: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/62.jpg)
Postgresql
Mysql
MongoDB
54
![Page 63: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/63.jpg)
Document slice
”large” document
One field from a document
55
![Page 64: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/64.jpg)
select data->’key1’->’key2’ from table;
select data->’key1’, data->’key2’ from table;
56
![Page 65: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/65.jpg)
select data->‘key1‘->‘key2‘ from table;
select data->’key1’, data->’key2’ from table;
56
![Page 66: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/66.jpg)
select data->’key1’->’key2’ from table;
select data->‘key1‘, data->‘key2‘ from table;
56
![Page 67: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/67.jpg)
57
![Page 68: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/68.jpg)
Document slice
”large” document
10 fields from a document
58
![Page 69: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/69.jpg)
59
![Page 70: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/70.jpg)
Solutions?
: set storage external
: different query
60
![Page 71: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/71.jpg)
Document slice
create type test as (”a” text, ”b” text);insert into test_jsonbvalues(’{”a”: 1, ”b”: 2, ”c”: 3}’);select q.* from test_jsonb,jsonb_populate_record(NULL::test, data) as q;
a | b—+—1 | 2
(1 row)61
![Page 72: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/72.jpg)
Document slice
create type test as (”a” text, ”b” text);insert into test_jsonbvalues(’{”a”: 1, ”b”: 2, ”c”: 3}’);select q.* from test_jsonb,jsonb_populate_record (NULL::test, data) as q;
a | b—+—1 | 2
(1 row)61
![Page 73: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/73.jpg)
Queries
![Page 74: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/74.jpg)
Pitfalls
: No Json path out of the box (jsquery, SQL/JSON)
: Queries with an array somewhere in the middle
: Iterating through document
: Update inside document
63
![Page 75: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/75.jpg)
[{”items”: [
{”id”: 1, ”value”: ”aaa”},{”id”: 2, ”value”: ”bbb”}
]}, {
”items”: [{”id”: 3, ”value”: ”aaa”},{”id”: 4, ”value”: ”bbb”}
]}]
64
![Page 76: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/76.jpg)
WITH items AS (SELECT jsonb_array_elements(data->’items’)AS item FROM test
)SELECT * FROM itemsWHERE item->>’value’ = ’aaa’;item—————————{”id”: 1, ”value”: ”aaa”}{”id”: 3, ”value”: ”aaa”}
(2 rows)
65
![Page 77: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/77.jpg)
WITH items AS (SELECT jsonb_array_elements (data->’items’)AS item FROM test
)SELECT * FROM itemsWHERE item->>’value’ = ’aaa’;item—————————{”id”: 1, ”value”: ”aaa”}{”id”: 3, ”value”: ”aaa”}
(2 rows)
65
![Page 78: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/78.jpg)
{”items”: {
”item1”: {”status”: true},”item2”: {”status”: true},”item3”: {”status”: false}
}}
66
![Page 79: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/79.jpg)
WITH items AS (SELECT jsonb_each(data->’items’)AS item FROM test
)SELECT (item).key FROM itemsWHERE (item).value->>’status’ = ’true’;key——-item1item2(2 rows)
67
![Page 80: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/80.jpg)
WITH items AS (SELECT jsonb_each (data->’items’)AS item FROM test
)SELECT (item).key FROM itemsWHERE (item).value->>’status’ = ’true’;key——-item1item2(2 rows)
67
![Page 81: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/81.jpg)
68
![Page 82: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/82.jpg)
SQL vs JSONB
”simple” document
btree
insert
69
![Page 83: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/83.jpg)
70
![Page 84: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/84.jpg)
SQL vs JSONB
”simple” document
btree
select
71
![Page 85: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/85.jpg)
72
![Page 86: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/86.jpg)
JSON vs JSONB
”simple” document
btree
insert
73
![Page 87: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/87.jpg)
74
![Page 88: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/88.jpg)
JSON vs JSONB
”simple” document
btree
select
75
![Page 89: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/89.jpg)
76
![Page 90: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/90.jpg)
: Jsonb is more that good for many use cases
: Reasons for performance difference is mostly
database itself
: Benchmarks above are only ”hints”
: You need your own tests
77
![Page 91: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/91.jpg)
: Jsonb is more that good for many use cases
: Reasons for performance difference is mostly
database itself
: Benchmarks above are only ”hints”
: You need your own tests
77
![Page 92: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/92.jpg)
: Jsonb is more that good for many use cases
: Reasons for performance difference is mostly
database itself
: Benchmarks above are only ”hints”
: You need your own tests
77
![Page 93: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/93.jpg)
: Jsonb is more that good for many use cases
: Reasons for performance difference is mostly
database itself
: Benchmarks above are only ”hints”
: You need your own tests
77
![Page 94: NOSQL INSIDE SQL - BEST PRACTICES · 2017. 12. 5. · Documenttypes “simple”document 10key/valuepairs(100characters) “large”document 100key/valuepairs(200characters) “complex”document](https://reader034.fdocuments.net/reader034/viewer/2022052104/603f018c3ad95c7cb951af3c/html5/thumbnails/94.jpg)
Questions?
github.com/erthalion
@erthalion
9erthalion6 at gmail dot com
78