New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL...
Transcript of New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL...
![Page 1: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/1.jpg)
Click to edit Master Date
Strictly private and confidential — © OmniTI Inc. 2010OmniTI — Corporate Presentation
Engineering better user experience in
Web applications & Internet architectures
Strictly private and confidential — © OmniTI Inc. 2012OmniTI — Corporate Presentation
Engineering better user experience in
Web applications & Internet architectures
May 16th, 2012
Using Postgres' built in replication facilities
Getting Hot and Streamy with Postgres
![Page 2: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/2.jpg)
Who am I?
Phil SorberDBA @ OmniTI
o We do full stack development including databaseso We build large siteso Our website - http://omniti.como Surge - http://omniti.com/surge/2012o We're hiring - http://omniti.com/is/hiring
Contact meo [email protected] @PhilSorbero Blog - http://philsorber.blogspot.com
![Page 3: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/3.jpg)
It's just another block in the WAL
Write Ahead Logo roll-forward recovery aka REDOo flushed to disk to guarantee commit durabilityo sequential writeso lower cost than flushing page cache
Allows us to do cool thingso Crash recoveryo Binary backupso Replication!
![Page 4: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/4.jpg)
What's Logged?
wal_levelo minimal (crash recovery)o archive (disaster recovery + replication)o hot_standby
almost everything, except...o unlogged tables (hence the name)o temporary tableso hash indexes?! (generally don't use these)
More infoo Attend Amit's talk in MRT 219 tomorrow @ 4:30
![Page 5: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/5.jpg)
PITR (Point In Time Recovery)
Introduced in 8.0Used for
o On-line Backup - pg_start/stop_backup
Config Options (Master)o archive_mode=on (8.3) (RR)o archive_commando archive_timeout (8.2)o wal_level=archive (9.0) (RR)
Toolso omnipitr-archiveo omnipitr-backup-master
![Page 6: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/6.jpg)
PITR
PITR (Point In Time Recovery) cont.
![Page 7: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/7.jpg)
Warm Standby (Log Shipping)
Introduced in 8.2 (technically possible since 8.0)Used for
o Failovero On-line Backup (not officially supported)
Config Option (Slave) - restore_command (RR)Tools
o omnipitr-syncho pg_basebackup (9.1)o omnipitr-restoreo pg_standby (8.3)o omnipitr-backup-slave
![Page 8: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/8.jpg)
Warm Standby
Warm Standby (Log Shipping) cont.
![Page 9: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/9.jpg)
Streaming Replication
Introduced in 9.0Used for
o Near real-time replication
Benefits over Log Shippingo Less hassle in setting up archive and restore commandso Less lag due to low activity (archive_timeout)
Disadvantageso Master does not keep WAL segments indefinitelyo More potential load on the master
![Page 10: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/10.jpg)
Streaming Warm Standby
Streaming Replication cont.
![Page 11: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/11.jpg)
Streaming Replication II
WAL kept in synco wal_sender reads from WAL on disk (usually cached)o wal_receiver writes to WAL on disk
Hybrid Modeo Streaming Replication & Log Shippingo Lets the slaves sustain more down timeo Less IO on master used for finding "old" WALo Still have to deal with the hassle of log shipping
pg_hba.confo replication pseudo databaseo replication privilege (9.1)
![Page 12: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/12.jpg)
Hybrid Streaming Warm Standby
Streaming Replication II cont.
![Page 13: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/13.jpg)
Streaming Replication III
Config OptionsMaster
o max_wal_senders (RR)o wal_keep_segmentso wal_level=archive (RR)o archive_timeout=0
![Page 14: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/14.jpg)
Streaming Replication IV
Config OptionsSlave (recovery.conf) (RR)
o standby_mode=ono primary_conninfo
'host=192.168.1.50 port=5432 user=foo password=foopass'o trigger_fileo archive_cleanup_commando recovery_target_timeline (8.0,9.1)o restore_command
![Page 15: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/15.jpg)
Streaming Replication V
Toolso pg_basebackupo omnipitr-syncho pg_archivecleanupo omnipitr-cleanup
![Page 16: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/16.jpg)
Hot Standby
Introduced in 9.0Independent of Streaming ReplicationRead only queries
o SELECT (no functions that alter)o pg_dump
Conflictso Soft (I/O)o Hard (Updates)
![Page 17: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/17.jpg)
Hybrid Streaming Hot Standby
Hot Standby cont.
![Page 18: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/18.jpg)
Hot Standby II
Config OptionsMaster
o wal_level=hot_standby (RR)o vacuum_defer_cleanup_age
Slaveo hot_standby=on (RR)o hot_standby_feedbacko max_standby_archive_delayo max_standby_streaming_delay
![Page 19: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/19.jpg)
Synchronous Streaming Replication
Introduced in 9.1
Also known as 2-safe replication
Commit does not return until data written to WAL and flushed to disk of both primary and standby
Read only transactions and rollbacks do not wait
![Page 20: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/20.jpg)
Hybrid Synchronous Streaming Hot Standby
Synchronous Streaming Replication cont.
![Page 21: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/21.jpg)
Synchronous Replication II
Config Optionssynchronous_commit = on (default)
o Can be set per transaction
synchronous_standby_nameso Only first entry must respondo Failover listo Set to empty for asynchronous behavior
![Page 22: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/22.jpg)
Synchronous Replication III
Pro'so Greater durability
Con'so Latencyo Contentiono Points of failure
![Page 23: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/23.jpg)
Multiple Standby Failover
Timeline Changes
Missing and extra transactions
History Files
![Page 24: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/24.jpg)
Pre Failover
Multiple Standby Failover cont.
![Page 25: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/25.jpg)
Post Failover
Multiple Standby Failover cont.
![Page 26: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/26.jpg)
Cascading Replication
Coming in 9.2
Same options as regular streaming
Asynchronous only
Hot standby feedback propagates upstream
Promoting cascading standby terminates downstream replication
![Page 27: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/27.jpg)
Cascading Replication
Cascading Replication cont.
![Page 28: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/28.jpg)
Hybrid Cascading Replication
Cascading Replication cont.
![Page 29: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/29.jpg)
Questions?
![Page 30: New Getting Hot and Streamy with Postgres - PGCon · 2020. 1. 4. · Streaming Replication II WAL kept in sync o wal_sender reads from WAL on disk (usually cached) o wal_receiver](https://reader035.fdocuments.net/reader035/viewer/2022070112/6057d865d8659f43c4132679/html5/thumbnails/30.jpg)
Thank You!
http://omniti.com
+1 443 325 1357