Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability...
Transcript of Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability...
![Page 1: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/1.jpg)
Agenda Isolate Services Durability Availability Conclusion
Implementing High Availabilitywith PostgreSQL
Dimitri [email protected]
25 Oct. 2012
Dimitri [email protected] Implementing High Availability
![Page 2: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/2.jpg)
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
![Page 3: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/3.jpg)
Agenda Isolate Services Durability Availability Conclusion
Dimitri Fontaine
2ndQuadrant FrancePostgreSQL Major Contributor
pgloader, prefix, skytools, debian, . . .CREATE EXTENSION
CREATE EVENT TRIGGER
Bi-Directional ReplicationPartitioning
Dimitri [email protected] Implementing High Availability
![Page 4: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/4.jpg)
Agenda Isolate Services Durability Availability Conclusion
Dimitri Fontaine
2ndQuadrant FrancePostgreSQL Major Contributor
pgloader, prefix, skytools, debian, . . .CREATE EXTENSION
CREATE EVENT TRIGGER
Bi-Directional ReplicationPartitioning
Dimitri [email protected] Implementing High Availability
![Page 5: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/5.jpg)
Agenda Isolate Services Durability Availability Conclusion
Dimitri Fontaine
2ndQuadrant FrancePostgreSQL Major Contributor
pgloader, prefix, skytools, debian, . . .CREATE EXTENSION
CREATE EVENT TRIGGER
Bi-Directional ReplicationPartitioning
Dimitri [email protected] Implementing High Availability
![Page 6: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/6.jpg)
Agenda Isolate Services Durability Availability Conclusion
Proven Architectures, implemented lots at Hi-Media
Hi-Media (turnover scale: 200 millions e)
3 different activities to get money from the web
Allopass, HiPay: internet (micro) paymentTelecom ServiceAdvertising
PostgreSQL is the heart of the technical platform
Business needs complianceCapacity to adapt to changes
Dimitri [email protected] Implementing High Availability
![Page 7: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/7.jpg)
Agenda Isolate Services Durability Availability Conclusion
Proven Architectures, implemented lots at Hi-Media
Hi-Media (turnover scale: 200 millions e)
3 different activities to get money from the web
Allopass, HiPay: internet (micro) paymentTelecom ServiceAdvertising
PostgreSQL is the heart of the technical platform
Business needs complianceCapacity to adapt to changes
Dimitri [email protected] Implementing High Availability
![Page 8: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/8.jpg)
Agenda Isolate Services Durability Availability Conclusion
Proven Architectures, implemented lots at Hi-Media
Hi-Media (turnover scale: 200 millions e)
3 different activities to get money from the web
Allopass, HiPay: internet (micro) paymentTelecom ServiceAdvertising
PostgreSQL is the heart of the technical platform
Business needs complianceCapacity to adapt to changes
Dimitri [email protected] Implementing High Availability
![Page 9: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/9.jpg)
Agenda Isolate Services Durability Availability Conclusion
Proven Architectures, implemented lots at Hi-Media
Hi-Media (turnover scale: 200 millions e)
3 different activities to get money from the web
Allopass, HiPay: internet (micro) paymentTelecom ServiceAdvertising
PostgreSQL is the heart of the technical platform
Business needs complianceCapacity to adapt to changes
Dimitri [email protected] Implementing High Availability
![Page 10: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/10.jpg)
Agenda Isolate Services Durability Availability Conclusion
PostgreSQL: Your data is our job
How to ensure both duralibity and availability of your data?
Usual needs:
ReliabilityStabilityPerformancesGrowth capacity (think commercial success)Continuity and Innovation
Dimitri [email protected] Implementing High Availability
![Page 11: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/11.jpg)
Agenda Isolate Services Durability Availability Conclusion
PostgreSQL: Your data is our job
How to ensure both duralibity and availability of your data?
Usual needs:
ReliabilityStabilityPerformancesGrowth capacity (think commercial success)Continuity and Innovation
Dimitri [email protected] Implementing High Availability
![Page 12: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/12.jpg)
Agenda Isolate Services Durability Availability Conclusion
PostgreSQL: Your data is our job
How to ensure both duralibity and availability of your data?
Usual needs:
ReliabilityStabilityPerformancesGrowth capacity (think commercial success)Continuity and Innovation
Dimitri [email protected] Implementing High Availability
![Page 13: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/13.jpg)
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
![Page 14: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/14.jpg)
Agenda Isolate Services Durability Availability Conclusion
Glossary
Some vocabulary
AvailabilityDuralibity (ACID)ArchitecturesReplications
Dimitri [email protected] Implementing High Availability
![Page 15: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/15.jpg)
Agenda Isolate Services Durability Availability Conclusion
Glossary
Some vocabulary
AvailabilityDuralibity (ACID)ArchitecturesReplications
Dimitri [email protected] Implementing High Availability
![Page 16: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/16.jpg)
Agenda Isolate Services Durability Availability Conclusion
Glossary
Some vocabulary
AvailabilityDuralibity (ACID)ArchitecturesReplications
Dimitri [email protected] Implementing High Availability
![Page 17: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/17.jpg)
Agenda Isolate Services Durability Availability Conclusion
Glossary
Some vocabulary
Availability of services or of data?Duralibity (ACID)ArchitecturesReplications
Dimitri [email protected] Implementing High Availability
![Page 18: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/18.jpg)
Agenda Isolate Services Durability Availability Conclusion
Glossary
Some vocabulary
Availability of services or of data?Duralibity (ACID)ArchitecturesReplications
Dimitri [email protected] Implementing High Availability
![Page 19: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/19.jpg)
Agenda Isolate Services Durability Availability Conclusion
Glossary
Some vocabulary
Availability of services or of data?Duralibity (ACID)ArchitecturesReplications
Dimitri [email protected] Implementing High Availability
![Page 20: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/20.jpg)
Agenda Isolate Services Durability Availability Conclusion
Needs first
Needs evolve, solutions must adapt
Start simpleSome first classic stepsHigh Availability of DataHigh Availability of ServicesRead Only Load BalancingRead Write Load Balancing
Dimitri [email protected] Implementing High Availability
![Page 21: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/21.jpg)
Agenda Isolate Services Durability Availability Conclusion
Needs first
Needs evolve, solutions must adapt
Start simpleSome first classic stepsHigh Availability of DataHigh Availability of ServicesRead Only Load BalancingRead Write Load Balancing
Dimitri [email protected] Implementing High Availability
![Page 22: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/22.jpg)
Agenda Isolate Services Durability Availability Conclusion
Needs first
Needs evolve, solutions must adapt
Start simpleSome first classic stepsHigh Availability of DataHigh Availability of ServicesRead Only Load BalancingRead Write Load Balancing
Dimitri [email protected] Implementing High Availability
![Page 23: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/23.jpg)
Agenda Isolate Services Durability Availability Conclusion
Let’s start simple
Our projet life cycle
Let’s start with the example of a quite simpleproject released as a web application seeing itsneeds evolve with its success.
Dimitri [email protected] Implementing High Availability
![Page 24: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/24.jpg)
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
![Page 25: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/25.jpg)
Agenda Isolate Services Durability Availability Conclusion
Scaling out 101
Services Availability
Front servers are statelessWatch out for max_connectionsDon’t you use persistent connections!pgbouncer
Dimitri [email protected] Implementing High Availability
![Page 26: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/26.jpg)
Agenda Isolate Services Durability Availability Conclusion
Scaling out 101
Services Availability
Front servers are statelessWatch out for max_connectionsDon’t you use persistent connections!pgbouncer
Dimitri [email protected] Implementing High Availability
![Page 27: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/27.jpg)
Agenda Isolate Services Durability Availability Conclusion
Scaling out 101
Services Availability
Front servers are statelessWatch out for max_connectionsDon’t you use persistent connections!pgbouncer
Dimitri [email protected] Implementing High Availability
![Page 28: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/28.jpg)
Agenda Isolate Services Durability Availability Conclusion
Scaling out 101
Using more than a single server and a connection pool
Dimitri [email protected] Implementing High Availability
![Page 29: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/29.jpg)
Agenda Isolate Services Durability Availability Conclusion
pgbouncer
pgbouncer is able to reuse client and server side connections.
Dimitri [email protected] Implementing High Availability
![Page 30: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/30.jpg)
Agenda Isolate Services Durability Availability Conclusion
pgbouncer
pgbouncer is able to reuse client and server side connections.
Dimitri [email protected] Implementing High Availability
![Page 31: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/31.jpg)
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
![Page 32: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/32.jpg)
Agenda Isolate Services Durability Availability Conclusion
Getting serious: backups
Backup Strategy is the single most important step towards dataavailability
Nightly pg_dump -Fc
Don’t forget pg_dumpall–globals-only
Data Retention7 days of nightly backups7 weeks of weekly backups12 months of monthly backups30 years of yearly backups?
Dimitri [email protected] Implementing High Availability
![Page 33: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/33.jpg)
Agenda Isolate Services Durability Availability Conclusion
Getting serious: backups
Backup Strategy is the single most important step towards dataavailability
Nightly pg_dump -Fc
Don’t forget pg_dumpall–globals-only
Data Retention7 days of nightly backups7 weeks of weekly backups12 months of monthly backups30 years of yearly backups?
Dimitri [email protected] Implementing High Availability
![Page 34: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/34.jpg)
Agenda Isolate Services Durability Availability Conclusion
Getting serious: backups
Backup Strategy is the single most important step towards dataavailability
Nightly pg_dump -Fc
Don’t forget pg_dumpall–globals-only
Data Retention7 days of nightly backups7 weeks of weekly backups12 months of monthly backups30 years of yearly backups?
Dimitri [email protected] Implementing High Availability
![Page 35: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/35.jpg)
Agenda Isolate Services Durability Availability Conclusion
Failover, 101
pg_dump, pg_restore
protection against errors and omissionsbeware of restoring timestill a must have for data durabilitywhat about data availability?
Dimitri [email protected] Implementing High Availability
![Page 36: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/36.jpg)
Agenda Isolate Services Durability Availability Conclusion
Failover, 101
pg_dump, pg_restore
protection against errors and omissionsbeware of restoring timestill a must have for data durabilitywhat about data availability?
Dimitri [email protected] Implementing High Availability
![Page 37: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/37.jpg)
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
![Page 38: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/38.jpg)
Agenda Isolate Services Durability Availability Conclusion
Failover, 201
Using physical backups and Point In Time Recovery
Point In Time Recovery, 8.1warm standby, 8.2Archiving and crash recoveryarchive_command
restore_command
walmgr.py, WAL-E
Dimitri [email protected] Implementing High Availability
![Page 39: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/39.jpg)
Agenda Isolate Services Durability Availability Conclusion
Failover, 201
Using physical backups and Point In Time Recovery
Point In Time Recovery, 8.1warm standby, 8.2Archiving and crash recoveryarchive_command
restore_command
walmgr.py, WAL-E
Dimitri [email protected] Implementing High Availability
![Page 40: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/40.jpg)
Agenda Isolate Services Durability Availability Conclusion
Failover, 201
Using physical backups and Point In Time Recovery
Point In Time Recovery, 8.1warm standby, 8.2Archiving and crash recoveryarchive_command
restore_command
walmgr.py, WAL-E
Dimitri [email protected] Implementing High Availability
![Page 41: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/41.jpg)
Agenda Isolate Services Durability Availability Conclusion
Failover, 201
Using physical backups and Point In Time Recovery
Point In Time Recovery, 8.1warm standby, 8.2Archiving and crash recoveryarchive_command
restore_command
walmgr.py, WAL-E
Dimitri [email protected] Implementing High Availability
![Page 42: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/42.jpg)
Agenda Isolate Services Durability Availability Conclusion
Warm Standby
Implementing Warm Standby
Dimitri [email protected] Implementing High Availability
![Page 43: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/43.jpg)
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
![Page 44: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/44.jpg)
Agenda Isolate Services Durability Availability Conclusion
Application Split
When you have separate backoffice and production requirements
Cross replicationSlony, Londiste, BucardoSpecific processing, batchesOff-line processingStill Transactional processingSkytools comes with PGQ
Dimitri [email protected] Implementing High Availability
![Page 45: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/45.jpg)
Agenda Isolate Services Durability Availability Conclusion
Application Split
When you have separate backoffice and production requirements
Cross replicationSlony, Londiste, BucardoSpecific processing, batchesOff-line processingStill Transactional processingSkytools comes with PGQ
Dimitri [email protected] Implementing High Availability
![Page 46: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/46.jpg)
Agenda Isolate Services Durability Availability Conclusion
Application Split
When you have separate backoffice and production requirements
Cross replicationSlony, Londiste, BucardoSpecific processing, batchesOff-line processingStill Transactional processingSkytools comes with PGQ
Dimitri [email protected] Implementing High Availability
![Page 47: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/47.jpg)
Agenda Isolate Services Durability Availability Conclusion
Application Split
When you have separate backoffice and production requirements
Cross replicationSlony, Londiste, BucardoSpecific processing, batchesOff-line processingStill Transactional processingSkytools comes with PGQ
Dimitri [email protected] Implementing High Availability
![Page 48: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/48.jpg)
Agenda Isolate Services Durability Availability Conclusion
Application Split
Implementing londiste and PGQ
Dimitri [email protected] Implementing High Availability
![Page 49: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/49.jpg)
Agenda Isolate Services Durability Availability Conclusion
Queueing with PGQ
Off-line processing is better done with PGQ
Mainly written in PLpgSQL (and C)Client API for pythonand PHP
some work is happening for JavaCooperative Worker (Skytools 3)
PGQ: Stable, Reliable, Easy to monitor
Dimitri [email protected] Implementing High Availability
![Page 50: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/50.jpg)
Agenda Isolate Services Durability Availability Conclusion
Queueing with PGQ
Off-line processing is better done with PGQ
Mainly written in PLpgSQL (and C)Client API for pythonand PHP
some work is happening for JavaCooperative Worker (Skytools 3)
PGQ: Stable, Reliable, Easy to monitor
Dimitri [email protected] Implementing High Availability
![Page 51: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/51.jpg)
Agenda Isolate Services Durability Availability Conclusion
Queueing with PGQ
Off-line processing is better done with PGQ
Mainly written in PLpgSQL (and C)Client API for pythonand PHP
some work is happening for JavaCooperative Worker (Skytools 3)
PGQ: Stable, Reliable, Easy to monitor
Dimitri [email protected] Implementing High Availability
![Page 52: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/52.jpg)
Agenda Isolate Services Durability Availability Conclusion
Queueing with PGQ
Off-line processing is better done with PGQ
Mainly written in PLpgSQL (and C)Client API for pythonand PHP
some work is happening for JavaCooperative Worker (Skytools 3)
PGQ: Stable, Reliable, Easy to monitor
Dimitri [email protected] Implementing High Availability
![Page 53: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/53.jpg)
Agenda Isolate Services Durability Availability Conclusion
Disaster Recovery and Service Continuity
PostgreSQL 9.1 offers Synchronous Replication and Hot Standby
Hot Standby(A)Synchronous ReplicationStandby connects with libpq
recovery.conf
Continue ArchivingSwitchabe per transactionGood performance level
Dimitri [email protected] Implementing High Availability
![Page 54: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/54.jpg)
Agenda Isolate Services Durability Availability Conclusion
Disaster Recovery and Service Continuity
PostgreSQL 9.1 offers Synchronous Replication and Hot Standby
Hot Standby(A)Synchronous ReplicationStandby connects with libpq
recovery.conf
Continue ArchivingSwitchabe per transactionGood performance level
Dimitri [email protected] Implementing High Availability
![Page 55: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/55.jpg)
Agenda Isolate Services Durability Availability Conclusion
Disaster Recovery and Service Continuity
PostgreSQL 9.1 offers Synchronous Replication and Hot Standby
Hot Standby(A)Synchronous ReplicationStandby connects with libpq
recovery.conf
Continue ArchivingSwitchabe per transactionGood performance level
Dimitri [email protected] Implementing High Availability
![Page 56: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/56.jpg)
Agenda Isolate Services Durability Availability Conclusion
Disaster Recovery and Service Continuity
PostgreSQL 9.1 offers Synchronous Replication and Hot Standby
Hot Standby(A)Synchronous ReplicationStandby connects with libpq
recovery.conf
Continue ArchivingSwitchabe per transactionGood performance level
Dimitri [email protected] Implementing High Availability
![Page 57: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/57.jpg)
Agenda Isolate Services Durability Availability Conclusion
Disaster Recovery and Service Continuity
Implementing Hot Standby
Dimitri [email protected] Implementing High Availability
![Page 58: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/58.jpg)
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
![Page 59: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/59.jpg)
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
PL/proxy
Scale-up or Scale-out?Remote Procedure CallShardingDistributed databaseAutonomous TransactionsStored Procedures
Dimitri [email protected] Implementing High Availability
![Page 60: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/60.jpg)
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
PL/proxy
Scale-up or Scale-out?Remote Procedure CallShardingDistributed databaseAutonomous TransactionsStored Procedures
Dimitri [email protected] Implementing High Availability
![Page 61: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/61.jpg)
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
PL/proxy
Scale-up or Scale-out?Remote Procedure CallShardingDistributed databaseAutonomous TransactionsStored Procedures
Dimitri [email protected] Implementing High Availability
![Page 62: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/62.jpg)
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
PL/proxy
Scale-up or Scale-out?Remote Procedure CallShardingDistributed databaseAutonomous TransactionsStored Procedures
Dimitri [email protected] Implementing High Availability
![Page 63: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/63.jpg)
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
PL/proxy
Scale-up or Scale-out?Remote Procedure CallShardingDistributed databaseAutonomous TransactionsStored Procedures
Dimitri [email protected] Implementing High Availability
![Page 64: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/64.jpg)
Agenda Isolate Services Durability Availability Conclusion
PL/Proxy
Installing plproxy
Example (install.sql)=# create extension plproxy;=# create extension hashlib;
Dimitri [email protected] Implementing High Availability
![Page 65: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/65.jpg)
Agenda Isolate Services Durability Availability Conclusion
PL/Proxy
pl/proxy is the integrated sharding layer. Now you have to write allyour SQL in server side functions.
Example (admin/change_group_status.sql)create or replace function admin.change_group_status(
user_name text, status integer)returns void as $BODY$
CLUSTER ’fl_cluster’;RUN ON hash_string(user_name, ’lookup3le’);
$BODY$;
Dimitri [email protected] Implementing High Availability
![Page 66: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/66.jpg)
Agenda Isolate Services Durability Availability Conclusion
Distributed Sequences 1/2
Same schema on every node, best to avoid overlapping sequences.
Example (distributing sequence)ALTER SEQUENCE foo_id_seq
INCREMENT BY 16;
Dimitri [email protected] Implementing High Availability
![Page 67: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/67.jpg)
Agenda Isolate Services Durability Availability Conclusion
Distributed Sequences 2/2
Same schema on every node, best to avoid overlapping sequences.
Example (distributing sequence)ALTER SEQUENCE foo_id_seq
MINVALUE 297000000000000000MAXVALUE 297999999999999999;
http://www.fotolog.com/<user>/297000000000017139/
Dimitri [email protected] Implementing High Availability
![Page 68: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/68.jpg)
Agenda Isolate Services Durability Availability Conclusion
Distributed Sequences 2/2
Same schema on every node, best to avoid overlapping sequences.
Example (distributing sequence)ALTER SEQUENCE foo_id_seq
MINVALUE 297000000000000000MAXVALUE 297999999999999999;
http://www.fotolog.com/<user>/297000000000017139/
Dimitri [email protected] Implementing High Availability
![Page 69: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/69.jpg)
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
Implementing plproxy
Dimitri [email protected] Implementing High Availability
![Page 70: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/70.jpg)
Agenda Isolate Services Durability Availability Conclusion
Scaling Writes
Implementing plproxy
Dimitri [email protected] Implementing High Availability
![Page 71: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/71.jpg)
Agenda Isolate Services Durability Availability Conclusion
1 AgendawhoamiAvailability, DuralibityArchitectures and Replications
2 Isolate ServicesTrafic growth
3 DurabilityData DurabilityData Availability
4 AvailabilityServices AvailabilitySharding
5 ConclusionPostgreSQL Replication: Looking back, looking forwardQuestions
Dimitri [email protected] Implementing High Availability
![Page 72: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/72.jpg)
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
![Page 73: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/73.jpg)
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
![Page 74: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/74.jpg)
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
![Page 75: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/75.jpg)
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
![Page 76: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/76.jpg)
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
![Page 77: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/77.jpg)
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
![Page 78: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/78.jpg)
Agenda Isolate Services Durability Availability Conclusion
Distributed High Availability
Retrospective and Future of PostgreSQL Replication
8.1, PITR8.2, Warm Standby8.3, pg_standby9.0, Hot Standby9.1, Synchronous Replication9.2, Cascading Replication9.3, Bi-Directional Replication
Dimitri [email protected] Implementing High Availability
![Page 79: Implementing High Availability - with PostgreSQL · Agenda Isolate Services Durability Availability Conclusion 1 Agenda whoami Availability,Duralibity ArchitecturesandReplications](https://reader035.fdocuments.net/reader035/viewer/2022081402/605203c66860e110654765e2/html5/thumbnails/79.jpg)
Agenda Isolate Services Durability Availability Conclusion
Questions?
Meet with us on the booth, join us in the Hallway Track!
Want to win a blue elephant?http://2012.pgconf.eu/feedback/
Dimitri [email protected] Implementing High Availability