Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for...

30
SRECon EU, July 2016 Greg Burek Lessons from Automatic Incident Resolution for a Million Databases

Transcript of Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for...

Page 1: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

SRECon EU, July 2016

Greg Burek

Lessons from Automatic Incident Resolution for a Million Databases

Page 2: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka
Page 3: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

The Twelve-Factor App

Page 4: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka
Page 5: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka
Page 6: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

Department of Data

Page 7: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

PostgresqlRedisKafka

Page 8: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

~ Million Databases

Tens of thousands of AWS Instances

Page 9: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

Some Databases

Hundreds of AWS Instances

Page 10: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka
Page 11: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

“The goal is to build systems that can scale linearly with

machines & sub-linearly with people” - Caitie McCaffrey

Tackling Alert Fatigue

Page 12: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

Monitor and alert on your business

Page 13: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka
Page 14: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

Monitor and alert on your business

Usually, don’t alert on machine specific metrics

Page 15: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

Write runbooks and playbooks

Page 16: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

Turn playbooks into code

Page 17: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka
Page 18: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

“The goal is not to never get paged, the goal is to never get

paged for the same thing twice” - Astrid Atkinson

Engineering for the long game

Page 19: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

Verify monitoring before restarting the world

Page 20: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka
Page 21: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

Circuit breakers

Page 22: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

Automation can’t handle the unknown

Wake someone up on exceptions and timeouts

Page 23: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka
Page 24: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

Have a REPL/console

Page 25: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

Aggregate and review trends

Page 26: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka
Page 27: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

Humans can break

Automation can be simplistic

Humans + Automation for a resilient and operable system

Page 28: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

1. Monitor and alert on your business2. Write playbooks3. Make playbooks into automation4. Checks and balances of automation5. Circuit breakers6. Alert on exceptions and timeouts7. Admin console8. Aggregate and review trends

Page 29: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

[email protected]@gregburek

Page 30: Resolution for a Million Databases Lessons from …...Lessons from Automatic Incident Resolution for a Million Databases The Twelve-Factor App Department of Data Postgresql Redis Kafka

State Machines


Lessons and The Way Forward - Konsultasi Muamalat · INDONESIA’S BANKING CRISIS RESOLUTION Lessons and The Way Forward* Sukarela Batunanggar** 2 December 2002 (Preliminary) Abstract

Lessons and The Way Forward - Konsultasi Muamalat · INDONESIA’S BANKING CRISIS RESOLUTION Lessons and The Way Forward* Sukarela Batunanggar** 2 December 2002 (Preliminary) Abstract

ACEH CONFLICT RESOLUTION: LESSONS LEARNED AND THE ...

ACEH CONFLICT RESOLUTION: LESSONS LEARNED AND THE ...

Multimedia Databases Text I. Outline Spatial Databases Temporal Databases Spatio-temporal Databases Multimedia Databases Text databases Image and video.

Multimedia Databases Text I. Outline Spatial Databases Temporal Databases Spatio-temporal Databases Multimedia Databases Text databases Image and video.

Scaling and hardware provisioning for databases (lessons learned at wikipedia)

Scaling and hardware provisioning for databases (lessons learned at wikipedia)

Online Dispute Resolution (ODR) within Developing Nations€¦ · Identify lessons for the future ODR Forum hosts and the National Center for Technology and Dispute Resolution (NCTDR).

Online Dispute Resolution (ODR) within Developing Nations€¦ · Identify lessons for the future ODR Forum hosts and the National Center for Technology and Dispute Resolution (NCTDR).

Lessons from environmental peacebuilding programming · 2019-10-07 · 2 1. Summary This rapid literature review collates lessons from conflict resolution/peacebuilding programmes

Lessons from environmental peacebuilding programming · 2019-10-07 · 2 1. Summary This rapid literature review collates lessons from conflict resolution/peacebuilding programmes

Role of Deposit Insurance in Bank Resolution Framework – Lessons from the Financial Crisis Session III – Lessons from Financial Crisis for Bank Resolution:

Role of Deposit Insurance in Bank Resolution Framework – Lessons from the Financial Crisis Session III – Lessons from Financial Crisis for Bank Resolution:

Banking Crisis Resolution: Lessons Learnt

Banking Crisis Resolution: Lessons Learnt

Databases in LHC Experiments - Agenda (Indico)€¦ · Databases in LHC Experiments: Usage and Lessons Learned . A. Vaniachine ... production/processing and analysis chain of the

Databases in LHC Experiments - Agenda (Indico)€¦ · Databases in LHC Experiments: Usage and Lessons Learned . A. Vaniachine ... production/processing and analysis chain of the

Reversal and Reduction, Resolution and Reform€¦ · Reversal and Reduction, Resolution and Reform Lessons from the Financial Crisis in Europe and Central Asia to Improve Outcomes

Reversal and Reduction, Resolution and Reform€¦ · Reversal and Reduction, Resolution and Reform Lessons from the Financial Crisis in Europe and Central Asia to Improve Outcomes

Text Databases. Outline Spatial Databases Temporal Databases Spatio-temporal Databases Data Mining Multimedia Databases Text databases Image and video.

Text Databases. Outline Spatial Databases Temporal Databases Spatio-temporal Databases Data Mining Multimedia Databases Text databases Image and video.

Washington Times Conflict Resolution Lessons

Washington Times Conflict Resolution Lessons

Lessons from Finacial Crisis for Bank Resolution – Presentation on Europe and Germany Role of Deposit Insurance in Bank Resolution Framework – Lessons.

Lessons from Finacial Crisis for Bank Resolution – Presentation on Europe and Germany Role of Deposit Insurance in Bank Resolution Framework – Lessons.

Reversal and Reduction, Resolution and Reformdocuments.worldbank.org/curated/en/490141468257678018/... · 2016-09-20 · Reversal and Reduction, Resolution and Reform Lessons from

Reversal and Reduction, Resolution and Reformdocuments.worldbank.org/curated/en/490141468257678018/... · 2016-09-20 · Reversal and Reduction, Resolution and Reform Lessons from

Scalable Entity Resolution Using Probabilistic … Entity Resolution Using Probabilistic Signatures on Parallel Databases Yuhang Zhang AUSTRAC Kee Siong Ng AUSTRAC / ANU Michael Walker

Scalable Entity Resolution Using Probabilistic … Entity Resolution Using Probabilistic Signatures on Parallel Databases Yuhang Zhang AUSTRAC Kee Siong Ng AUSTRAC / ANU Michael Walker

LESSONS LEARNED - GilbaneLESSONS LEARNED One of our key differentiators in resolving conflict resolution is understanding our lessons learned from previous projects and sharing this

LESSONS LEARNED - GilbaneLESSONS LEARNED One of our key differentiators in resolving conflict resolution is understanding our lessons learned from previous projects and sharing this

Crisis Management and Resolution: Early Lessons from … ·  · 2011-03-10Early Lessons from the Financial Crisis Stijn Claessens, ... Crisis Management and Resolution: Early Lessons

Crisis Management and Resolution: Early Lessons from … ·  · 2011-03-10Early Lessons from the Financial Crisis Stijn Claessens, ... Crisis Management and Resolution: Early Lessons

Hawks and Doves: Peace Keeping and Conflict Resolution€¦ · Controlling Violence and Conflict Resolution Peacekeeping and Conflict Resolution Skills and Training 6. Lessons Learned

Hawks and Doves: Peace Keeping and Conflict Resolution€¦ · Controlling Violence and Conflict Resolution Peacekeeping and Conflict Resolution Skills and Training 6. Lessons Learned

Dispute Resolution: Lessons to be Learned from the Experience of Disputes in the Construction Industry

Dispute Resolution: Lessons to be Learned from the Experience of Disputes in the Construction Industry

PRACTICAL LESSONS FROM THE NPL RESOLUTION IN LATVIApubdocs.worldbank.org/en/596021527523057585/NPL... · PRACTICAL LESSONS FROM THE NPL RESOLUTION IN LATVIA Elmārs Zakulis Head of

PRACTICAL LESSONS FROM THE NPL RESOLUTION IN LATVIApubdocs.worldbank.org/en/596021527523057585/NPL... · PRACTICAL LESSONS FROM THE NPL RESOLUTION IN LATVIA Elmārs Zakulis Head of