Post on 07-Jan-2017
Feature Flaggingto Reduce Risk inDatabase Migrations
By Patrick KaedingLead Engineer at LaunchDarkly@pkaeding @launchdarkly
About Me & LaunchDarkly•Current: Lead Engineer at LaunchDarkly• Past: Engineer at Atlassian• LaunchDarkly is a cloud-based feature flag management platform• Separate deploying code from releasing features•Manage risk of releasing features•Measure impact (both KPI and Ops performance)
@pkaeding @launchdarkly
Database Migrations• Sooner or later, it is inevitable• Schema changes•Moving to larger database cluster•Moving from one kind of DB to another
@pkaeding @launchdarkly
Maintenance window
Database Migrations (Bad Old Days)
Application Old Database
@pkaeding @launchdarkly
Database Migrations (Bad Old Days)
Old Database New Database
Migrate
Maintenance window (application offline)
@pkaeding @launchdarkly
Maintenance window
Database Migrations (Bad Old Days)
Application New Database
@pkaeding @launchdarkly
Maintenance Window Approach•Pros
SimplicityCan always be used (since both code paths don’t need to work at the same time)
•ConsDowntime If there is a problem, redeploy old code and restore from backup
All or nothing
@pkaeding @launchdarkly
Zero-Downtime Approach
Database Migrations (The Less Bad Days)
Application Old Database
Write
Read
@pkaeding @launchdarkly
Zero-Downtime Approach
Database Migrations (The Less Bad Days)
Application Old Database
Write
Read
MigrateWriteNew Database
@pkaeding @launchdarkly
Zero-Downtime Approach
Database Migrations (The Less Bad Days)
Application New Database
Write
Read
@pkaeding @launchdarkly
Zero-Downtime Approach•Pros
No downtime If there is a problem, you can keep running from the old database/schema
•ConsAll or nothingNeed to write a reverse migration if you want to roll back
@pkaeding @launchdarkly
The Feature-Flagged Approach
Database Migrations (Modern Times)
Application Old Database
Write
Read
@pkaeding @launchdarkly
Feature-Flagged Approach
Database Migrations (Modern Times)
Application Old Database
Write 100%
Read 100%
Write 10%
New Database
@pkaeding @launchdarkly
Feature-Flagged Approach
Database Migrations (Modern Times)
Application Old Database
Write 100%
Read 100%
Write 100%
New Database
@pkaeding @launchdarkly
Feature-Flagged Approach
Database Migrations (Modern Times)
Application Old Database
Write 100%
Read 100%
Write 100%
New Database
Read 50%
@pkaeding @launchdarkly
Feature-Flagged Approach
Database Migrations (Modern Times)
Application Old Database
Write 100%
Read 100%
Write 100%
New DatabaseRead 100%
Migrate
@pkaeding @launchdarkly
The Feature-Flagged Approach
Database Migrations (Modern Times)
Application New Database
Write
Read
@pkaeding @launchdarkly
Feature-Flagged Approach•Pros
No downtimeGradual rollout allows you to monitorYou can roll back just as easily (or partially roll back)Compare results with live data
•ConsLonger period for both databases to be operational
@pkaeding @launchdarkly
Demonstration
@pkaeding @launchdarkly
This demo is based on a true story…• LD Events & user data were stored in Mongo
Events drive A/B test results, feature status indicators, auto-complete widgets, and more
•Unsharded replica set couldn’t handle the load• Start sharding, or move on to Dynamo• Integrity checks found issues with null/empty values in our dynamo code
@pkaeding @launchdarkly
What about…• Schema changes? New fields? Restructured data?•Which users get the new DB?•Open source alternatives?
@pkaeding @launchdarkly
Questions• pkaeding@launchdarkly.com•@pkaeding•@launchdarkly• https://launchdarkly.com•Blog article: http://kaed.in/db-blog•Demo code: http://kaed.in/dbcode
@pkaeding @launchdarkly