MyHeritage - Test Automation in a Continuous Deployment Environment
Optimize your delivery and quality with the right release methodology and tools - Guy Tsype,...
-
Upload
droidcontlv -
Category
Technology
-
view
166 -
download
0
Transcript of Optimize your delivery and quality with the right release methodology and tools - Guy Tsype,...
Optimize delivery and quality with the right combination of tools
MyHeritage
6,600,000,000Historical Records
2,500,000,000Profiles
85,000,000Users
Who are we?
• Personal goals - We should all want to produce and release a high quality app
• Poor quality impacts your bottom line• User satisfaction leads to high retention rates• Brand reputation • Pride
Why Quality Matters?
• Keeping the foot on the gas paddle when it comes to releasing new features and versions while maintaining high quality.
• Quality isn’t determined just by crashes, need a way to determine that your users get the experience you want them to.
Challenges In a Fast Pace Environment
• High device fragmentation that leads to greater complexity
Challenges In a Fast Pace Environment
• The need to support a high number of OS versions
Challenges In a Fast Pace Environment
• Keeping track of your app versions and their contents over long periods of time.
• Preserving your API’s integrity over time.
Challenges In a Fast Pace Environment
• Visibility• Controlled releases• Regression Prevention
Keys to Success
Visibility
• Design and implement your analytics strategy in a way that provides you the most visibility and information.
• Set a unique user ID so that you can monitor the same user over multiple analytic platforms
Visibility
Visibility
• Monitoring crash reports• Monitoring version adoption
Controlled App Releases
• Try to stick to a pre-defined version release schedule• Periodic releases - New version every 2-3 weeks• Feature flags • Staged Rollout
Regression Prevention
• Beta/Alpha testing• Unit tests – Build machine • End to end Automations• API automations
Regression prevention – Nightly Builds
Nightly Unit Tests End to End UI Test
Nightly Coverage
How it all fits togetherVersion Release
New Feature Development
Testing
Alpha/Beta Phase
How it all fits together
• Writing unit tests • Writing E2E/Integration/Automation tests • Implement feature flags • Fixing the top crashes that were collected from previous version
New Feature Development Phase
How it all fits together
• Running Unit Test from nightly build machine• Running E2E/Integration from nightly build machine• Manual QA• Monitor test usage to see if the user experience is what was intended• Analyze user feedback
Testing and Beta analysis
How it all fits together
• Set feature flags to their production values • Final QA sanity check • Staged rollout over about 2 days – Bug fixes if needed
Version Release Phase
How it all fits together
• Crash report monitoring – Collect top crashes• Version adoption monitoring • Key app event monitoring (e.g. in-app purchases) • Monitoring new feature analytics for quality and usage insights • Monitoring Google play reviews and support tickets
Post Version Release Analysis
• Visibility• Controlled releases• Regression Prevention
Summary
Thank you