Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross...
Transcript of Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross...
![Page 1: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/1.jpg)
21/05/15
1
Continuous Delivery &
The Changing Role of the Tester
Dr Kelvin Ross ANZTB 2015, Auckland 22 May 2015
2
Disruptive Technology
![Page 2: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/2.jpg)
21/05/15
2
3
Disruptive Technology
4
Disruptive Technology
![Page 3: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/3.jpg)
21/05/15
3
5
Disruptive Technology
CLOUD SOCIAL
BIG DATA MOBILE
DISRUPTIVE FORCES
![Page 4: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/4.jpg)
21/05/15
4
Biggest Disruptors The Last 5 Years? Agile
Offshoring Virtualisation
8
Disruptive Forces
Customer Testing
DevOps
Big Data
![Page 5: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/5.jpg)
21/05/15
5
10
Startup Corporate
“You can't just ask customers what they want and then try to give that to them. By the time you get it built, they'll want something new.”
Steve Jobs
![Page 6: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/6.jpg)
21/05/15
6
12
14
Scientific Method for Customer Experiments
to Guide Software Development
![Page 7: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/7.jpg)
21/05/15
7
15
FAIL FAST
16
![Page 8: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/8.jpg)
21/05/15
8
17
3 – 6 Months
18
1 – 2 weeks
![Page 9: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/9.jpg)
21/05/15
9
19
“You can't just ask customers what they want and then try to give that to them. By the time you get it built, they'll want something new.”
Steve Jobs
SPEED is the new black
20
![Page 10: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/10.jpg)
21/05/15
10
21
DEV OPS
![Page 11: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/11.jpg)
21/05/15
11
24
No Silos
25
CI CD CD
Continuous Integration
Continuous Deployment
Continuous Delivery
![Page 12: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/12.jpg)
21/05/15
12
Source: Principles and Practices in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks
Rapid Releases
Source: http://readwrite.com/2014/01/01/three-reasons-your-startup-needs-devops-or-else
![Page 13: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/13.jpg)
21/05/15
13
CD Platform
Source Control
Build System
Pipeline Management
& Visualization
Artifact/Package
Repository
Automated Configuration Management
(IaC)
Test infrastructure
Automated Deployment
System
Source: Continuous Delivery at Scale John Esser, Director Engineering Productivity, Ancestry.com ALM Forum Seattle, 2014
Git Flow
Version & Configuration Management
![Page 14: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/14.jpg)
21/05/15
14
30
Infrastructure
Application Code
Static Data
Test Data Version Everything
31
Data Versioning
Infrastructure/Platform Versioning
![Page 15: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/15.jpg)
21/05/15
15
32
Cloud Computing
New Strategies • Duplicate test
environments
• Copy of production
• Grid based test automation
• Blue/Green Deployments
• Resilience testing
33
![Page 16: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/16.jpg)
21/05/15
16
34
Elastic Computing
Time
Which is better?
Source: Principles and Practices in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks
Release Pipeline
![Page 17: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/17.jpg)
21/05/15
17
36
UNIT
INTEGRATION
SYSTEM
• Fast • Isolated • Aligned with Code
• Slower • UI focused • Later in the processes
• Focused on linkages • Harnesses, Mocks and
Stubs
Testing Pyramid – Right Way
Source: Steve Neely, [email protected] @stenvoon http://bit.ly/cdpaper ALM Forum Seattle, 2014
Release Pipeline
![Page 18: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/18.jpg)
21/05/15
18
Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks
Deployment Gating
Source: Principles and Practices in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks
Production Testing
![Page 19: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/19.jpg)
21/05/15
19
40
Source: Principles and Practices in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks
Failure Monitoring
41
FAIL FAST
![Page 20: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/20.jpg)
21/05/15
20
42
FAIL FAST MTBF ✗
43
FAIL FAST MTTR ✓
![Page 21: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/21.jpg)
21/05/15
21
45
Customer Testing
Secured $80m in VC funding
Testing In The Wild Applause
![Page 22: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/22.jpg)
21/05/15
22
47
Elastic Resourcing
Time
Which is better?
48
Customer Testing
![Page 23: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/23.jpg)
21/05/15
23
49
A / B Tests
Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks E.g.“Google Analytics Content Experiments”
50
Feature Toggles
![Page 24: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/24.jpg)
21/05/15
24
51
Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks
Feature Toggles
52
Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks
Feature Toggles Progressive Delivery
To Minimise Risk
![Page 25: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/25.jpg)
21/05/15
25
53
Can We Skip Some Pre-Prod
Testing?
Early Customer Testing
55
BIG DATA Volume Velocity Variety
![Page 26: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/26.jpg)
21/05/15
26
56
How To Test Prediction
57
Big Data Challenges Staging test data sets
Test data privacy Variation of test data sets Production data variability
Controllability & Observability
Algorithm Evaluation Heuristic Testing
Exploratory Testing
A/B Testing Customer Sampling &
Monitoring
![Page 27: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/27.jpg)
21/05/15
27
New QA Skills
CI / CD Process
Data Analytics
Test Environment Provisioning
Production Monitoring Feature Architecture & Deployment Planning
Technical Influence
Automation
Experimentation
Canary Releasing
![Page 28: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/28.jpg)
21/05/15
28
Blue Green Deployments
Feature Toggles
![Page 29: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/29.jpg)
21/05/15
29
Feature Flippers
64
A / B Tests
Source: Advanced Topics in Continuous Deployment Mike Brittain,Engineering Director, Etsy @mikebrittain mikebrittain.com/talks E.g.“Google Analytics Content Experiments”
![Page 30: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/30.jpg)
21/05/15
30
Dark Launch
![Page 31: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/31.jpg)
21/05/15
31
68 In 5 Years???
![Page 32: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/32.jpg)
21/05/15
32
69
Skill Changes
70
DevOps Teams Embedded, Self Organising, Functional
Business Tester Automated Tester DevOps Tester
Non-Technical Functional Tester
![Page 33: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/33.jpg)
21/05/15
33
71
Reduced Test Management
Coordinate Resourcing Practice Support
Consultants Coaches Governance
Test Managers Test Leads
72
• Crowd Test Coordination • Customer Test Coordination • Feature Deployment Planning • User Experience and Failure
Monitoring
![Page 34: Continuous Delivery - ANZTB · Continuous Delivery & The Changing Role of the Tester Dr Kelvin Ross ANZTB 2015, Auckland ... • Grid based test automation • Blue/Green Deployments](https://reader035.fdocuments.net/reader035/viewer/2022070906/5f7ab1021185f374df0c835d/html5/thumbnails/34.jpg)
21/05/15
34
73
Q&A