Testistanbul 2016 - Keynote: "Why Automated Verification Matters" by Kristian Karl
-
Upload
turkish-testing-board -
Category
Technology
-
view
114 -
download
1
Transcript of Testistanbul 2016 - Keynote: "Why Automated Verification Matters" by Kristian Karl
1983 -Started programming1994 -Testing and automation1998 -Consultant2004 -Discovered Model-Based
Testing2010 -Test Manager at Spotify
● What can we check?● What is important to check?● How can we check it?● How to involve everyone?● How do we get ROI?
● Dev environments● Builds on pull requests and merge● Static code analysis● Code style checking● Dynamic code analysis● Verification● Archiving artefacts● Deployment
What can be automated?
BusinessLogic Coverage
ExecutionTime / Costs
CodeCoverage
TestType
UI
Integration
Unit Test
High
Medium
Low
Low
Medium
High
Long / High
Medium
Short /Low
Check forthe
expected
Findthe
unexpected
Business
Technology
● Functional testing● BDD● Specification by
example
● Unit tests● Component tests
● Exploratory testing● Scenarios● Usability testing
● Performance● Security
Our approach at SpotifyDisclaimer: There are 100+teams at Spotify, and theyall do things a bit different.Some teams do itvery differently...
Code change
Pre-merge build and
tests
Code and quality review
Risk analysis
Exploratory testing
Continuous Delivery
Pre-merge● Quick feedback loop● The total time for the build needs to
be time boxed● Style checking● Static code analysis● Unit tests● Stable functional tests● Stable integration tests● Packaging
Post-merge● All the steps from the pre-review
build● Longer running functional and
integration tests● System tests● Non-functional tests, like:
- resource leaks- performance
The Test The TestData Service
Some Production Service
Request:● Premium user● Not connected to Facebook● Using feature flag X
Query for user(s) matchingThe criterias
List a matching user(s)
Return selected userPick one user, and lock it,Prevent any other test from using that user.
Tim
e
App not running
Authentication form displayed
Home page
Start app
Close
Exit
Valid credentials
Bad credentials
Logout
Backend teams
● Backend teams work with smallish micro services.
● High degree of automation on all parts of the pipeline.
● No one but the team touches the team’s code.● They have operational responsibility.● Few defects found after definition of done.
Frontend [mobile & desktop] teams
● Building monoliths● 200+ people involved● Low degree of automated verification● Low testability● High number of defects are
found after definition of done.That hurts productivity...
Tragedy of the commonsIndividuals acting independently and rationally according to each other's self-interest, behave contrary to the best interests of the whole group
What can you do to improve?
● Ways of working are more important than tools and frameworks.
● Zero bug policy, instead of huge backlog of bugs.● Pairing and embed● Everyone writes and maintain
automated verification.● Testability will require
non-functionality.● Failed verifications must be
fixed ASAP.
Automated verification matters, because it can
● Increase your productivity● Give you confidence to deliver anytime● Enable you work with
defect prevention, rather than defect detection
References
● https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/● https://labs.spotify.com/2014/09/20/spotify-engineering-culture-part-2/● https://dl.dropboxusercontent.com/u/1018963/Articles/SpotifyScaling.pdf● https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-
automation-pyramid● http://testautomationpatterns.wikispaces.com/● http://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/● https://www.youtube.com/watch?v=IGQmdoK_ZfY● http://graphwalker.org/