December 6, 2014

Andrii Dzynia Road Manager to Continuous Delivery

@adzynia

Quality Built In

2

Spotify brings you the right music for

every moment!

Started in 2006 (in Sweden) Now 1500+ employees, 500+ engineers Over 30 million songs available Over 20,000 songs added every day 5 development centres across the globe

What is Quality?

3

4

Not just software defects, but stakeholders expectations as well

when expectations matches the reality.

Quality is a “state” ….

Why Quality is Important?

5

Cost of Bug/Change

6

What is a way to build high quality product?

7

8Do not release until it’s perfect

9

No one is perfect.

Implementing internal back office system

Example

11

Step 1 - Prototyping phase

Technical spike investigation Hacking different Back-End solutions

Ad-hoc design discussions Stub implementation of Front End

Getting early user’s feedback on Front End

12

WebApp with

stubs

Whatever works to get end-user understanding

Sketches on

whiteboard

13

Step 2 - Setting delivery pipeline

Add more unit tests Add more integration tests Add more functional tests Automate deployment configuration Add dashboards and system monitoring

Clarifying End User Needs

14

http://blog.crisp.se/2013/02/05/yassalsundman/continuous-delivery-vs-continuous-deployment

15

Step 3 - Connecting the dots

Connect back-end and front-end parts into

End-2-End system

Get user’s feedback and iterate

Keep releasing small, usable, incremental pieces16

17

What is under the hood ?

18

Organic Structure

19

Scaled Agile @ Spotify@joakimsunden

20

Test Automation Pyramid

21

How to avoid Ice-Cream effect?

22

Be strict with this rule.

Testable micro-services architecture 23

Unit Tests

Component Tests

Functional Tests

System/End-2-End Tests

Integration Tests

Unit Tests

Component Tests

Functional Tests

JS Unit Tests

JS Component Tests

UI Functional

Tests

24

Unit Tests

Check logic of minimal code snippet with no or mocked dependencies

https://github.com/mockito/mockitohttps://github.com/junit-team/junit

25

https://github.com/jsevellec/cassandra-unit

Very much like unit tests but emulating external components

e.g. cassandra database

Component Tests

26

Client App

Fake Back-End Server

Spotify Back End

Fake back-end for client apps

http://jsonstub.com

https://github.com/azagniotov/stubby4jhttps://github.com/dreamhead/moco

27

Integration Tests

Check an integration between components

You do not need whole system to check the integrationhttps://github.com/spotify/helios/blob/master/docs/testing_framework.md

28

@docker containers

Create Test Environment ‘on the fly’

https://www.docker.com

29

Check services API behaviour or

End user use cases

Functional Tests

http://seleniumhq.orghttp://hc.apache.org

30

http://graphwalker.org

Model Based Testing

32

Code Review &

Human Testing on

Pull Request

Delivery Pipeline33

34

Was not covered during this talk …

Load Testing Gradual Rollouts A/B Testing Feature Toggles Monitoring/Alerting Information Radiators

… but stay tuned

35

Technical Test Engineer Test Engineer

! = Manual Tester ! = Test Automator

~ Software Engineer in Test

Test Engineering Roles and Responsibilities

~ Context Driven Tester

36

Technical Test Engineer

Work as a software developer Advocate testability of the product

Argue on software design with software engineers Help with building new/injecting existed development tools

37

https://github.com/jsevellec/cassandra-unit

Cassandra Unithttps://github.com/mikaellanger/job-dsl-plugin

Jenkins job-dsl-plugin

https://github.com/spotify/heliosDocker orchestration

Spoticloude.g. cli control over amazon cloud http://dashing.io

Dashboards

Development Productivity Tools

38

Knows business domain Focused on exploring the product

Free to use any programming language to test specific use case Free to use any programming language to automate his work

Test Engineer

39

Mind Map as a Tool

https://www.mindmup.com

Product tree Scenarios Playbooks Checklists

Session notes

Quality Engineers40

http://www.satisfice.com/blog/archives/1372

TEST JUMPERS: ONE VISION OF AGILE TESTING

http://www.satisfice.com/blog/archives/1364“RESPONSIBLE TESTER”

http://www.satisfice.com/articles/omega_tester.pdfOMEGA TESTER

@jamesmarcusbach calls us Test Jumpers

41

Any other ways to improve

quality?

42

test ideas during healthy discussions

test requirements via end users collaboration

discuss system design and conner cases earlier in the planing/design meetings

define definition of done

Break Uncertainty

43

But have responsible person to figure out what consensus means in each case

Use Google Docs collaboration for finding group consensus

Social Programming

44

write “checks” during implementation shape your thoughts via pair discussions peer review before merging to master

45

User Stories Planning meeting Standup meeting Open workspace TDD Refactoring etc……

Not mentioned Engineering Practices

http://www.extremeprogramming.org/rules.html

Employee Early Builds Testing

46

47

One week to do whatever you want

Do it alone or with a team

Crazy ideas comes out

Hackathons

48

QA engineer at Spotify

http://continuousdelivery.com/2014/02/visualizations-of-continuous-delivery/

How to check you are on a right path ?

49

That is easy to spot 50

51

Ready for a journey?Make Quality explicit Find Quality promoters Define your way of Quality

improvements Set right Quality constraints Share results as early as possible

Keep looking further quality improvements Probably you will never end :)

52

@adzynia adzynia@spotify.com

Spotify Engineering Culture part 1 part 2

and …

Stop saying QA when you mean Testing

Check out http://www.spotify.com/jobs or @Spotifyjobs for more information

Want to join the band?