Reaching for Your Quality Stretch Goals: Testing at Realtor.com
-
Upload
klaus-salchner -
Category
Technology
-
view
160 -
download
0
Transcript of Reaching for Your Quality Stretch Goals: Testing at Realtor.com
![Page 1: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/1.jpg)
© 2015 Move, Inc. All rights reserved. Do not copy or distribute.
Reaching your Quality Stretch@ realtor.comKlaus SalchnerVP Engineeringhttps://www.linkedin.com/in/ksalchner@ksalchner
April/5, 2017
![Page 2: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/2.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 2
Agenda
• A/B Testing to measure customer impact – 5-6 min
• Group activity – 5-6 min
• Where in the stack to invest in test automation – 5-6 min
• Group activity – 5-6 min
• Testing for reliability, resilience und recovery – 5-6 min
• Group activity – 5-6 min
• Q&A – 5-6 min
![Page 3: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/3.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 3
A/B Testing to measure customer impact
• End-to-end quality goes beyond shipping software
• It is about outcome, value for the customer
• How do you measure whether you solved a real customer need?
![Page 4: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/4.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 4
A/B Testing to measure customer impact
• Start at the story level, instead of WHAT you build you talk about
what customer need you solve and WHY that matters
• Come up with a Hypothesis and what you will measure to make it
quantative, meaningful?
E-commerce industry is a great example
![Page 5: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/5.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 5
A/B Testing to measure customer impact
By VWO Blog
![Page 6: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/6.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 6
A/B Testing to measure customer impact
Optimizey Blog
![Page 7: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/7.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 7
A/B Testing to measure customer impact
By VWO Blog
![Page 8: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/8.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 8
A/B Testing to measure customer impact
By VWO Blog
![Page 9: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/9.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 9
A/B Testing to measure customer impact
By VWO Blog
![Page 10: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/10.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 10
Some of the A/B Testing tools
• Adobe Test & Target
• Optimizely
• Maxymizely
• and more https://conversionsciences.com/blog/ab-testing-tools
• or home-grown
![Page 11: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/11.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 11
The 8 rules of A/B Testing by Search Engine Watch
• Hypothesis
• One Variable at a time
• Clear and aligned success criteria
• Volume and statistical significance (also risk)https://www.optimizely.com/optimization-glossary/statistical-significance
• Test Groups and Splits (can be 50/50, 10/90, etc.)
• Randomization
• Always be testing, with common sense
• Documentation
![Page 12: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/12.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute.
Q&Ahttp://www.topproducer.com/careers
12
![Page 13: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/13.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 13
Agenda
• A/B Testing to measure customer impact – 5-6 min
• Group activity – 5-6 min
• Where in the stack to invest in test automation – 5-6 min
• Group activity – 5-6 min
• Testing for reliability, resilience und recovery – 5-6 min
• Group activity – 5-6 min
• Q&A – 5-6 min
![Page 14: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/14.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 14
Where in the stack to invest in test automation
• Always go to the lowest levels
• Always write small, reusable, fast tests
![Page 15: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/15.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 15
Where in the stack to invest in test automation
Experiencebehavior
Functionalbehavior
Databehavior
![Page 16: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/16.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 16
Where in the stack to invest in test automation
Services
![Page 17: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/17.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 17
Where in the stack to invest in test automation
![Page 18: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/18.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 18
Test automation best practices
• Test Early and Test often
Have strong acceptance criteria – is an outline for your testing
Write your unit testing as components are created and iterate – full
or lightweight TDD
Integrate the tests early on into your CI/CD so they run all the time
Run them all the time!
![Page 19: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/19.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 19
Test automation best practices
• Create good quality test data
Managing test data, especially at scale, is a hard problem to
manage
Many times there is a pre-set environment with pre-set data which
lives for long-times and as time passes gets messy and in the way
for teams to be efficient
![Page 20: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/20.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 20
Test automation best practices
• Create good quality test data
The nirvana is that test data gets generated as tests run, and
cleaned up afterwards
And that you have sets of test data you can load into an
environment as it gets stood up
And that you can spin up and tear down environments as needed
– really hard to achieve
![Page 21: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/21.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 21
Test automation best practices
• Create UI tests which are resistant to change
Use names and class IDs to identify objects in the UI
Never use text labels, they always change
Never use positions, they always change
![Page 22: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/22.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 22
Test automation best practices
• Decide what test cases to automate, like
Frequently used (key) features
Tests which tend to cause human errors
Repetitive tests run with each CI/CD build / deployment
Tests you need to run on multiple platforms / browsers
Tests which are very time consuming
![Page 23: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/23.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 23
Test automation best practices
• Review your test cases with PO / PM and Engineers
Get many eyes on it and understand the main flows and edge
cases
Iterate on the testing like you iterate on the development
Get it out to customers for validation and use the feedback
![Page 24: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/24.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 24
Test automation best practices
• BDD – Behavior Driven Dev
Test the behavior, business use case
• TDD – Test Driven Dev
Be pragmatic, define the test for a component and then write the
code, not the whole thing
https://codeutopia.net/blog/2015/03/01/unit-testing-tdd-and-bdd
![Page 25: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/25.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 25
Test automation best practices
• What are other doing or not doing?
![Page 26: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/26.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute.
Q&Ahttp://www.topproducer.com/careers
26
![Page 27: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/27.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 27
Agenda
• A/B Testing to measure customer impact – 5-6 min
• Group activity – 5-6 min
• Where in the stack to invest in test automation – 5-6 min
• Group activity – 5-6 min
• Testing for reliability, resilience und recovery – 5-6 min
• Group activity – 5-6 min
• Q&A – 5-6 min
![Page 28: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/28.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 28
Testing for reliability, resilience and recovery
• Reliability is a concern owned by your Services & APIs
Clustering, fail-over
Elastic scaling (up and down)
Error handling
More
![Page 29: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/29.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 29
Basic steps for reliability testing…
• Set up a test environment where ideally all services are set up for
elastic scaling (cloud)
• Run a set of automated tests to establish a baseline (can be really
any automated tests like Selenium or JMeter)
• Scale up and down the load and observe how the various services
will start to scale up and down
• It will also tell you which services are the most stressed parts of
your application
![Page 30: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/30.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 30
Basic steps for reliability testing – expected results
• You want to see that your various services scale up – if they don’t
then they will break (unless they are super-fast/efficient)
• You want to see that the services scale down again – if not then
your costs will go up and up
• During the scale up and down you want to see that the functionality
doesn’t break (validates things like state-less)
![Page 31: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/31.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 31
More reliability testing…
• You can also set the maximum for the scaling up and down and
observe how the system behaves as it reaches the maximum load
• It will point out the weaknesses in your architecture and where to
optimize
![Page 32: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/32.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 32
Testing for reliability, resilience and recovery
• Resilience is a concern owned by the consumer / client
Error handling
Graceful degradation of features, so if one or more services are
down the user can still use your application:
For example: If the Ratings & Review service is down I can still
find and view products, or purchase products, without delays
![Page 33: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/33.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 33
Testing for reliability, resilience and recovery
By Nextflix Blog
![Page 34: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/34.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 34
Testing for reliability, resilience and recovery
By Nextflix Blog
![Page 35: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/35.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 35
Testing for reliability, resilience and recovery
By Nextflix Blog
![Page 36: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/36.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 36
Testing for reliability, resilience and recovery
• The “golden standard” is Hystrix, former circuit-breaker, by Netflix
https://github.com/Netflix/Hystrix/wiki
• Calls to services are wrapped in commands, and you can define
thresholds to these commands
![Page 37: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/37.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 37
Testing for reliability, resilience and recovery
By Nextflix Blog
![Page 38: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/38.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 38
Basics of resilience testing…
• Run automated tests, like Selenium or JMeter, and while these run
start shutting off non-essential services
• Then observe that your experience still functions, fast and reliable
• And observe that some of the features depending on these services
degrade – can be either not showing up or showing relevant
messages to users
![Page 39: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/39.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 39
Testing for reliability, resilience and recovery
• Recovery testing is the exercise of what it takes to recover from a
major fault or an outage
Troubleshooting – like Logs, instrumentation, etc.
Restoring services
Worst case, restoring data
Root causing is part of it
![Page 40: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/40.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 40
Basics of recovery testing…
• Run automated tests, like Selenium or JMeter, and while these run
start shutting off essential services
• Then observe that your experience stops functioning or slows down
for users
• Then run through the manual and automated steps to recover,
without making the situation worse for users
![Page 41: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/41.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute.
Q&Ahttp://www.topproducer.com/careers
41
![Page 42: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/42.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 42
Agenda
• A/B Testing to measure customer impact – 5-6 min
• Group activity – 5-6 min
• Where in the stack to invest in test automation – 5-6 min
• Group activity – 5-6 min
• Testing for reliability, resilience und recovery – 5-6 min
• Group activity – 5-6 min
• Q&A – 5-6 min
![Page 43: Reaching for Your Quality Stretch Goals: Testing at Realtor.com](https://reader035.fdocuments.net/reader035/viewer/2022070518/58ea89c11a28ab983e8b4fd3/html5/thumbnails/43.jpg)
© 2016 Move, Inc. All rights reserved. Do not copy or distribute.
Q&Ahttp://www.topproducer.com/careers
43