Testing metrics webinar
30
Joel Montvelisky Chief Solution Architect Simplify Test Management How to Avoid Cutting Yourself with the Double Edge Sword of Testing Metrics?
-
Upload
practitest -
Category
Software
-
view
100 -
download
0
Transcript of Testing metrics webinar
Joel Montvelisky
Chief Solution Architect
Simplify Test Management
How to Avoid Cutting Yourself
with the Double Edge Sword of
Testing Metrics?
What do we want to cover today?
• Pros & Cons of working with Metrics
• Planning your metrics program and
maintaining it over time
• Tips and tricks for working with metrics with
internal and external Stakeholders
• Questions and Answers
A little about me…
Professions :(last 19 years)
- Tester
- QA Manager
- Blogger
- Speaker
- Solution Architect
PractiTest:- Chief Solution Architect
- QA Manager
About
SaaS ALM Solution
Simple & Intuitive – yet – Methodological & Flexible
Integrates with Jira, Selenium, Jenkins, Pivotal Tracker, QTP,
Bugzilla, Ranorex, SoapUI, etc…
Best Darn Support in the Business
Show of hands
“Raise your hand”
if Your Team or
Your Company has
a Standard Set of
Metrics.
Show of hands No.2
“Raise your hand”
if these Metrics
have caused
more harm than help
sometime in the past…
Fact (?)
Metrics, just like knives
(or fire, or TNT, or salt ),
are a great tool that can also be
dangerous in the wrong hands
Pros & Cons of
working with Metrics
Pro 1: Metrics are an effective
means of communication
It is trivial to talk about metrics to communicate
status, but using metrics you can also convey to the
team what is important and where and effort is needed.
Pro 2: Metrics are (usually) clear
Good metrics are easy to understand both for the people inside
your team and outside of it
Pro 3: Metrics allow us to compare things
You can benchmark and compare:
- Different teams / process
- Same process in different times / version
etc.
Pro 4: Metrics help us track progress
You can see how things and change over time, make
predictions, corrections, etc.
Pro 5: Metrics are easy to distribute
It is easy to create a short set of metrics and send them by mail,
publish them in a portal or post them on a Kitchen Monitor.
Pro 6: People like metrics
People & even managers tend to open metrics and
review them because they are compact.
There are many
additional Pros…
Let’s talk a little about the
Cons of working with Metrics!
1. People don’t like being measured(Many times this is for good reasons)
2. Metrics tend to present only one side
of the story
3. Metrics are going to be used by
managers who…
Think their personal experience is valid
in all situations.
Jump to conclusions without waiting to
listen to all the information.
Forget problems in paper are easier to
solve than those in the real world.
Like applying quick fixes to complex issues.
Believe what they see in the metrics can tell them all
the information they need to know.
Planning and Maintaining your
Metrics Program
The Laws of (Conservation of) Metrics(Apologies for being such a geek…)
1. Anything that is measured will improve over time.
2. Whenever something improves, it will be at the expense
of something else that will worsen accordingly.
3. You can only measure 5 (sometimes only 3) aspects of
anything. Trying to measure more will lead to confusion
and failure.
4. Metrics will become ineffective after a number of
iterations (kind of like the pesticide paradox).
Planning your Metrics Program
Define what are the objectives of your metrics?
Who is your audience?
Your management?
Your team?
Your end users?
Regulation?
HR?
Each of them need different information to make
different decisions.
Planning your Metrics Program
Make sure you have the right metrics.
5 Why’s - to make sure you reached the core metrics
SMART
Simple
Measurable
Actionable
Repetitive
Timely
Planning your Metrics Program
Some of your most important metrics may be be
available ONLY after your project is done…
Escaping Defects
Rejected Defects
Server Exception Levels
Support calls post Release
etc.
1. Metrics need to be constantly reviewed for:
- Accuracy
- Relevancy
2. As your project, technology and people change so will
your metrics.
Maintaining your Metrics Program
3. You need to change your metrics once in a while, bugs
and people learn to work around them.
4. Be ready to change metrics, even
half way during your project
Maintaining your Metrics Program
Tips and Tricks
for working with Metrics
Presentation is as important as content
Have diff metrics for diff people
Tips & Tricks
Vs.
Capture more data than you need today,
you may need it later.
Provide analysis of your results with every graph
(don’t trust the people to understand what they see).
Tips & Tricks
Make sure to provide
multiple angles.
Too many metrics confuse
tell a story, but keep it
interesting.
Tips & Tricks
Push metrics, don’t make people ask for them since it
won’t be relevant by the time you provide them.
Tips & Tricks
Questions & Answers
Developer Testing
Do’s and Don’ts
About
SaaS ALM Solution
Simple & Intuitive – yet – Methodological & Flexible
Integrates with Jira, Selenium, Jenkins, Redmine, Bugzilla, QTP,
Ranorex, SoapUI, etc…
Best Darn Support in the Business
