The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing
26
W14 Test Techniques 5/4/16 15:00 The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing Presented by: Stephen Burlingame Navicure Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 8882688770 9042780524 [email protected] http://www.stareast.techwell.com/
-
Upload
josiah-renaudin -
Category
Software
-
view
44 -
download
0
Transcript of The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing
W14 Test Techniques 5/4/16 15:00
The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing
Presented by:
Stephen Burlingame
Navicure
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.stareast.techwell.com/
Stephen Burlingame Navicure Stephen Burlingame’s introduction to software quality assurance occurred during his university senior project where he discovered critical defects in software designed to diagnosis breast cancer in digitized mammograms. Since graduation, Stephen has worked exclusively in the quality assurance field. In 2006, he joined the medical claims clearinghouse Navicure, Inc., where he is director of quality assurance, responsible for overseeing the testing for an application that processes more than one million transactions daily. Living in the Atlanta metro area with his wife, Julia, Stephen has served as president of Atlanta SPIN (Software and Systems Process Improvement Network). Â
The Canary in the Coal Mine
Creating an Early Warning QA Tool to Help Your Testing
1
Slides written by
Stephen Burlingame
Director Of Quality Assurance at Navicure Inc., a medical claims clearinghouse offering additional billing and analysis tools for medical professionals.
Board member of Atlanta SPIN, a non-profit organization dedicated to promoting software process improvement
2
Who’s familiar with vCAC from
VMWare?
3
vCAC from VMWare Example Interface
4
Created CentOS Installation with Custom Software Deployed
Now our QA team can request environments on as
needed basis
Software build is deployed and configured automatically
Near “push button” operation
5
Yea! We’ve created virtual
test environments! Our work
is completed and now we
can all go home!
In reality the hard work is not
complete
You’ve solved some difficult
problems but possibly created
new challenges
6
Environment Chaos!
Old Setup
CERT IMP
New Setup
CERT IMP
Virtual 1
Virtual 2
Virtual N
Virtual 3
7
Managing and keeping
track of environments is
getting out of hand!
QA engineers spend more time
figuring out what systems they
have and the health those
systems than ever before.
More systems = more problems
8
Problem: How Can We Monitor Those
Systems and Not Spend More Time in
Maintenance than Testing?
9
Solution: Create a Daily Report
This report will highlight system health and basic
performance
This report will focus on displaying key performance
indicators in graphical form that is intuitive
This report will alert on known conditions that indicate
defects
10
“QA Health Checker”
11
“QA Health Checker”
Data Examples
Start Simple!
o Disk space alerts
o Indicate services are up or down
o Look for directories present or missing or when last updated
12
“QA Health Checker”
Data Examples
Get More Complicated!
o Write SQL queries that look for “bad” data or missing data
o Look for errors in logs or indicate that error logs have content
o Look for scenarios that indicate system is experiencing issues
13
“QA Health Checker”
Data Examples
Let’s Get Crazy!
o Create a uniform CSS
o Implement basic pictographic principles
o Fit this report into retrospective discussions so you improve the metrics captured
14
15
16
17
A More
Technical Look
Test systems run nightly cron jobs
to gather data and place it on
their local hard drive
Central reporting machine knows
which IP addresses to read from
and pulls the stats into a prepared
report
Report is emailed to a distribution
list early every weekday
18
A More
Technical Look
Used Python as the scripting
language
Created a deployment script and
documented the release process
Documented the report contents
and general architecture
Used Confluence as the
documentation portal
19
Changing the Conversation With
Employees
20
Old Conversation
QA Engineers didn’t always know what was going on or
how interpret data. They complained about “phantom
issues” and “non-reproducible bugs”. Used vague terms like
“slow”, “buggy”, or “weird”. Documentation was simply not
enough.
21
New Conversation
QA Engineers know right away the types of issues they are
having or about to have in the near future. Team leads are
keeping their focus on early signs of issues leading up to
release. Management is having pro-active conversation
with employees rather than a reactive dialog.
22
Closing Thought
Recommendation:
Take a slight risk with this report and experiment
Keep it adjusting and evolving!
23
Time For Questions
Contact Information:
Email Address – [email protected]
If in Atlanta, come to an Atlanta SPIN meeting!
24
