Post on 11-May-2015
description
T21
Test Automation
5/8/2014 3:00:00 PM
Using DevOps to Improve
Software Quality in the Cloud
Presented by:
Glenn Buckholz
Coveros
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
Glenn Buckholz
Coveros
Currently, Glenn Buckholz leads CI and deployment automations efforts at Coveros. Using his fifteen years of industry experience, Glenn brings success to his customers. Beginning his career as a consultant implementing automated test frameworks, he introduced the concept of change management to many, many projects. Glenn then decided to become a part of honest society and settled down at the Public Company Accounting Oversight Board as their full time enterprise change manager. After several years, he ventured back into the real world at Coveros, specializing in implementing agile practices, implementing CI, and engineering configuration management instead of simply documenting it.
4/26/2014
1
© Copyright 2014 Coveros, Inc. All rights reserved.
Using DevOps to Improve Software
Quality in the Cloud
STAR EAST 2014 – 08 May 2014
Glenn Buckholz
glenn.buckholz@coveros.com
© Copyright 2014 Coveros, Inc. All rights reserved.
Agenda
�What did I do?
�What is it good for?
�Demo
�Architecture
�Workflow
�Orchestration
�Cost vs Velocity
�Data Analysis
�What is Coveros doing with it for customers?
4/26/2014
2
© Copyright 2014 Coveros, Inc. All rights reserved.
What did I do?
Combined several technologies that Coveros is
familiar with plus Puppet to demonstrate a full
implementation of CI (Continuous Integration)
This could be titled “Everything I wanted to do on
Forge.mil but am not allowed.”
Specifically, I cobbled together:
Jenkins
Puppet
Puppet master
EC2
Linux
CURL
ANT
© Copyright 2014 Coveros, Inc. All rights reserved.
What is it good for?
�Fail Faster
Used with permission from the people at Extra Credits.
http://www.youtube.com/extracredits
4/26/2014
3
© Copyright 2014 Coveros, Inc. All rights reserved.
Demo
© Copyright 2014 Coveros, Inc. All rights reserved.
Major Points
How is this DevOps?
Releases are engineered, the systems engineer
is simply pushing the button.
If the list of manual installation instructions is
more than one paragraph there is more
engineering work to do. Code is the
documentation.
Operations people now govern the access,
developers govern the how.
New testing concerns.
Deployment code.
OS templates.
Developers now have the job of convincing
operations to run their deployment code.
4/26/2014
4
© Copyright 2014 Coveros, Inc. All rights reserved.
Architecture
© Copyright 2014 Coveros, Inc. All rights reserved.
Workflow
Developer commits code.
Jenkins detects code change.
Proper RPMs are built and distributed to
repositories.
Jenkins contacts EC2 and spins up blank
instances.
Puppet is delivered to the blank instances.
Jenkins works with the Puppet Master for
orchestration.
Jenkins confirms that the application is
running.
Jenkins fires off automated tests.
4/26/2014
5
© Copyright 2014 Coveros, Inc. All rights reserved.
Code Commit – Different
ApproachesSomething has to trigger the testing. A code
commit is typically a significant enough
change to require some measure of retesting.
Unit test are the minimum criteria to the
publishing a change.
Even if the code is bad the system must
still run.
By convention, code commits should be
complete and significant.
Each commit now carries a cost.
This should be enforced by training.
For larger projects with large volumes of
© Copyright 2014 Coveros, Inc. All rights reserved.
RPMs A Packaging
To move quickly code must be delivered in a
consistent format.
Copying files is too unpredictable and error-
prone.
Permissions
Location
Clean up of old file
Dependency checking
Any of the following are good choices
RPM – Centos, RHEL
Emerge/Portage - Gentoo
Deb – Debian, Ubuntu
4/26/2014
6
© Copyright 2014 Coveros, Inc. All rights reserved.
Cloud Provider
The main issue is elasticity.
Number of servers will vary.
Tests will be running in parallel.
Need a way to catalog all the machines
Assign metadata to VMs
Ability to group VMs and test results
by deployment
Automatically mapping code commits
to cloud deployments is necessary for
proper bookkeeping.
© Copyright 2014 Coveros, Inc. All rights reserved.
OS templates
Each OS must originate with a standard template.
All change must be managed by Puppet or some
other deployment language
If the OS is not standard too much time will be spent
engineering around OS peculiarities
CM must be mastered
Every change to a standard OS must be recorded.
Every change must be programmed, no hand-
jamming
All deployments are now hands-off
You can almost get Continuous Delivery for free.
OS Customizations for only two reasons
Integrate into the Jenkins/Puppet
Integrate into the cloud provider from a standard
4/26/2014
7
© Copyright 2014 Coveros, Inc. All rights reserved.
OS Template Testing
Must Be Automated
RHEL, CENTOS, Windows change every
month, Manual checks must be limited to
keep up.
Security
STIG - Manual
OpenSCAP – STIG Automated
OWASP top 10 - Automated
Organizational Standards
Centralized Login
Standard directories
etc
© Copyright 2014 Coveros, Inc. All rights reserved.
Deployment Testing
This is a new area since deployments are now
code.
No traditional tools to test this type of
functionality
Currently methods are mostly manual.
Small automated tests must be constructed
to help with orchestration.
It either worked or it didn't
Manually sifting through logs to
troubleshoot.
Who does this?
Unfortunately, either a sysadmin or a
4/26/2014
8
© Copyright 2014 Coveros, Inc. All rights reserved.
Automated Testing
Automated testing must be an established
practice.
You cannot perform the volume of tests
required for CI efficiently without some sort
of automated check to separate the good
changes from the bad changes.
The cost and practice must be understood
within the development and testing
organization.
Testing resources must be able to handle max
capacity.
Tests must be able to run in parallel against
s
© Copyright 2014 Coveros, Inc. All rights reserved.
Role of Manual Testing
Manual Functional Testing DOES NOT GO AWAY!
Automated tests are not as effective as a person
because they are so specific.
Manual Testing is now a precious resource that
can only be applied in a limited fashion.
Time is what limits Manual testing
1-2 weeks of testing while code is written.
2-3 weeks of total testing effort.
Testing effort is divided.
Automation of new functionality testing.
Spot manual testing.
Manual free form regression.
Targeted regression based on automated results.
4/26/2014
9
© Copyright 2014 Coveros, Inc. All rights reserved.
Disposition of the Systems
Test reporting and the cloud provider must be
linked through a dashboard.
Manual testers must be able to verify failures.
Testers must have access to the broken
running system
Access will allow reproducibility and
verification
Developers must be able to identify broken
VMs
Access to the app to see the unexpected
behavior
© Copyright 2014 Coveros, Inc. All rights reserved.
Orchestration
This is the combination of architectural
components and services in a
meaningful fashion to produce a running
system.
For this example:
Database
App server
Database initialization
App server initialization
Coordinated starting of all the services
4/26/2014
10
© Copyright 2014 Coveros, Inc. All rights reserved.
Costs vs Velocity
Velocity comes with parallelism.
Tests running in parallel
Code being tested in a pipelined fashion.
Cost
Number of testing machines and infrastructure.
Number of parallel instances x machines per
instances
Storage – keeping failed instances around
Budget and Team size
Even with infinite money if you have a team of 5, the
400th VM is likely only giving very little marginal utility.
When money is the constraint, developer and tester
habits must best utilize the limited VMs wisely.
© Copyright 2014 Coveros, Inc. All rights reserved.
Data Analysis
In order to move quickly the right people must get
the right data.
Test Results
Quick access to the failed test
Quick access a screen shot of the failure
Ability to assign errors to the proper developers
Integrated bug tracking.
Server Analysis
Developers need to zero in on failed servers
quickly
Developers need to link failures to their code
quickly
Developers need to separate system/OS failures
4/26/2014
11
© Copyright 2014 Coveros, Inc. All rights reserved.
Dashboard (Data Analysis
Visualization)Centralization of results.
Sonar Cube
Custom pages
Views for Management, Testers, and
Developers.
Traceability
Features all have GUIDs
GUIDs linked to code commits
Unit tests linked to GUIDs
Automated tests are linked to feature
GUIDs
VMs linked to list of new GUIDs
© Copyright 2014 Coveros, Inc. All rights reserved.
What is Coveros Doing with it for
Customers?Training and Process improvement
Being the exemplar on a project to
move this practice to other projects.
Showing the change in the cost to test
a feature within an organization
Implementing pieces of the CI to show
marginal improvements.
Using the gains marginal gains to sell
the whole process.
4/26/2014
12
© Copyright 2014 Coveros, Inc. All rights reserved.
Level of Effort
The demonstration needs to be customized
for each environment.
Very few projects that I have dealt with are
“the same”. While this demo can be a guide,
I don't believe it can be generalized to a
solution, each environment must be
customized.
This implementation required research into
the APIs and inner workings of several
complicated software suites and services.
The engineering effort is commensurate with
the complexity of the architecture.
© Copyright 2014 Coveros, Inc. All rights reserved.
Why Link All These Technologies
Together?The industry is moving this way in various forms.
AWS templates
Service Mesh
JuJu
Hadoop
Flexiant
Even though the products are varied, they all share
general themes.
Inventory
Relational Mapping
OS Templates
Virtual Network Capabilities
If you know one you can have a feel for them all.