Enabling Agile Testing Through Continuous Integration Agile2009
-
Upload
sstolberg -
Category
Technology
-
view
2.821 -
download
2
description
Transcript of Enabling Agile Testing Through Continuous Integration Agile2009
Enabling Agile Testing Through Continuous Integration
Sean Stolberg, Pacific Northwest National Laboratory
Where We’re Going Today
Introduction - Being thrown into the deep end of Agile Testing
Our Plan – Deciding to Implement Agile Practices
The Problem
The Solution - Continuous Integration (CI)
What is CI?
Implementing CI
Assessing the Team’s CI Practices
Assessing Our New Agile Development Practices and Testing Capabilities
Retrospective
Being Thrown Into the Deep End
Me
Eight years as traditional software tester/QA engineer.
November 2007 – First job as a tester on an Agile team.
First two sprints
Humbling
All the “moves” I knew didn’t work too well.
Kept coming up short on time, test coverage, and accumulation of technical debt.
Re-evaluation
“What I’m doing is not working, so what will?”
Practices We Decided to Implement
1. Define and execute “just-enough” acceptance tests. [1]
2. Automate as close to 100% of the acceptance tests as possible. [2]
3. Automate acceptance tests using a “subcutaneous” test approach with a xUnit test framework. [2]
[1] “eXtreme Rules of the Road: How a tester can steer an eXtreme Programming project toward success”, Lisa Crispin, STQE Jul/Aug 2001
[2] “Testing Extreme Programming”, Lisa Crispin and Tip House, 2003, Addison Wesley
Practices We Decided to Implement (continued)
5. Run all acceptance tests in the regression test suite with the build, daily (at a minimum). [4]
6. Develop unit tests for all new code during a sprint. [5]
7. Run all unit tests with every build. [4]
8. Run multiple builds per day. [4]
[4] “Continuous Integration”, Martin Fowler, http://www.martinfowler.com/articles/continuousIntegration.html
[5] “Code the Unit Test First”, http://www.extremeprogramming.org/rules/testfirst.html
The Problem
We didn’t have an automation framework.
How could we:
Implement our new development practices?
Run all acceptance tests in the regression test suite with the build?
Run all unit tests with every build?
Run multiple builds per day?
The Solution
Continuous Integration
Commonly used by teams already successfully implementing Agile Testing.
Provides framework to implement several of the practices.
We needed to find out more.
What is Continuous Integration?
Describes a set of software engineering practices that
speed up the delivery of softwareby
decreasing integration times
Emerged in the XP community.
Martin Fowler and Kent Beck – helped make it popular.
What CI Might Look Like
1. Automated source repository change detection Change is detected resulting to a chain of events being put
into motion
2. Get latest source
3. Compile (or run through other check; interpreted code)
4. Run unit tests
5. Deploy to test environment
6. Run acceptance tests
7. Send results of testing
8. Publish build if tests pass
Speeding Up the Feedback Loop
Continuous Integration
Compile
Unit Testing
Deploy
Acceptance Testing
Publish build
Notify
Report
Tools Used
• First step in build
automation
• Configuration
• Dual Quad Core Xeon
1.6ghz
• 8GB Ram
• 4 x 146GB 15k RPM
SCSI drives in RAID-5.
• Unified build environment
and administration
• Get build machines out
from under the desks and
into the server room
Virtual Server Host
Virtual Build Server
IF a source code change is detect
• Refresh source code
• Build (ABS using VS2008)
• Clean build
• Build All
• Build Installer
IF no build errors were found
• Prep the test VM for product
install and testing (VBScript
helper script called via STAF)
• Copy install files to test VM
(STAF)
• Install app-under-test on test
VM (STAF)
• Discover and deploy all Unit
and Acceptance tests to test
VM (custom C# helper
app/STAF)
• Execute tests on test VM
(custom C# helper app/STAF)
• Copy test result .xml file(s)
from test VM to build machine
(STAF)
• Process the test result .xml
files (custom C# helper app)
Communication
Send mail with
• test results (PASS or FAIL
with details of failures)
• Link to location of build
• Build logs
• Change logs
Assessing our CI Practices
Maintain a Single Source Repository
Before CI: Seapine Surround SCM source repository.
After CI: No change.
Automate the Build
Before CI: Some partial automation; still very manual. Heterogeneous build environments.
After CI: Yes, with Automated Build Studio (ABS).
Make Your Build Self-Testing
Before CI: Not self testing.
After CI: NUnit framework unit and acceptance tests.
Assessing our CI Practices (continued)
Everyone Commits Every Day
Before CI: Unknown, varied probably.
After CI: We can only hope.
Every Commit Should Build the Mainline on an Integration Machine
Before CI: No, it was not happening.
After CI: Yes, ABS's Continuous Integration Tasks helped us do this.
Keep the Build Fast
Before CI: Multi-pass builds, unordered dependencies.
After CI: 10 to 15 min; refactoring needed sooner than later.
Assessing our CI Practices (continued)
Test in a Clone of the Production Environment
Before CI: Yes, but not automated
After CI: Using clean virtual machine test clients to install and test
Make it Easy for Anyone to Get the Latest Executable
Before CI: Not all projects using the common build repository. Some private file share locations for production code
After CI: All products building to common location now. Build mail contains link to new build location
Assessing our CI Practices (continued)
Everyone can see what's happening
Before CI: Limited to ad hoc e-mails, no web, no reporting, different project worked differently.
After CI: Use ABS's web interface to see the progress of builds, and email build and test status.
Automate Deployment
Before CI: Not being done.
After CI: Yes, automatically deploy the build, then test it.
Assessing Our Agile Practices
Define and execute “just-enough” acceptance tests
Made acceptance test definition a required task during sprint planning.
Developers and customer liked visibility into testing.
Automate as close to 100% of the acceptance tests as possible
Now tried to automate as close to 100% acceptance tests as possible.
Slowed our accumulation of technical test debt.
Assessing Our Agile Practices (continued)
Automate acceptance tests using a “subcutaneous” test approach with an xUnit test framework
NUnit test framework used to automate our acceptance/functional testing using the API of the application.
Acceptance tests versioned code in the repository and possible to run in a daily build.
Run all acceptance tests in the regression test suite with the build, daily (at a minimum)
NUnit acceptance tests now ran with a daily build.
Practice enabled using our new continuous integration system.
Assessing Our Agile Practices (continued)
Develop unit tests for all new code during a sprint
Developers started putting more emphasis on developing unit tests during the same sprint.
Practice encouraged due to ease with which unit tests could now be run with the build.
Run all unit tests with every build
NUnit tests now ran with every build.
Run multiple builds per day
Builds started automatically when changes to the source repository were checked in.
Retrospective
Personal “biggest discoveries”
Testing in sync with development.
Embracing fully automated acceptance tests
Exciting and intimidating.
Hard to convince developers to go through the automation “hump-of-pain”
But it’s worth it.
Devoting personal time toward investigation and proof-of-concept paid off.
Can’t imagine working on a team not using Agile Testing or development
Get “hooked” on injecting testing and quality early into the cycle.
Adds immediate value.
Recommended Reading
Crispin, Lisa, Tip House. Testing Extreme Programming. Addison-Wesley Professional, 2002
Gerard Meszaros, xUnit Test Patterns: Refactoring Test Code.Addison-Wesley Signature Series, 2007
Gregory, Janet, Lisa Crispin. Agile Testing: A Practical Guide for Testers and Agile Teams. Addison-Wesley Signature Series, 2008
References
[1] “eXtreme Rules of the Road: How a tester can steer an eXtremeProgramming project toward success”, Lisa Crispin, STQE Jul/Aug 2001
[2] “Testing Extreme Programming”, Lisa Crispin and Tip House, 2003, Addison Wesley
[3] “Continuous integration”, http://en.wikipedia.org/wiki/Continuous_Integration
[4] “Continuous Integration”, Martin Fowler, http://www.martinfowler.com/articles/continuousIntegration.html
[5] “Code the Unit Test First”, http://www.extremeprogramming.org/rules/testfirst.html
Questions???