How much testing is enough for software that can condemn a man to death? Traceability in an Agile...
-
Upload
craeg-strong -
Category
Technology
-
view
86 -
download
0
Transcript of How much testing is enough for software that can condemn a man to death? Traceability in an Agile...
1
Traceability in an Agile Federal Agency Context
Craeg Strong CTO, Ariel Partners
October 21, 2014 Washington, DC
How Much Tes,ng Is Enough… When it Comes to Ma8ers of Life and Death?
2
So&ware Development since 1988 Large Commercial & Government Projects Turned Around Projects With Agile Apache Ant Open-‐Source Contributor New York & Washington DC Area
CTO, Ariel Partners CSM, CSP, CSD, CSPO, PSM,
PMI-‐ACP, PMP
www.arielpartners.com [email protected]
@ckstrong1
Craeg Strong
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
3
Agenda
1. What Requirements Traceability Is 2. Why & When It Is Important 3. Some Typical Agile PracZces 4. Why They Are Insufficient 5. AlternaZve Approach: BDD/ATDD With SpecFlow 6. How/When/Who/Why 7. Solving The Problem With Agility 8. How To Get There: TransiZonal Strategies
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
4
Context
Examples Use Microsoft technology Equivalents Exist in Other Language Ecosystems (Java, LAMP)
q Air traffic control q Loss reserve calculation for long-tail insurance q Threat evaluation for detainee at border crossing q Nuclear power plant control q Digitized forensic evidence match at murder trial q Trading risk circuit breaker for high frequency or options
trading q Resource allocation for disaster relief
Heavily Regulated Industries Mission-‐CriZcal Systems Life-‐CriZcal Systems Financial Systems Legacy Systems Without Automated Tests
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
5
What Is Requirements Traceability?
• Inventory of all requirements • …traced to features implemented in the software, • …traced to test cases that cover the functionality • …with indicator for most recent pass/fail test status
Typical Artifact: Requirements Traceability Matrix (RTM)
System shall…
Module XYZ
Test Case 123
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
6
Why Is Requirements Traceability Important?
Ø Thousands of complex features Ø …developed over many years Ø …by people who are no longer on the
program Ø …few automated tests Ø …a brittle code base
How can we know we have tested everything?
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
7
Agile Approach: Automated Unit Tests
q One test generally tests one class q Written by developers q Intended to be run after every change q “Classic” unit tests: code only q xUnit extended family
q Database testing q Graphical user interface (GUI)
testing q Smoke testing q Performance testing
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
8
Measuring Unit Test Coverage
q Measures how much of the codebase is covered by at least one unit test
q Typical target number: 80% © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
9
LimitaZons of Unit TesZng Difficulty Creating Unit Tests q Legacy projects may not have good automated test coverage q Code may not be built for testability Only A Partial Solution q Non developers cannot understand unit tests q Coverage of code does not guarantee a feature works as needed
Build The Thing Right Build The Right Thing
GUI
UZliZes
Security / Logging Service
Business Logic
GUI Module Database Server
Persistence
Security / Logging
UZliZes
Service Business Logic
Feature
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
10
What about Test Driven Development (TDD)?
Test Driven Development q A Design technique q Tests are written first in order to drive the design q Additional tests may be added later
Why Does It Work? q There are generally multiple valid ways to design a piece of code q Writing a test first tends to produce more testable, flexible and
modular design
More Typical Alternative q Tests written during/after code q May require more refactoring to achieve desired results TDD: Good Practice; But Does Not Solve The Problem
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
11
SoluZon: Behavior-‐Driven Development (BDD)
q Different Process q Different Participants q Different Focus: Build The Right Thing
User Story #53
Acceptance Criterion 1
Acceptance Criterion 2
Acceptance Criterion 3
Automated Acceptance Test #53.1
Given a claim has been reported… When the claim is entered… Then the system should return…
Also Known as Acceptance Test Driven Development (ATDD)
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
12
Feature File BDD With SpecFlow
1. Write Feature
Step DefiniZons 2. Generate Template
3. Fill In Code
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
13
BDD With SpecFlow: Roles and Tools
Developer
Tester
Analyst
Product Owner
Syntax HighlighZng Support Gherkin Plugin
Syntax HighlighZng Support Gherkin Plugin
PicklesDoc
Document GeneraZon
CodedUIGUI-‐Level AutomaZon
Test AutomaZon Base Fwk
(or Selenium)
(or MSTest)
Three Amigos
Project Portal/Wiki
Given… When…
Then… Given… When… Then…
Whiteboard
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
14
BDD With SpecFlow: Process 3. Tester Completes
“Happy Path” Scenario
Tester
Given…
When…
Then…
5. Developer Implements Code And Test Steps
Developer SpecFlow Pair
2. Create Dra& SpecificaZon
During MeeZng
User Story 240 -‐ Acceptance Criteria 1) XYZ Field should be read-‐only 2) Total should match 999.99
Scenario: Ensure XYZ Field is
read-‐only when…
Scenario: Ensure Total is…
Analyst
1. Three Amigos Discuss User Story or Bug
Developer Tester Analyst
PO @ignore
Scenario XYZ
Given…
When…
Then…
4. Developer Adds Dra& SpecificaZon
To TFS
Developer
Big? Risky?
Small?
Fix Now
Technical
Product Backlog
User Story 999 Cleanup XYZ….
8. Rework, Refactoring, or
Cleanup IdenZfied?
7. Review Completed
Test
Developer Tester Analyst
6. Tester, Analyst Add
Addl Scenarios
Given…
When…
Then…
Given…
When…
Then…
Analyst Tester
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
15
GeneraZng DocumentaZon From SpecificaZons
q Pickles Report Generator q Features organized In tree structure q More sophisticated commercial options available
(SpecFlow+) © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
16
Organizing Automated Tests: By Type
q Named after class being tested
q Test project structure mirrors source code product structure
q Named after feature q Test project structure
mirrors business domain
Unit Tests Feature Tests
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
17
Organizing Automated Tests: Filter By Category
Category ExecuAon Speed Dependencies ExecuAon Context
@CI Very Fast None: Code Only Developer, before every check-‐in
@DB Slower Database structure, may use pre-‐set test data
CI Build
@smoke Slowest Full Running System Nightly Build
@CI Scenario: Ensure abc…
@DB
Scenario: Ensure xyz…
@smoke
Scenario: Ensure 123…
q Each individual test case is categorized according to its dependencies
q “Run All Tests” automatically runs the correct set of tests depending on execution context
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
18
Organizing Automated Tests: Test Frameworks
SpecFlow tests can be wri8en at different levels – even within the same feature file
Feature File
Unit!
!
CodedUI CodedUIStep DefiniZons – ImplementaZon AlternaZves
Gherkin
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
19
Organizing Automated Tests: Examples
Acceptance Criteria 1. Confirm that a trade that would trigger margin call will require
secondary authorization protocol 2. Confirm that a trader is not allowed to enter a quantity of
more than 50,000 shares for an options trade 3. Confirm that an out of the money trade will be flagged
User Story #13628 Enforce Trade Limits For Option Trades
Possible Strategy Why?
CodedUI Test using Moq Specific GUI is important and needs to be tested
NUnit test, no DB • CalculaZon is most important asset to test, GUI may change
• Legacy system is being rewripen and replacement system must support same logic
NUnit Test with DB • Significant base data setup required, mocking is not pracZcal
• Significant logic to be tested in the persistence layer
CodedUI Test with DB Full-‐stack smoke test is desired (requires more maintenance; there should be relaZvely few of these)
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
20
Extending BDD For Traceability q Epics Linked to User Stories in TFS q SpecFlow Tests Map To User Stories
User Story 1098 Feature: 1098 Scenario: XYZ Scenario: ABC
Epic 1608 User Story 2689
User Story 3104
User Story 7510 Epic 2387
User Story 7513
Feature: 2689 Scenario: XYZ Scenario: ABC
Feature: 3104 Scenario: XYZ Scenario: ABC
Feature: 7510 Scenario: XYZ Scenario: ABC
Feature: 7513 Scenario: XYZ Scenario: ABC
Pros 1. Simple 2. Easy to track
Cons 1. User story abstraction does not really
match feature 2. Requires maintenance
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
21
Challenges for Maintaining Traceability
Issues 1. A user story may describe only part of a true business
feature 2. Later user stories may modify, contradict, or replace earlier
user stories
Mitigation: Let the tests help 1. A failing SpecFlow test may indicate an older specification
needs maintenance 2. If the earlier SpecFlow test does not fail, consider whether it
is really important to update an earlier user story with new information
User Stories are designed to be ephemeral – only ac,ve within their Sprint or Release
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
22
The Agile RTM
q All system functions expressed as user stories q All user stories have acceptance criteria q Each criterion translated to an automated test using structured
English (Gherkin) q Customized report matches epics and user stories to automated
acceptance tests q Test fails unless software is implemented correctly
Links to Theme Record in TFS
Links to Epic Record in TFS
Links to User Story in TFS
Links to Feature DocumentaZon
Generated via Pickles
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
23
GeneraZng the Agile RTM
TFS Build
PowerShell Script
TFS Themes, Epics, User Stories
Console Runner Results of Automated Tests
PicklesDoc
Document GeneraZon SpecificaZons
Agile RTM Project Portal / Wiki Combine & Transform XML Files
Publish
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
24
Building A Test AutomaZon Capability BUILD TESTING CAPABILITY
q Technical Frameworks
q NUnit q SpecFlow q Moq
q Reporting Capability q Test Results q Code Coverage q Functional Coverage
q Encouraging Adoption q Examples Everywhere q Mentoring q Pair Programming
MAKE THE SYSTEM TESTABLE
q Upgrade dot net framework
q Drop support for old
operating systems
q Upgrade third party libraries
q Refactor code to isolate
dependencies
q Simplify project structure
q Different components may
need different strategies
Both Pieces Are Required For Success © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
25
TransiZonal Strategy: StarZng with Manual Test Cases
Epic
Manual TestCase
Manual TestCase
Manual TestCase User
Story
Manual TestCase
Legacy Req
Legacy Req
q Import legacy requirements into TFS
q Convert high-level requirements into epics
q Capture manual test cases in MTM
q Link MTM test cases to their requirements and/or epics
q Capture all new work using user stories
q Link legacy system requirements to epics alongside new user stories
Epic
Legacy Req
Legacy Req
Gradually migrate legacy “Shall” requirements to user
stories and acceptance criteria
Plan
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
26
ConverZng From Requirements to User Stories
Why do this? q System has long projected lifespan q Planned re-‐plarorm, migraZon, upgrade, or significant redevelopment
User Story
Legacy Req 1 to 1
Legacy Req Acceptance Criterion 1 to 1
User Story
Legacy Req 1 to many
User Story
User Story
Legacy Req
User Story many to 1
Legacy Req
Legacy Req
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
27
Legacy System Replacement
Benefits of SpecFlow for re-‐development q SpecificaZons confirm new system meets requirements q SpecificaZons represent up to half of overall effort q SpecificaZons provide documentaZon for the new system q Highlights areas where new system differs from old system q Significantly reduces risk
Legacy System Replacement System
Significant Reuse
Gherkin
SpecificaZons
Step DefiniZons
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
28
Summary
1. Traceability is important for mission criZcal and life criZcal so&ware
2. Automated test frameworks can provide the assurance we need ü Did we build the right thing ü Did we build it right ü Are all the required funcZons working correctly
3. SpecFlow has been proven effecZve
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
29
QuesZons?
© Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
We are available for consulZng or Agile coaching. Feel free to contact us!
Ariel Partners (646) 467-‐7394 [email protected] www.arielpartners.com @arielpartners
Thank You!