Innovation day 2013 2.4 frederik mortier (verhaert) - test management
42
1 CONFIDENTIAL Test management: How to organize your research and test campaign during the feasibility study
-
Upload
verhaert -
Category
Technology
-
view
237 -
download
0
description
Transcript of Innovation day 2013 2.4 frederik mortier (verhaert) - test management
1
CONFIDENTIAL
Test management: How to organize your research and
test campaign during the feasibility study
2
Frederik Mortier
Consultant Applied Physics & Systems 2012- now
Researcher @ Ghent University 2011-2012
MSc. in Engineering – Electro-Mechanical 2009-2011
MSc. in Industrial Sciences – Electro-Mechanical 2005-2009
3
“A feasibility study is not only meant
to confirm a good idea…”
4
“…, it could also detect a bad idea!”
5
CONFIDENTIAL
Outline
6
Outline
Context within NPD
Problems & Risks
Efficient Test Approach
Conclusion
7
CONFIDENTIAL
Context within NPD
8
Concept Design &
Development Production
Visualization Testing Simulation Review
Quick&Dirty Detailed Conclusion
Utility Allowability Desirability Feasibility Usability
When
What
How
Context within NPD
9
Cut
Tangible
• Visualize
• Simulate
• Test
• Review
• Roadshow
Risk focus
• Criticalities
• Added value
• 360°
Early
• Rapid prototyping
• First time right
Options
Context within NPD
Risk reduction methods
10
Risk Management
Risk
identification
Risk analysis
Risk evaluation Technology
management
Development
Tools &
methods
Verification
Risk
mitigation
Requirement
mgmt
Risk assessment Risk treatment Risk monitoring
Validation
Context within NPD
11
Why Feasibility Study?
Investigate feasibility of the idea in the application
•Confirm / deny added value
•Determine required costs
By determining:
Feasibility Study
Strengths vs.
Weaknesses
Opportunities vs. Threats
Needed resources
Context within NPD
Risk mitigation
12
Context within NPD
Added Value – What do we want to find out?
Feasibility
• Is the core concept feasible ?
Proof-of technology
• Does my core technology work in the lab ?
Field demon-strator
• Will my design function in the application?
13
Context within NPD
Added Value – Validation vs. Verification
Needs Requirements Design Test setup Implementation Documentation
Validation
Check if developed product fulfills its purpose
(intended use)
Are we building the right product?
Field test : full system test in real life
Demonstrator in application
Test in production if the operator can measure a
hole Ø and can detect faults.
Validation = application owner (customer)
Verification
Check if developed product is compliant with
specifications
Are we building the product right?
Lab test / Analysis / Simulation
Breadboards / Proof-of-concept in lab
Test on a sample plate if we can reach e.g accuracy,
w.r.t. reference technology.
Verification = development partner (engineer)
14
Context within NPD
Cost Required
Why wait for the risk to come true ?
More freedom to adjust in early project phases
Early detection of risk allows for measures at
lower cost and with less impact on schedule
Normal
Time
Learning curve
System specification
Conceptual design
Detailed design
Tooling / purchase
Integration
Test
Reversed
Fail fast, fail cheap!
15
Context within NPD
Why Testing?
Different aspects of technical part of feasibility study:
•Better guarantee on final results
•Simulations often too complex/time consuming
•Sometimes fastest method
•New technologies need testing (no previous experience)
Feasibility testing is done to learn!
Avoid unnecessary testing Example: blister packaging & NASA
Visualization Testing Simulation Review
16
NASA example
Source: http://msis.jsc.nasa.gov/sections/section04.htm
17
Context within NPD
Test Management
Managing of test activities:
• Preparing and managing test plan
• Preparation of tests
• Choice of test approach + testing
• Interpretation of test results
• Documentation / reporting results
Iterative process during feasibility study
18
CONFIDENTIAL
Problems & Risks
19
Problems & Risks
Risks during feasibility phase
Risks dealt with are the same as in the overall risk management
• Budget
• Planning
• Performance
Perfor-mance
Planning
Risk manage
ment
Budget
Keep 3 axes in balance !
20
Budget risks
Uncontrolled budgets
Timing
• Long test times
• Documentation
Test setups / tools
• Renting vs. buying
• Too advanced setups too soon
• Rework due to unforeseen events
Perfor-mance
Planning
Risk manage
ment
Budget
Problems & Risks
21
Planning risks
Planning
Long duration of test campaign
• Test conclusions too late
• High criticality items too late
Tests vs. Resources
• Equipment availability
• Personnel availability
Lead times
• Equipment/tools
• Software
• Test stands
Perfor-mance
Planning
Risk manage
ment
Budget
Problems & Risks
22
Performance risks
Performance
Wrong result
• Wrong measurement method
• Wrong measurement tool
• Inaccurate results
• Wrong test order
Wrong basic requirement
• Wrong test criteria
“risk based methods in NPD”
Perfor-mance
Planning
Risk manage
ment
Budget
Problems & Risks
23
CONFIDENTIAL
Efficient Test Approach
24
Outline
Objective of testing
Evaluate performance / behavior to remove / reduce project risk by
testing on “representative hardware”
Ways of testing
• Proof of concept bread boarding
• Works like real application demonstrator
• Looks/works/tastes like real product prototype
Efficient test approach
Test Management
25
Quick & Dirty Testing
Test Plan Development
Detailed Testing
Conclusion
Efficient test approach
Test flow
26
Quick & Dirty Testing
Test Plan Development
Detailed Testing
Conclusion
Efficient test approach
Test flow
27
Efficient test approach
Quick & Dirty
What?
• Basic testing of hypothesis
• Basic materials/tools/budgets
• Wide window
• Unexpected phenomena
When?
• Prior to full test plan development
• Sometimes during FFE
Why?
• Upstream testing reversed learning curve!
• Variables and operating windows
• Significance of these variables risk based testing
• Reduced cost no expensive tools / instrumentation
• Reduced timing short tests, short tooling lead times
Increased test output with reduced risk
28
Efficient test approach
Quick & Dirty Examples
Basic testing:
29
Efficient test approach
Quick & Dirty
So why don’t they?
• Corporate culture administration during testing
• Availability tools & instrumentation
• Downstream test approach
• Used of testing known things or by known methods
• Afraid to get hands dirty..? Hot potato issue!
• Foresee time & budget for Quick & Dirty testing
• Ensure basic equipment is at hand
Low threshold for testing should be encouraged
30
Quick & Dirty Testing
Test Plan Development
Detailed Testing
Conclusion
Efficient test approach
Test flow
31
Efficient test approach
Test plan development
1. Gathering inputs
• Desk search
• Quick & dirty test results
• Specification document
• Risk assessment
• ..
2. Define test actions
List items according to function / module / ..
3. Define acceptance criteria
Based on gathered information
4. Assign priorities to each test based upon risk/criticality
Risk based testing
Assign risks well thought
32
Efficient test approach
Test plan development
5. Define test methods based on:
• Variables
• Measurement method
• Automatic vs. manual
6. Define setup & tools
• Measurement method
• Cost (rent vs. hire)
• Location (EMC, Clima-chamber)
7. Estimate timing
• Lead time setup & tools
• Estimated testing time
Test plan will give priority / budget / timing overview!
33
Quick & Dirty Testing
Test Plan Development
Detailed Testing
Conclusion
Efficient test approach
Test flow
34
Efficient test approach
Detailed Testing
• Instrumentation
Calibration & sanity check!
Setting up instruments correctly
• Test stand
Designed to be versatile & adaptable
adaptation on the fly for faster results
• Testing
Test order based on priorities (high to low)
Reassess priorities/tasks as you go
Eyes open for undefined phenomena & assess their influence
test plan iteration
35
Efficient test approach
Detailed Testing
• Documentation
On the fly most efficient, less hassle for test engineer use templates!
Take pictures store them for later use, small effort
Need for computer at test stand + internet
36
Efficient test approach
Breadboard examples
More advanced testing
• Use of data-acquisition
• Use of rapid prototyping
• Use of automation devices
37
Quick & Dirty Testing
Test Plan Development
Detailed Testing
Conclusion
Efficient test approach
Test flow
38
Efficient test approach
Test conclusion
Test action acceptance
• Fill in test plan acceptance fail/pass
• Comment on possible fails
• Identify show stoppers as soon as possible
Feasibility Study
Strengths vs.
Weaknesses
Opportunities vs. Threats
Needed resources
39
CONFIDENTIAL
Conclusion
40
Efficient test approach
Conclusion
Reduce the barrier for Quick & Dirty testing
Lower threshold to start testing + getting a feel with the matter
Define test window + parameters for more detailed & accurate testing
Plan your testing based on risks
start with highest priorities/risks first
Reassess test plan priorities as you go
Test plan helps you control:
• Priorities / risks
• Cost tooling/setup
• Timings
Perfor-mance
Planning
Risk managem
ent
Budget
Keep 3 axes in balance !
Maximize output, reverse the learning curve
Reduce overall risk
Risk based testing
42
VERHAERT MASTERS IN INNOVATION® Headquarters
Hogenakkerhoekstraat 21
9150 Kruibeke (B)
tel +32 (0)3 250 19 00
fax +32 (0)3 254 10 08
More at www.verhaert.com
VERHAERT MASTERS IN INNOVATION® Netherlands
ESIC European Space Innovation Centre
Kapteynstraat 1
2201 BB Noordwijk (NL)
Tel: +31 (0)618 12 19 19
More at www.verhaert.com
MASTERS IN INNOVATION® is a platform set up by VERHAERT to train, stimulate and incubate
you as an innovator.
We provide an extensive training program with different tracks and covering critical areas of new
products and business innovation.
Furthermore we manage the VERHAERT venturing program and organize our Innovation Day, an
annual conference on best practices and insights on new products & business innovation.
