Model Based Test Planning for Autonomous Systems...Example Results: Position Response...
Transcript of Model Based Test Planning for Autonomous Systems...Example Results: Position Response...
Innovations in Engineering
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Model Based Test Planning for Autonomous
Systems
Merging Technology Acquisition and T&E
September 15, 2011
1
Team MembersTroy Jones, George Sass, Melissa Durfee, Sean Buckley, Nick Borer,
Stephen York, Eric Nelson, Mike Ricard, Glenn Ogrodnik, Scott Ingleton
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Implementation
Concept of
Operations
Architecture
Requirements
Design
Integration, Test
& Verification
System
Verification &
Validation
Operations &
Maintenance
Behavioral
Model
Project Vision
2
Research Markov Reliability Analysis and DOE for System-Level test planning
Complementary to Model-Based Engineering
Performance Based Evaluation
Inject failure conditions
Force off-nominal decisions
Continuous Model Refinement
Feedback performance data over time
Improve predictions of future reliability
Implementation
Concept of
Operations
Architecture
Requirements
DesignIntegration, Test
& Verification
System
Verification &
Validation
Operations &
Maintenance
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Lifecycle Autonomous System Test Design
3
0 10 20 30 40 50 60 70 80
propulsion Omission
Att Psi Omission
Att theta Omission
pressure Omission
fin7 Omission
fin5 Omission
fin6 Omission
Att r Omission
fin4 Omission
sonar fls Omission
sonar bathy Omission
Att P Omission
sonar rsls Omission
sonar lsls Omission
0 nominal
Att q Omission
Att Phi Omission
Position Error: CurrentMag
|T|
current only rank
Current Magnitude
Markov Reliability Sensitivity
Ma
p
Mis
ma
tch
Current
Direct
ion
100%
1(0,0,0)
Priorities?
50%
Mismatch
Environmental
Conditions
B
Current
TemperatureWeather
A
C E
D
Design of Experiments
Mission
Conditions
Map
Quality
Objectives
Failure
Conditions
Sensors
AcutatorsSoftware
180°
Test Condition Ranking
Mission
Simulations
& Analysis
G(z)
K
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Behavioral Markov Reliability Analysis
Draper developed PARADyM Tool
System Markov Model
Component connections & logical dependencies
Component reliability values (MTBF)
Model Outputs
Probabilities
– Any failure condition over system life
– System Loss
Reliability Metrics
– System Reliability
– Sensitivity of Reliability to Component Failures
4
1
2
3
4
Failure
rate = a
Failure
rate = b
Failure
rate = c
a
b
c
Operational State
System Loss State
0 FL 1 FL
P(System Loss) = Σ(System Loss States)
Reliability = Σ(Operational States)
Motor
Pu
mp
Tank
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Case Study: Unmanned Underwater Vehicle (UUV)
Based on Typical Small UUV
Non-redundant
architecture
ASTM F41 Software
Architecture
Component Reliability
Estimates
Many hardware item
values (motors, valves)
from U.S. Army service
data
Computer & Sonar values
from vendors of similar
hardware
Simulated Failure Nodes
5
PCControls
Vehicle
Payload
Sensors
ACPerforms Mission
Planning, Commands
Steering & Speed,
Payload Use
Scheduling, SAVC
Performs Low-
Level Vehicle
Control and
Management
Simulation Models
(Ocean + Vehicle)
Sink 1
Sink 2
Injected
FailureSource
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Predominantly equivalent sensitivities (2 Fault Level Analysis)
Non-fault tolerant system design
Sonars and Battery Scanners stand out
Cascading failures to other components
Markov Reliability Analysis Results: UUV
6
Component Reliability
Sensitivity
Component Reliability
SensitivityAttitude Sensor H Bathy Looking Sonar
Attitude Sensor P Forward Looking Sonar
Attitude Sensor R Depth Pressure Sensor
Rate Sensor P Propulsion Motor Controller
Rate Sensor Q Propulsion Motor
Rate Sensor R Propulsion Sensor
Global Positioning System Fin Actuator Motor Controller
Doppler Velocity Sonar Fin Actuator1
Temperature Sensor Fin Actuator2
Saltwater Detector Sensor Fin Actuator3
Battery1 Fin Actuator4
Battery Scanner System Fin Sensor1
Power Distribution System Fin Sensor2
Vehicle Controller Fin Sensor3
Autonomous Controller Fin Sensor4
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
DOE: UUV System Responses
7
Type Response Description Rationale
Perf
orm
an
ce
Position Error (t) Deviation from baseline mission
path over time
Position errors cause data
collection errors
Attitude Error (t)
[φ,θ,ψ]
Deviation from baseline attitude
over time
Attitude errors cause data
collection errors
Speed Error (t) Deviation from baseline speed
over time
Speed influences
execution time, stealth,
energy
Reco
vera
bilit
y Energy Consumption Energy consumption for mission Must operate within
available energy limits
Mission Time Total mission time Establish expectations for
recovery/communication
Surface Position
Error
Deviation from designated end-of-
mission surface point
Large errors on surfacing
impact recovery
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Environmental Experiment Design
Available Environmental Factors (3)
Uniform Current Magnitude & Direction
Prior Terrain Knowledge
DOE Design
2 Level, 3 Factor Full Factorial – min/max
levels with median center point
Center point detects curvature
8
Min Median Max
Current
Magnitude 0 Knots 1Knots 2Knots
Current
Direction 0° 90° 180°
Map
Mismatch 0% 50% 100%
0°
180°
90°50%
Mismatch
100%
Mismatch
RunOrder CenterPt
CurrentMagnitude
(knots)
Current Direction
(deg)
Map Mismatch
(%)
1 1 2 0 100
2 1 2 180 100
3 1 0 0 100
4 0 1 90 50
5 1 0 0 0
6 1 2 180 0
7 1 0 180 100
8 1 2 0 0
9 1 0 180 0
Experimental Design with Center Point0%
Mismatch
Actual Terrain A priori Terrain MapCurrent
Direction
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Short Mission Evaluation Scenario
Short Mission Goals
Terrain avoidance
Waypoint following
Case Study Scenario
Design
Vary ocean currents,
map quality
~ 300 seconds
Approach & avoid
terrain on way to
waypoint
Basis of all case study
simulations
9
Draper Simulation
Framework (DSF)
Mission Visualization
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Execution & Data Analysis
10
Draper Simulation + Matlab ® Minitab ® MS Excel ®
Baseline
Execute Simulations 1..N
Parse Experiment Data Files
Identify Baseline Data Set
E ...E 2
E 1
Delta E ...Delta E 2
Delta E1
Delta(N)=E(N)-Baseline
Box-Cox Transformations
General Linear Model (GLM):
• Regression
• Main Effects & Interactions
• T-Test Values
0 10 20 30 40 50 60 70 80
propulsion Omission
Att Psi Omission
Att theta Omission
pressure Omission
fin7 Omission
fin5 Omission
fin6 Omission
Att r Omission
fin4 Omission
sonar fls Omission
sonar bathy Omission
Att P Omission
sonar rsls Omission
sonar lsls Omission
0 nominal
Att q Omission
Att Phi Omission
Position Error: CurrentMag
|T|
T-Test Ranking
(Pivot Tables)
Graphing
Box-Coxx2,√x ...
Max of |T-value| Column Labels
Row Labels MapMismatch
Energy**2
pressure_omission
Mission Time 143.44
attitude_rOmission 143.44
pressure_omission
fins_fin4Omission 116.2
Surface Position Error**-2
pressure_omission
Main Effects &
Interactions
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Results – Recoverability Responses
Many Failures Not Significant
Encouraging robust behavior
Vehicle recoverable from ~12
other failures
Candidates for Higher Fidelity
Tests:
Map Mismatch (navigation
error) with attitude sensor &
fin failures
Pressure sensor failures in
strong currents
Caveats
Propulsion failures not
registered as “significant”
Scenario setup/duration and
simulation boundaries mask
effects
11
0 20 40 60 80 100 120 140 160
pressure_omission
attitude_rOmission
pressure_omission
fins_fin4Omission
pressure_omission
Ener
gy*
*2
Mis
sio
n T
ime
Surf
ace
Po
siti
on
Er
ror*
*-2
Recovery Responses: Signficant Factors & Failures
CenterPt
CurrentDirection
CurrentDirection*CurrentMagnitude
CurrentDirection*MapMismatch
CurrentMagnitude
CurrentMagnitude*MapMismatch
MapMismatch
|T|
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Results – Mission Performance Responses
Key Failures to Test Further
Certain attitude & rate
Sensors (Phi, p, q)
All Sonars (agrees with
Markov)
Observations
Sensor failures more
important than fin failures
Similar sensor failure
rankings in other responses
Propulsion rank likely
incorrect for longer
scenarios
Position Control
12
0 10 20 30 40 50 60 70 80
propulsion Omission
Att Psi Omission
Att theta Omission
pressure Omission
fin7 Omission
fin5 Omission
fin6 Omission
Att r Omission
fin4 Omission
sonar fls Omission
sonar bathy Omission
Att P Omission
sonar rsls Omission
sonar lsls Omission
0 nominal
Att q Omission
Att Phi Omission
Position Error: CurrentMag
|T|
current only rank
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Conclusions
Lifecycle Autonomous System Test design process
developed & evaluated
Unmanned Underwater Vehicle (UUV) Case Study
Markov Reliability Analysis – shows utility, needs
better integration
Design of Experiments – valuable insights to
prove relationships and focus subsequent tests
Future Work
Integration of Markov Analysis with Draper
Simulation
Fault injection into software layers of the system
Innovations in Engineering
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Supplemental Slides
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
UUV Reliability Studies
Little Published Reliability Data
Many University projects
Inconsistent record keeping
Multi-Year Reliability Studies on Autosub
UK Oceanographic Survey UUV
Extreme deep diving
Terrain following
Science data collection
The Ocean is a Harsh Mistress
Majority of failures in [sensor] hardware (leaks, shorts)
Humans caused more faults than software
Estimated 63% Survival Probability for long Under-Ice mission
Lessons
Need Test & Evaluation rigor in development (happening now)
Don’t always assume software is the long-pole
15
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0
5
10
15
20
25
Sensing
Hardware
Vehicle
Hardware
Human Software Ship
Autosub Fault Pareto4 Years, 240 Missions, 1600 NM
Relia
bility
Autosub Reliability Data & Model
Data from Griffiths, G. et al., “On the Reliability of the Autosub
Autonomous Underwater Vehicle”, Underwater Technology, 2002
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Results – Mission Performance Responses
Key Failures to Test Further
Certain attitude & rate
Sensors (Phi, p, q)
All Sonars (agrees with
Markov)
Observations
Equivalent to Position
Control, plus pressure
sensor failures
Repeat of Phi & q as top
ranking failures (not
predicted by Markov)
Attitude Control
16
current only rank
0 10 20 30
Att Psi Omission
Att theta Omission
Att r Omission
fin7 Omission
fin4 Omission
fin5 Omission
fin6 Omission
pressure Omission
sonar bathy Omission
sonar fls Omission
sonar rsls Omission
sonar lsls Omission
Att P Omission
0 nominal
Att q Omission
Att Phi Omission
Attitude Error: CurrentMag
|T|
current only rank
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
0 10 20 30 40
propulsion Omission
Att theta Omission
Att r Omission
fin7 Omission
fin4 Omission
Att Psi Omission
pressure Omission
fin5 Omission
Att q Omission
fin6 Omission
Att Phi Omission
sonar fls Omission
sonar bathy Omission
Att P Omission
0 nominal
sonar rsls Omission
sonar lsls Omission
Speed Error: CurrentMag
|T|
Results – Mission Performance Responses
Key Failures to Test Further
Certain attitude & rate
Sensors (Phi, p)
All Sonars (agrees with
Markov)
Observations
Repeat of Phi as high
ranking failure
All performance
responses highly
dependant on current
Consistent high
rankings of similar
failures
Speed Control
17
current only rank
UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved
Example Results: Position Response
18
-400-300
-200-100
0100
200300
0
500
1000
1500
2000
2500
0
20
40
60
80
100
120
Crosstrack (ft)
UUV Path During Select Bathymetric Sonar Failures
Downrange (ft)
Dep
th (
ft)
Nominal Failure Case
Bathy Fails, Map Mismatch 100%
Bathy Fails, 50% Map Mismatch, 2 knot Side Current
Bathy Fails, 4 knot Tail Current, Mission Incomplete
System Baseline