Post on 10-Feb-2016
description
Injecting Faults for Error Evaluation
NASA Glenn Research CenterKalynnda Berens, SAICRichard Plastow, SAIC
SAS 2004 - Fault Injection 2
Mission Success Starts With Safety
Introduction Applications often consists of software components
plus custom development, merged into a coherent package. COTS, GOTS, open source, etc.
Source code is usually not available for review of quality and reliability. Visibility into the component is only what’s available via a
public interface What is the quality of that component? What faults lay inside the component?
Applications interface with hardware and other software and can be influenced by failures in those systems.
SAS 2004 - Fault Injection 3
Mission Success Starts With Safety
Fault Injection on Interfaces Interfaces (hardware, software, human) are a
major source of errors and induced faults Software and system testing looks at anticipated
off-nominal situations, but often misses unusual situations or combinations of faults
Mishap investigation has shown that multiple faults or unexpected anomalies are key players in accidents and mission failures
SAS 2004 - Fault Injection 4
Mission Success Starts With Safety
Example System
ApplicationCOTSLibrary
COTS Operating System
Other Applications
on same system
External Systems
System Hardware
Input Sensors
Control Outputs
SAS 2004 - Fault Injection 5
Mission Success Starts With Safety
Fault Injection Flow Diagram
No
Identify Interfaces and Critical Sections
Error/Fault Research
Estimate Effort Required
Obtain Source Code and Documentation
Start
Sufficient time and funds?
Importance Analysis
Select Subset
Test Case Generation
Fault Injection Testing
Document Results, Metrics, Lessons Learned
Feedback to FCF Project
End
Yes
Results
SAS 2004 - Fault Injection 6
Mission Success Starts With Safety
Interface Identification Artifacts and Documentation
Software and System Requirements and Design specifications
Interface SpecificationsUser and Training ManualsHardware DocumentationOther project documentation
For FCF, “Signals List” Source code
SAS 2004 - Fault Injection 7
Mission Success Starts With Safety
Error Research Sources of Error/Fault Information
Vendor documentationPublic bug list Internet SourcesSoftware logsError databasesProject ExperiencePrevious Test ResultsPersonnel Experience
SAS 2004 - Fault Injection 8
Mission Success Starts With Safety
Estimation of Effort Determine level of effort, funding, time
constraints If complete effort not possible
Perform importance analysis of interfaces, software units
Safety Complexity Use by other system elements Expected number or types of faults
Prioritize and select by importance
SAS 2004 - Fault Injection 9
Mission Success Starts With Safety
Testing Test case generation based on identified errors plus
permutations on possible input values Consider multiple faults Consider faults while system is off-nominal from a
previous fault Consider effects of system load/stress Consider state-specific effects Instrument software to observe effects of injected
faults External or observable effects State changes (or lack of) Effects on safety-critical functions
SAS 2004 - Fault Injection 10
Mission Success Starts With Safety
Results: First Project: Tempest
Written in Java 1.1 Configurable Cross platform operability Implements HTTP GET and HEAD Request and
Server Side Includes Has some Basic Security Features Debug Mode monitoring Commercially available
SAS 2004 - Fault Injection 11
Mission Success Starts With Safety
Tempest Critical Errors Inappropriate system operation with modified
configuration file Non-compliance with HTTP standard System crash with invalid port numbers
Port 49151.45 -> opened port 80 File access in server machine outside of
authorized directories System did not operate as per user
documentation
SAS 2004 - Fault Injection 12
Mission Success Starts With Safety
Results: Fluids and Combustion Facility Permanent, multi-user facility for ISS microgravity
experiments Fluids Integrated Rack (FIR) and Combustion Integrated
Rack (CIR) Operates for 10 years, so robustness important CANbus processors selected for fault injection
Health and Status Monitoring Cannot be upgraded in flight Mature requirements, design, and interface definition Source code available
SAS 2004 - Fault Injection 13
Mission Success Starts With Safety
CANbus Processors Air Thermal Control Unit (ATCU) Color Camera Package (CCP) FOMA Control Unit (FCU) FSAP Diagnostic Board Input/Output Processor (IOP) IPSU Diagnostic Board* Mass Data Storage Unit (MDSU)* Nd:YAG Laser Package* Water Thermal Control System (WTCS) White Light Package
* Not yet tested
SAS 2004 - Fault Injection 14
Mission Success Starts With Safety
FIR System Diagram
IOP Main Processor IOP HRDL
Processor
IOP Video Switch
Processor
IOP CAN Node
Processor
Input-Output Processor (IOP)
FSAP Main Processor
FSAP CAN Node
Processor
FSAP
IPSU Main Processor
ISPU CAN Node
Processor
Common IPSULaser Diode
CAN Processor
White Light CAN
Processor
DCMCAN
Processor
Nd:Yag CAN Processor
PI Package
ECS CANbus
ATCU CAN Processor
WTCS CAN Processor
Optics Bench CANbus
Ethernet
MDSU Main Processor
MDSU CAN Node
Processor
MDSU
SAS 2004 - Fault Injection 15
Mission Success Starts With Safety
CANbus Processor State Diagram
Off-Nominal (O-N)
Power Down (P)
Power On
Initialization
Power Off
Operational (OP)
Power Down Cmd
Error
Success
Operational Cmd
Error
Error
Operational Cmd
Power Down Cmd
SAS 2004 - Fault Injection 16
Mission Success Starts With Safety
Testing Software
SAS 2004 - Fault Injection 17
Mission Success Starts With Safety
Test Setup
SAS 2004 - Fault Injection 18
Mission Success Starts With Safety
FCF Fault Injection Process Interface Identification and prioritization Obtain hardware, source code for testing
environment Error/Fault search on selected interfaces and
components Static analysis using Understand™ tool Analysis of previous testing, defects Test case generation, source code
instrumentation, and test execution
SAS 2004 - Fault Injection 19
Mission Success Starts With Safety
Types of faults injected Out-of-range Unexpected input Multiple errors Timing Flood the input with values Remove Input/Output Interrupt Input/Output
SAS 2004 - Fault Injection 20
Mission Success Starts With Safety
FCF Results Software previously qualified 35 errors, 3 critical
Loss of the output connection caused a continuous reboot
Changing the processor address caused a hang condition
Going to the input limits caused invalid telemetry to be sent.
Project corrected 20 errors 4 errors still in process
Testing still in progress
SAS 2004 - Fault Injection 21
Mission Success Starts With Safety
Final Steps In-depth case study (ISS flight payload)
Update Fault Injection Methodology documentRecord all the details – problems as well as
successesCompare results to other defect detection
mechanismsWritten for those who want to try the technique
Release FI Methodology and Case StudyDecember, 2004
SAS 2004 - Fault Injection 22
Mission Success Starts With Safety
Passing the torch Potential applications
Any software project using COTS software or with hardware interfaces
Data and Case StudiesFault Injection Methodology (draft)
Available through SARPCase Study (FCU main processor)
Available December, 2004