The Case for Automating System Level Radar Verification...
Transcript of The Case for Automating System Level Radar Verification...
![Page 1: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/1.jpg)
Copyright © 2015 Raytheon Company. All rights reserved.
Customer Success Is Our Mission is a registered trademark of Raytheon Company.
Edward Dou
Christian Reascos
August 19, 2015
The Case for Automating System
Level Radar Verification Testing
![Page 2: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/2.jpg)
Agenda Introduction to Complex Systems
Verification of Airborne Radars
Introducing Automation
Benefits / Cost Analysis
Conclusion
8/11/2015 2
![Page 3: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/3.jpg)
Complex System “…how parts of a system give rise to the collective
behaviors of the system, and how the system interacts with
its environment.” [NECSI]
Interactions with complex systems are not
always predictable 8/11/2015 3
![Page 4: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/4.jpg)
Airborne Radar Systems
8/11/2015 4
Parallel Processing
Various Jobs
Priority
Many competing tasks determined
by environmental data
Results Based
Limited Timeline
Pilot Interrupts
![Page 5: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/5.jpg)
Verification Construct
8/11/2015 5
Verification
Test
New
Requirements
Verification
Collateral
Regression
Overall
Functionality
Check
Software
Stress Test
• Built-in Test
• A/A Mode Entry
• A/G Mode Entry
• …
• Race Conditions
• Boundary Bugs
• Exception Handling
• …
• Existing Requirements
• Performance Requirements
• …
• Functional Requirements
• Performance Requirements
• ICD Requirements
• …
![Page 6: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/6.jpg)
Introducing Automation
8/11/2015 6
Automation
Tool
Operator
Action
Radar Response
Radar Performance Data
Pilot’s Control
(optional) Interim Panel Radar/Avionics HW
Simulated Threat
Environment
Real-Time
Statistics Automating Script
Automating Data Analysis
![Page 7: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/7.jpg)
Example
8/11/2015 7
Range Quality
0 nmi 5% Good (>85%)
10 nmi 86%
20 nmi 93% Satisfactory (60%-85%)
30 nmi 89%
40 nmi 78% Bad (<60%)
50 nmi 65%
60 nmi 67%
Automating Script
Automating Data Analysis
![Page 8: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/8.jpg)
Automating Tests
Functionality check – Functional Requirement
– Foundational Test
Software stress test – Stability & Reliability
– Staple Test
New requirements verification – Functional Requirement
– Performance Requirement
– Interface Requirement
Collateral regression – Existing Performance Requirement
8/11/2015 8
Test Characteristics Automation Benefit
Somewhat beneficial /
Potentially impactful
Very beneficial
Beneficial for
Performance Requirement
Very Beneficial
![Page 9: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/9.jpg)
Benefits and Costs
8/11/2015 9
COSTS
BENEFITS
• Testing Speed
• Test Repeatability
• Test Coverage
• Ease of Troubleshooting
• Continuous Coverage
• Instant System Evaluation
• Faster Data Collection
• Consistent Data Analysis
• Material (upgrades)
• Script Development
• Automation Tool Development
• Integration
• Maintenance
• Training
• Time to Implement
![Page 10: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/10.jpg)
Benefit Cost Analysis If B/C ≥ 1.0, accept the project as economically acceptable
If B/C < 1.0, the project is not economically acceptable
8/11/2015 10
Benefit
Est. Value
(1 Build)
Est. Value
(15 Builds)
Testing Time Saving $14,000 $210,000
Troubleshooting Time $3,200 $48,000
Reduced Rework $9,000 $132,000
Data Analsis Time Saving $2,400 $36,000
Reduced Resources $22,000 $54,000
TOTAL $50,600 $480,000
Cost
Est Dollar
Value
Material (including upgrades) $45,000
Script Development $12,000
Automation Tool Development $50,000
Integration $75,000
Maintanance $20,000
Training $8,000
Time to Implement (down time) $10,000
TOTAL $220,000
For the first build
B/C = (Total Benefits) / (Total Costs) = $50,600 / $220,000 = 0.23
For 15 builds
B/C = $480,000 / $220,000 = 2.18
* Example is for illustration purposes only -- values are representative
![Page 11: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/11.jpg)
Programmatic Environment Business Benefits
– Time Saving
– Resource Allocation
– Product Quality
– Money Saving
– Customer Contentment
8/11/2015 11
![Page 12: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/12.jpg)
Conclusion The benefits outweigh the costs
Technical benefits – Improvements over the current verification process
Business benefits – Quality, Time, and Money
Future Considerations – Intelligent simulation of pilot’s actions
– Learning system
8/11/2015 12
Automation saves time and improves the quality
of software updates in complex radar systems
![Page 13: The Case for Automating System Level Radar Verification ...itea.org/images/pdf/conferences/2015_Symposium...Automating Tests Functionality check –Functional Requirement –Foundational](https://reader035.fdocuments.net/reader035/viewer/2022071105/5fde9f0f8d5b6f0d373e6945/html5/thumbnails/13.jpg)
References New England Complex Systems Institute. About Complex Systems. n.d. Web. 16 July
2015. <http://necsi.edu/guide/>
M. E. Davis, "A history of battlefield surveillance radar." Radar Conference (RadarCon),
2015 IEEE. IEEE, 2015.
Raytheon Company. Raytheon's Advanced Combat Radar (RACR). 20 Sep. 2012, Web. 15
Jun. 2015. <https://www.youtube.com/watch?v=RSVziY1Zwn8>
Raytheon Company. Raytheon F 16 Integrated Fighter Mission Scenario. 4 Feb. 2014,
Web. 15 Jun. 2015. <https://www.youtube.com/watch?v=SnDUJfsNrbo>
H. Kaur, S. Ahamad, and G. N. Verma. "Elements of Legacy Program Complexity."
International Journal of Research in Engineering and Technology. IJRET, 2015
J. Ladyman, J. Lambert, and K. Wiesner. "What is a complex system?." European Journal
for Philosophy of Science 3.1 (2013): 33-67.
H. S. Mir and A. Guitouni. "Variable Dwell Time Task Scheduling for Multifunction Radar."
Automation Science and Engineering, IEEE Transactions on 11.2 (2014): 463-472.
SE Handbook Working Group INCOSE, INCOSE Systems Engineering Handbook v. 3.2.2,
INCOSE, 2011.
D. Hoffman, “Cost Benefit Analysis of Test Automation,” Software Quality Methods LLC,
1999.
L. Blank, A. Tarquin, Engineering Economy, 5th ed, McGraw Hill, 2002, pp. 308 - 310.
8/11/2015 13