Post on 20-Aug-2020
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A
Reference Number 15-S-2335; 08-14-2015
Applying Big Data Analytics in T&E
August 2015
Ryan Norman
TRMC Initiative Lead for Knowledge Management
ryan.t.norman.civ@mail.mil
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A
Reference Number 15-S-2335; 08-14-2015 2
Source: IDC’s Digital Universe Study, sponsored by EMC, Dec 2012
More information over the next two years than in the entire history of mankind!
Name (Symbol) Value
2020 40,000 EB
2005 130 EB
2010 1,250 EB
2015 8,000 EB EX
AB
YT
ES
Worldwide Exponential Growth of Data
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A
Reference Number 15-S-2335; 08-14-2015
Test & Evaluation Growth of Data
• Larger Test Footprints–4-on-4 test flights (more systems per test)
–Much faster weapons systems
–Geographic separation not as effective as it
used to be
• Demand for Shorter Acquisition
Cycles–More concurrent testing
–More real-time analysis
• Increased System-of-Systems
Test Complexity–“Five Futures” (EW, UAV, NCO/W, DE,
Hypersonics)
– Integrated fleet (F-18E/F, E-18G, F-35, SM
VI, UAV)
–“Swarming” UAVs
Increased System Complexity
Integratedweapon system(1-20 Mbps ea.)
Integratedavionics(2-10 Mbps)
Separationvideo(1-8 Mbps)
Cockpitvideo(1-5 Mbps)
Integratedcommunications(0.5-2 Mbps)
Flight test transducers(0.5-10 Mbps)
Multi-spectralsensors(1-12 Mbps)
Integrated self-defense system(0.5-2 Mbps)
Total Throughput: 7.5Mbps – 70Mbps+
3
T&E Mission: Acquire data and discern into knowledge
44
Big Data / Knowledge Management (KM)Challenges & Needs
T&E Infrastructure Challenges:
– How do we conduct T&E of increasingly complex, data-driven systems?
– How do we enable more efficient & continuous system evaluation?
Need: A DoD-wide KM capability for T&E to help
achieve better acquisition outcomes and reduce costs
– Trusted processes across government and
industry that identify problems sooner rather than later
– Accessibility of knowledge & data to legitimate users
– Discoverability of knowledge & data obtained
over time
– Availability of knowledge through common tools &
technologies – including DoD T&E cloud solutions
– Leverages proven Industry techniques / practices
Big Data Analytics depends on effective Knowledge Management
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A
Reference Number 15-S-2335; 08-14-2015
The TRMC “Blueprint”:Putting Test Capabilities on the DoD Map
Risk mitigation needsTechnology shortfalls
Risk mitigation solutions Advanced development
Capabilities
Service Modernization and
Improvement Programs
Acquisition Programs and Advanced
Concept Technology Demonstrations
T&E Multi-
Service/Agency
Capabilities
DoD Corporate
Distributed Test
Capability
TRMC Joint Investment
Programs
Transition
Requirements
Strategic Plan for
DoD T&E Resources
Annual T&E
Budget
Certification
(6.3 Funding) (6.6 Funding)(6.4 Activity)
DT&E / TRMC
Annual Report
Defense Strategic Guidance
Service T&E Needs and Solutions Process
Acquisition Process
5
6UNCLASSIFIED – DISTRIBUTION STATEMENT A
Realizing Improved DoD T&E Knowledge Management
1. Understand and Document T&E challenges & needs
– (FY12) Completed Data Management for Distributed Testing (DM-DT) Study− Result: Developed functional requirements for T&E enterprise distributed Data Management
– (FY13) Comprehensive Review of T&E Infrastructure report published− Key Recommendation: Use DoD cloud solution for T&E data
− Key Recommendation: USD(AT&L) establish a DoD-wide KM capability for T&E to help achieve
better acquisition outcomes and reduce costs
2. Execute proofs of concept that inform an enterprise approach to T&E
Knowledge Management
– (FY14-15) Joint Strike Fighter Knowledge Management (JSF-KM) project− Goal: Assess KM technologies and methodologies in support of an existing acquisition program
– (FY15-16) Collected Operational Data Analytics for Continuous Test & Evaluation
(CODAC-TE) project− Goal: Apply KM technologies and methodologies across the lifecycle
3. Develop investment plan that achieves strategic objectives:
– Integrate T&E infrastructure into cohesive Knowledge Management enterprise
– Modernize T&E practices & processes to leverage Big Data analytics techniques
– Apply Big Data analytics tools & techniques to the T&E mission space
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A
Reference Number 15-S-2335; 08-14-2015
JSF Big Data Analysis Challenges
• More data is being collected for JSF than can be quickly
analyzed
– JSF-KM Deliverable: Big Data analysis tools applied to JSF test data
• Data sets are too large to download, requiring engineers and
analysts to travel
– JSF-KM Deliverable: Remotely accessible Knowledge Management
system installed at Edwards and Nellis
• Many data sets are not well organized
– JSF-KM Deliverable: Knowledge Management portal that catalogs,
organizes & retrieves JSF data across selected test locations
• Utility of data limited by our ability to interpret & use it
– JSF-KM Deliverable: Big Data visualization tools applied to JSF test data
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A
Reference Number 15-S-2335; 08-14-2015
User
Eglin
Pax
River
Joint Strike Fighter Knowledge Management (JSF-KM) Test Concept
• DT & OT data storage in government facilities
• Collect / Store more precise data during OT
• Search / Analyze Edwards & Nellis
data from any secure location
• Bring enhanced JMETC infrastructure to JSF T&E
• Apply commercial Big Data and Knowledge Management tools to DoD requirements
• Knowledge shared across
JSF DT / OT T&E locations
• Scalable to other JSF T&E locations
8
JPO
Pote
ntia
l Expansio
n
UNCLASSIFIED - Distribution A
JSF-KM Improvements to Existing T&E Capabilities
9
DT Today
OT Today
With JSF-KM
Parallel
Data
Ingest
30 minutes
(multiple aircraft)
Raw Data
Available
Video/Data at Post-
Mission Debrief Big Data
Analytics
Govt. Analyst
Data Request
Analysis
Note: Numbers reflect single 2 hour flight mission
Data
Ingest Raw Data
AvailableGovt. Analyst
Data RequestAnalysis
2 hours
(per aircraft)1 day 1 week
30 seconds
Data
Ingest Raw Data
AvailableGovt. Analyst
Data RequestAnalysis
1-2 hours
(per aircraft) 10 minutes 4-5 hours
Data Ready for
Use @ (Govt)
30 seconds
30 seconds
Data Ready
for Use @ LM
>20 weeks of data
available online
Data Ready for
Use @ (Govt)3 weeks of data
available online
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A
Reference Number 15-S-2335; 08-14-2015
JSF-KM Success Stories October 2014 – May 2015
• Resolved test data time correlation issues
– Time stamps in data files found to be corrupted post-mission
– JSF-KM analysis tools were able to correct the time correlation issue
– Without JSF-KM, at least five missions would have been re-flown
• Video available during post-mission debrief due to JSF-KM data
ingest improvements from DART Pod
– Existing tools could not process video in time to support post-mission de-brief
– Without JSF-KM, there would be no flight video during post-mission debrief
• Discovery of avionics box issue
– Pilot and Analyst discovered problem from video data available 30 minutes after landing
– Avionics Box was replaced before another mission was flown
– Without JSF-KM, problem would not have been discovered for several days
10
Return on Investment has been realized before
deploying any Big Data analytics capabilities
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A
Reference Number 15-S-2335; 08-14-2015
Collected Operational Data Analytics for Continuous Test & Evaluation (CODAC-TE) Proof of Concept
• Background: US Army Aberdeen
Test Center (ATC) has ~30TB of
underutilized data – including 20TB
of in-theater operational data
• Goal: Utilize Big Data analytics
across multi-commodity DT, OT,
and in-theater system data to
discover “unknown unknowns” for
current and future Army systems
• Use Cases: Mine-Resistant Ambush
Protected (MRAP) Theater Data;
Camouflage Effectiveness
• Leverages High-Performance
Computing Major Shared Resource
Center and ATC expertise
Challenges being addressed:
• Insufficient data science expertise
• Current analytical systems inadequate for today’s data volume and velocity
• Lacking tools and techniques for discovering unknown unknowns and conducting
complex trends analyses
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A
Reference Number 15-S-2335; 08-14-2015
Knowledge Management (KM) Initiative Summary
• TRMC is acting upon the KM recommendations from the
Comprehensive Review of T&E Infrastructure. Strategic Goals:
– Integrate T&E infrastructure into cohesive Knowledge Management enterprise
– Modernize T&E practices & processes to leverage Big Data analytics techniques
– Apply Big Data analytics tools & techniques to the T&E mission space
• TRMC-funded proofs of concept will deliver proven capabilities
– Enable Big Data analytics during JSF T&E
– Improve transfer of knowledge between fielded and next-gen systems
– Inform T&E investment plan that advises future infrastructure, process, and
workforce decision-making
• Improved T&E KM will help achieve better acquisition outcomes and
reduce costs
– Identify & Diagnose problems sooner and continuously
– Inform acquisition decisions through larger knowledge base
– Achieve T&E infrastructure efficiencies
12
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A
Reference Number 15-S-2335; 08-14-2015
Questions?
Ryan Norman
TRMC Initiative Lead for Knowledge Management
ryan.t.norman.civ@mail.mil
13
14UNCLASSIFIED – DISTRIBUTION STATEMENT A
What is Knowledge Management (KM)?
• Data represents a fact or statement without relation to other
things
– Example: It is raining
• Information is data that has been placed in a meaningful
context
– Example: The temperature dropped 15 degrees & then it started raining
• Knowledge is a collection of information with some extracted
value
– Example: If the humidity is very high & the temperature drops substantially, the
atmosphere is often unlikely to hold the moisture – so it rains
• Knowledge Management (KM) is the process of efficiently
capturing, handling, distributing, and using knowledge
• KM is based on two critical activities:
– The capture & documentation of explicit & implied knowledge
– The dissemination of this knowledge within an organization(s)
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A
Reference Number 15-S-2335; 08-14-2015
Long-term DoD T&E Enterprise Knowledge Management Vision
15
Result: T&E data used more effectively & efficiently during acquisition
User• The primary product of T&E is data & knowledge
• Embrace KM & Big Data Analytics to efficiently handle & securely share T&E data
• Organize T&E data to build knowledge across all DoD acquisitions
• Federate distributed data repositories to enable execution & automated search scenarios that cannot occur today
• Utilize modern mechanisms to enable collaboration between SMEs in government and industry
16UNCLASSIFIED – DISTRIBUTION STATEMENT A
A Selection of Items Under Assessment
• Process:
– What business process changes to T&E are needed to enable an enterprise
approach?
– What best practices and capabilities from industry are applicable to T&E?
– What best practices and capabilities from the Intelligence Community are
applicable to T&E?
• Technology:
– What costs more overall: processing power or network bandwidth?
– More efficient / effective for T&E: large-scale crawl and index (eg. hadoop) or
targeted domain-specific intelligence?
– How can we leverage existing infrastructure investments in supercomputing
and knowledge management?
• Relationships:
– What are the “touch points” for T&E knowledge across acquisition?
– How to we promote trust and knowledge sharing with and among Industry?
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT A
Reference Number 15-S-2335; 08-14-2015
Joint Strike Fighter Knowledge Management (JSF-KM) Effort Overview
• TRMC is investigating tools, techniques, resources, policies &
procedures needed to more efficiently & effectively use T&E data
• TRMC is partnering with Joint Strike Fighter (JSF) to ascertain
how Big Data tools & Cloud Computing technologies could assist
acquisition programs during T&E
• TRMC will establish a next-generation Knowledge Management
(KM) capability that utilizes the latest in virtualization
technologies, methodologies, & best practices for JSF OT data
• Capability will enable remote search & retrieval of JSF OT&E data
17
JSF JPO is assisting TRMC in prototyping
a Big Data / Enterprise KM system