Assured autonomy & building trust into AI systems

John J. Forte

John.Forte@jhuapl.edu

A discussion on how T&E will play a role in driving assurance into tomorrow systems

2March 28, 2019

Some context on autonomy & building AI capabilities

3March 28, 2019https://www.nsf.gov/attachments/137968/public/nsf_v05.pdf

Importance of trust in enabling AI capabilities

4

Pattern Recognition

Knowledge Representation & Reasoning

Self-Evaluation & Self-Guided Learning

Creativity & Innovation

Perceive • Decide • Act • Team

TRUST

Stair Steps to Capability Realization

March 28, 2019

Importance of trust in enabling AI capabilities

5

Countering Disruptions to Capability Realization

TRUSTTRUST

Pattern Recognition

Knowledge Representation & Reasoning

Self-Evaluation & Self-Guided Learning

Creativity & Innovation

Perceive • Decide • Act • Team

Counter –Pattern Recognition

Counter –Knowledge Representation & Reasoning

Counter –Self-Evaluation & Self-Guided Learning

Counter –Creativity & Innovation

March 28, 2019

Trust through vulnerability assessments and adversarial AI

6

• Leverage APL vehicle laboratory• Assess Tesla auto pilot system• Create and test adversarial techniques for

ML

March 28, 2019

GROUND

Trust through test & evaluation and formal methods

7

ACAS XA

ACAS XU

TCAS II Non-cooperative

• Leverage APL research in national airspace collision avoidance

• Drive autonomy into legacy architectures• Validate future sensing and collision

avoidance approaches• Determine the scalability of formal methods

within an increasingly autonomous airspace

Airbus Blueprint for the Sky [2018]

March 28, 2019

AIR

Trust through test & evaluation and simulation

8

Transmission Area

No-Go Area

Mission Area1

Mission Area2

Mission Area3

Keyport

Barriers

Waypoint 2

Recovery Point

Launch PointWaypoint 1

Obstacle 1

Obstacle 2

Obstacle 3Obstacle 4

• Leverage APL research in autonomous unmanned underwater vehicles

• Developed the Range Adversarial Planning Tool• Set pass/fail boundaries for autonomous

performance• Investigate exploits and vulnerabilities in

autonomous operations

March 28, 2019

SEA

With more reliance on AI-enabled capabilities and autonomous agents, what will trust look like in the cyber domain?

9March 28, 2019

Design Open, Resilient & Zero Trust Architectures

Embrace Automation & AI-enabled capabilities

Increase Response Speed

Balance the Silicon-Carbon Ratio

Become more data driven

Driving trust and assurance in tomorrow’s highly autonomous world – T&E challenges and insights

10

Future ecosystems will be complex systems of systems environments with physical- and cyber-based autonomous agents… just a few factors for T&E to consider may include:

March 28, 2019

• Evaluating continuous internal and external updates to network infrastructure for assurance

• Alignment and agility of operational, business and safety requirements (e.g. ops suitability versus safety tradeoff analysis)

• Cyber-physical vulnerability and performance weakness analyses will need to expand

• Validating ‘machine reasoning’ congruent with human intent… learning and leveraging the field of ‘explainable AI’

• Getting simulations/models to run faster to support real-time and even predictive analysis (in simulation and the field)

Challenge:Enable the safety, trust and security of the convergence of the Internet of Things (IoT), Artificial Intelligence (AI), Machine Learning (ML) and robotics within real-world environments

Approach:- Leverage the world class cybersecurity, AI and autonomous systems research and analysis

capabilities at JHU to rapidly surge against this challenge- Establish an ecosystem across greater academia, working with government and industry

partnerships to make arguments in assurance

JHU Institute for Assured Autonomy

11

Assure the future autonomous world

Fundamental & Applied Research

Partnerships & Collaboration

Education & Workforce Development

Translation & Entrepreneurship

IAAIAA Functions

March 28, 2019

JHU IAA Research Areas

12

Assured Transportation Research Area: Resiliency for highly unsupervised autonomous transportation through all potential conditions of faults, attacks and failures Autonomous Courier Testbed

Assured Public Safety & Security Research Area: Assured intelligent and connected sensing and monitoring systems for the safety and security of citizens as well as critical campus (city) infrastructure

Autonomous Safety & Emergency Response Systems Testbed

Assured Health Systems Research Area: Trusted autonomous medical systems and human-machine teaming methodologies for efficient, reliable and private health services

• Adversarial AI & Cyber Red Teaming• Assurance Assessments & T&E• Software Assurance & Formal Methods• Social and Economic Impact Analysis

• Fall-Back & Fail-Safe Modes• Information Integrity & Privacy• Human Machine Teaming & Goal

AlignmentIntelligent Autonomous Health

Technology Testbed

Cross-Cutting Fundamental Research:

March 28, 2019

Many aspects can undercut the advantages of a future AI-enabled world

Trust, which includes elements of ethics, safety, privacy and performance, is a major factor for the adoption and scalability of autonomous and AI-enabled capabilities

Final thoughts

13

As leadership across government, academia and industry collide in the space of intelligent and autonomous systems, how can we work together in building and codifying trust and assurance as broadly accepted principles of AI system design?

March 28, 2019

What new and/or expanded thinking will we need in our T&E and V&V approaches in order to standardize best practices and drive trust and assurance?

Questions

14March 28, 2019

Detailed view of IAA functions

15

• Adversarial AI & Red Teaming• Assurance Assessments &

T&E• Software Assurance & Formal

Methods

• Fall-Back & Fail-Safe Modes

• Information Integrity & Privacy

• Human Machine Teaming & Goal Alignment

Transportation Systems

Public Safety & Security Systems Health Systems

Fundamental & Applied Research

APL WSE

Partnerships & Collaboration

KSAS Medicine SAIS Carey Bloomberg

Academia Government Industry

• Cybersecurity• Artificial Intelligence &

Machine Learning• Computer Science• Robotics

• Formal Verification• Data Science• Cognitive Science• Communications &

Networks

Workshops & Forums

Certificate Programs

Degree & Research Support

Education & Workforce Development Translation & Entrepreneurship

IAA

Inventions, IP Protection &

Commercialization

Policy & Economic Impact Translation

Innovation Funding Opportunities

JHTV

Entrepreneurship Tracks in Degree

Programs

Industry Cooperative

Programs

Societal Impact

March 28, 2019