Quality Factors for AI

Dr. Diana Alina Serbanescu

Team Lead of Research Group: Criticality of AI-based Systems

Weizenbaum-Institut für die vernetzte Gesellschaft

Criticality of AI-Based Systems

• Transportation

• Energy distribution

• Health

• Decision / Expert Systems

• Finance / Banking

• Laws and regulations

• etc. (everywhere)

“AI technologies already pervade our lives. As they become a central force in society, the field is shifting from simply building systems that are intelligent to building intelligent systems that are human-aware and trustworthy.”

One Hundred Year Study on Artificial Intelligence AI100https://ai100.stanford.edu/sites/default/files/ai_100_report_0906fnlc_single.pdf

TS

SUT

SUT

Pattern

Timer

Stimuli (IN)

Reactions (OUT)

[tMAX]

TS – Test System SUT – System under Test

Timer T;TS.send(Stimuli(IN));T.start(t_max);if(pattern == Reactions(OUT) && timer>0)

test_verdict := PASS;else

test_verdict := FAIL;

J. Grossmann, D. A. Serbanescu, and I. K. Schieferdecker, “Testing Embedded Real Time Systems with TTCN-3,” in Second International Conference on Software Testing Verification and Validation, ICST 2009, Denver, Colorado, USA, April 1-4, 2009, 2009, pp. 81–90.

Electronic Control Unit (ECU) implemented in the car-door

“control signal”

“feedback”

Real-time tester

Quality Assurance of Real-Time SystemsBackground

Machine Learning

Human Centricity

Embodiment

Usability

Trustworthiness Transparency

Bias

Autonomous Systems

Behavior

Emotion

Design

Human-machine interactionRobots

Emergence

Requirements

Safety and SecurityPrivacy

Data

Understandability

Specification

Quality Context for AI

Normative meaning of bias - algorithm results that are systematically prejudiced due to erroneous assumptions.

Bias — A Reason for Failure

Examples ofAI gone wrong

• Image search engines such as Google may show biased results

• For example: Gender bias in occupations like CEO [Kay 2015]

Examples of AI gone wrong• Google Image classification

rolled out with Google Photos may classify people of colour as gorillas

• The problem remains –Google has disabled the target class gorilla for now

Examples ofAI gone wrong

• The Compass software predicts future criminals (recidivism model) and it has been shown to be biased against people of color [Angwin 2016]

• Note: we don't know what's inside this model – its proprietary. Most likely its not very sophisticated or doesn’t count as AI

− Training datasets may contain historical data, or they may be collected from populations that do not represent diversity

− Data used to train AI can contain implicit racial, gender, or ideological biases. What is fairness?

− Biases are reflecting the system designs which are used to make decisions in various branches, from governments to businesses

− Human factor - exclusive view of the world (power), lack of training

− Can undermine trust between human and machine

− Understand, detect, prevent

Bias — Sources

Worth reading• Artificial Intelligence's White Guy

Problem, Kate Crawford• Human Decisions and Machine

Predictions, Kleinberg, Jon• AI 100 Report

More examples of AI gone wrong

• Several Tesla crashes have occurred recently• In most cases the autopilot did not detect a stationary or slow moving object• This is a known limitation/bug of the autopilot as stated in the manual: „Traffic-Aware

Cruise Control cannot detect all objects and may not brake/decelerate for stationary vehicles, especially in situations when you are driving over 50 mph (80 km/h) and a vehicle you are following moves out of your driving path and a stationary vehicle or object is in front of you instead.“

• As for now Tesla was not able to fix the bug

In Contrast: the Vision of ExAi

• Systems need to be optimized not only for task performance but also for auxiliary criteria such as safety or fairness [Doshi-Velez 2017]

• These criteria a difficult to optimize for (e.g. more data and better models may not help with biased data) and are difficult to quantify (e.g. fairness)[Doshi-Velez 2017]

• Explainability or interpretability have been suggested as a fall-back criteria

• Definition Explanation: is a giving reason or justification for an action or belief [Preece2018]

• „if the system can explain its reasoning, we then can verify whether that reasoning issound with respect to these auxiliary criteria.“[Doshi-Velez 2017]

A New Requirement for Quality Engineering:Explainability

• Example #1: Our auxiliary criteria is, that we want the system to be right for the right reasons

• Note: Systems might pick up any correlation in the data (correlation ≠ causation)

• Saliency/LRP Methods can provide insights what was important for classification

A New Requirement for Quality Engineering:Explainability

• Example #2: Our auxiliary criteria is, that we want the user to develop appropriate trust in the system. He should know when to trust the system (rather than distrust it) and when to distrust it (rather than blindly trusting it) [Lee 2004]

• Left: LRP based methods show what features speak for an against a prediction

• Right: Uncertainty visualization help users to make better decisions [Fernandes2018, Joslyn 2012]

A New Requirement for Quality Engineering:Explainability

• "Classical" AI Systems that where mostly algorithmically designed before the AI winter often had interpretability as a built in feature [Preece 2018] e.g. Rule Extraction for Artificial Neural Network

• Todays more heuristically and data driven approaches often lack build-in interpretability. [Preece 2018].

• This is especially true for deep learning systems with their sub-symbolic nature [Preece 2018].

• Post-hoc explanations

Explanable AIA (Re—)emerging Discipline

How to employ AI for social good?

Politics and regulations

Sustainable Development Goals

Before solutions we need knowledge

Define our systems of values:

- not necessarily who we are but who we want to become?

Forums of discussions and open debate with all stakeholders

Quality of AI - Technology for Social Good

• Experimentation Lab “AI for Social Good”:Platform for Human – AI Interactions

• Education• Workshops• Presentations • Experiments• Performances

Strategy

“A computer will do what you tell it to do, but that may be much different from what you had in mind.”

Joseph Weizenbaum.

Experimentation Lab AI for Social Good

Questions?

Thank You!