Ac2017 8. metrics forprivacysafety-slides

Thomas Fehlmann, Eberhard KranichEuro Project Office, Zürich & Duisburg

NESMA Autumn Meeting, Soestduinen, 7th November 2017

Speaker & Authors

Agenda

Test Metrics

Privacy Metrics

Safety Metrics

Metrics in Contracts

| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

Agenda

Test Metrics

Privacy Metrics

Safety Metrics


Test Metrics


Bad Mathematics in Software Testing

▪ What is the Size of Software?

▪ Lines of (undocumented?) Code?

▪ What is a Software Defect?

▪ An entry in a bug tracking system??

▪ What is a Defect Density?

▪ Number of bug entries per line of code in a bug tracking repository????

What Defect Density has this

Google Car’s Software?

Compared toNissan’s Software?


Testing refers to Functionality not Code

▪ Code is not the object of testing of software or systems

▪ Test metrics refer to functionality

▪ Test metrics cannot refer to code

▪ Most code is open source

▪ Well code-tested by Daily Builds & JUnit’s Green Bar


123456789101112131415161718

#include "mbed.h"

DigitalIn button1( SW2 ); // Right Button on ARM BoardDigitalOut led( LED1 );

int main(){led = 1; // red (RGB LED use inverse logic

// 1 = OFF, 0 = ON)

while ( true ) {if ( button1 == 0 ) // Button pressedled = 0;elseled = 1;}}

Software Models

▪ Modeling Software by

▪ ISO/IEC 20926 IFPUG

▪ ISO/IEC 19761 COSMIC

▪ Others….

▪ Sizing Software according

▪ Functionality & Test Intensity

▪ Non-functional Characteristics

▪ Privacy

▪ Safety

▪ Other constraints


The IFPUG Model of Software (ISO/IEC 20926)

▪ Transactions

▪ EI: Elementary Input

▪ EO: Elementary Output

▪ EQ: Elementary Query

▪ Data Elements

▪ ILF: Internal Logical Files

▪ EIF: External Logic Files

Software Boundary

ILF

User

(Person or application)

EIF

EI EO EQ


The IFPUG Model of the Employee Database

▪ Transaction Map

▪ 4 EI, 1 EO, 1 EQ

▪ 2, 12 and 13 DET

▪ 9 FTR

▪ 1 ILF, 1 EIF

▪ 3 and 12 DET

▪ For Test Size, we need sizing parts

▪ IFPUG Counting Rules are not compliant to the VIM and the GUM


T001 T003 T005 T006

T002

T004

D002 D001

BoundaryIFP=33

ILF

12 / 1

Employee Data

EIF

3 / 1

Active Directory

EI

13 / 1

Add Employee

EQ

12 / 2

View Employees

EI

13 / 1

Merge Employees

EO

12 / 2

Weekly Report

EI

13 / 1

Update Employee

EI

2 / 2

Terminate Employee

The COSMIC Model of Software (ISO/IEC 19761)

▪ Functional Processes

▪ Devices & Applications

▪ Entry and eXit

▪ Persistent Data

▪ Read and Write

▪ Data Movement

▪ Moves Data Groups

▪ Trigger triggers a functional process

Software Boundary

Functional

Process

Persistent Data

TriggerEntry

Entry

eXit

eXit

Write Read

Device User Application User


The COSMIC Model – Data Movement Map▪ Data Movement Map

▪ One object per functional process

▪ It has six Functional Processes ▪ Add Employee▪ Update Employee▪ Merge Employees▪ View Employee▪ Weekly Report▪ Terminate Employee

▪ 31 Data Movements yields 29 CFP

▪ Some data movements move identical data groups


9 Entry (E) + 11 eXit (X) + 5 Read (R) + 4 Write (W) = 29 CFP

HR User Add New Employee Update Employee Merge Employees View Employee Weekly Report Terminate Employee Employee Database Active Directory

1.// Add Employee

Add

2.// Get Employee ID

3.// Write Employee Data

4.// Show Employee ID

5.// Confirm Transaction

6.// Update Employee

Update

7.// Read Employee Data

8.// Write Employee Data

9.// Confirm Update

10.// Select Employee 1

Merge

11.// Read Data for Employee 1

12.// Select Employee 2

13.// Read Data for Employee 2

14.// Show Data for Merge

15.// Select Data to Merge

16.// Consolidate Merge

17.// Confirm Merge

18.// Request Employee Data

View

19.// Get Employee Data

20.// Read Access Rights

21.// Show Employee Data

22.// Confirm Data Retrieval

23.// End of Week

End of Week

24.// Get Data for Weekly Report

25.// Read Access Rights

26.// Weekly Report

27.// Confirm Report

28.// Terminate Employee

Terminate

29.// Block IT Access

30.// Terminate Employment

31.// Confirm Termination

Visualizing Software Testing

▪ Tester sees selected sequences in the Data Movement Map

▪ Tester can “walk” the data movements when planning or executing tests

▪ Makes functionality visible to the development team

▪ Localizes defects that impact functionality

▪ Supports communication between testers, users, and developers

Functional

Process

Other

Application

Some

Device

8.// Move some data

9.// Move some data

10.// Move some data


Other

Device


Functionality, Defect Size, and Defect Density

▪ What happens if data movements don’t work as expected, have defects instead?

▪ Testers mark and count data movements where defects have been detected

▪ One Size Metric:

▪ ISO/IEC 19761 COSMIC

Functional

Process

Other

Application

Some

Device

8.// Move some data

Move some data



Other

Device

Test Size = 4

Defect Count = 1


Functional Size Number of Data Movements needed to implement required functionality

Test Story Collection of Test Cases aiming at certain functionality

Test Size Number of Data Movements executed in Tests

Defect Count Number of Data Movements affected by some defect detected in a Test Story

Agenda

Test Metrics

Privacy Metrics

Safety Metrics



Agenda

Test Metrics

Privacy Metrics

Safety Metrics


Privacy Metrics


Current Approach in Assessing Security

▪ The Center for Internet Security (CIS) presents the CIS Controls for Effective Cyber Defense Version 6.0, a recommended set of actions that provide specific and actionable ways to stop today's most pervasive and dangerous cyber attacks

▪ The CIS Security Benchmarks program provides well-defined, un-biased and consensus-based industry best practices to help organizations assess and improve their security

▪ https://www.cisecurity.org


https://www.cisecurity.org/

Critique at Current Approaches to Security

▪ This is not software security but system security

▪ Assessment concerns a system as a whole

▪ Cyber defense must start at the component level

▪ Where is the system as a whole in the Web of Things?

▪ The Web of Things changes equally fast as requirements in agile software development

TheTraditionalApproach


How to Measure Privacy for Software?

▪ Privacy Value▪ Value = 0: No privacy. It’s public.

▪ Value = 1: Disclosure is inconvenient

▪ Value = 2: Disclosure can be harmful

▪ Value = 3: Disclosure costs money

▪ Value = 4: Disclosure makes guilty

▪ Value = 5: Disclosure threatens lives

▪ Public Exposure▪ Value = 0: No encryption. It’s public.

▪ Value = 1: Weak encryption

▪ Value = 2: Strong encryption

▪ Value = 3: Two-way encryption

▪ Value = 4: Data never leaves system

▪ Value = 5: Computer-internal data

▪ The principle for Each Model Element is

Privacy Protection = Privacy Value * Public Exposure


𝑃𝑟𝑖𝑣𝑎𝑐𝑦 𝐼𝑛𝑑𝑒𝑥 =

𝑃𝑟𝑖𝑣𝑎𝑐𝑦 ∗ 𝐸𝑥𝑝𝑜𝑠𝑢𝑟𝑒 + 1 − 1

26 − 1∗ 5

The Privacy Index

▪ The product

Privacy Value * Public Exposure

is in the range 0 to 25

▪ Privacy Index is range 0⋯5▪ Five 5 is the index for

maximum privacy

▪ Zero 0 privacy means public data

▪ The Privacy Index range is on a polynomial scale between zero and five

Zero = No Privacy

All is open

Five = Full

Privacy Protection

0....5


The Privacy Index

▪ For contract purposes, use a graphical representation


4

3

2

1

0

5

Privacy ProtectionIm

pact

on

Priv

acy

Data Movement

Low Privacy Index

Medium Privacy Index

High Privacy Index

Major Privacy Index

No Privacy Needed

1235 4 0

0.9

Index 0.9: W004 Terminate Employment

Index 1.2: X009 Confirm Report

Index 1.5: R001 Get Employee ID, +8

Index 1.8: X011 Confirm Termination, +1

Index 2.0: X007 Confirm Data Retrieval

1.5

1.21.5

4.4

2.0

Index 2.4: E001 Add Employee , +11

Index 3.2: X005 Confirm Merge

Index 4.4: R002 Read Employee Data, +12.43.2

Agenda

Test Metrics

Privacy Metrics

Safety Metrics



Agenda

Test Metrics

Privacy Metrics

Safety Metrics


Safety Metrics


Safety for Software

▪ Software impacts physical world

▪ Level 1: Low – break

▪ Level 3: Medium

▪ Level 5: High – crash

▪ Probability Levels

▪ Less likely on motorways

▪ Highly probably in villages and urban areas


Measure Safety Risk

▪ Impact through actuators

▪ Steering Wheel in Autonomous Cars

▪ Closed Door at Smart Homes

▪ Indicated by an index 1⋯5

▪ Other classical software risks remain:

▪ Causing losses for business

The principle for Each Model Element is

Safety = Probability * Impact

0....5


Zero = All Safe!

Highly unlikely

Five = Danger!

Safe Our Souls!

What means Safety for Software?

▪ Impact Levels

▪ Level 1: Low

▪ Level 3: Medium

▪ Level 5: High

▪ Probability Levels

▪ By percentages1

2

3

4

5

ProbabilityIm

pact

on

Saf

ety

Data Movement

Major Safety Risk

High Safety Risk

Medium Safety Risk

Low Safety Risk

Risk Incurred/ Feature

100%80%60%20% 40%0%

3.2

Index 3.2: E010 Terminate Employee

2.0

Index.2.0: X010 Block IT Access

Index 1.2: W004 Terminate Employment

0.6

Index 0.8: X011 Confirm Termination

Index 0.6: X008 Weekly Report, +24


0.8

1.2

Agenda

Test Metrics

Privacy Metrics

Safety Metrics



Privacy of Data Movements

▪ How private is a Data Movement?

▪ Can it be intercepted?

▪ Not really if machine-internal

▪ Quite easy if HTTPS

▪ Less easy with two-way authentication

▪ Most difficult with private key encryption

▪ ISO/IEC 19761 COSMIC provides a model for measuring privacy that is well suited for distributed systems such as an IoT Concert, Embedded Systems, or Mobile Apps

Software Boundary

Functional

Process

Persistent Data

TriggerEntry

Entry

eXit

eXit

Write Read

Device User Application User


Name Label Data Movement Sub-Process Description

1) E001 Add Employee Enter new employee data

2) R001 Get Employee ID Unique ID for employee identification

3) W001 Write Employee Data Store in database

4) X001 Show Employee ID Confirm new employee added

5) X002 Confirm Transaction Confirmation or error message

Data Movements

23) E008 End of Week Timing Signal

24) R006 Get Data for Weekly Report Select employee data that changed during past week

25) E009 Read Access Rights Get external IT access rights

26) X008 Weekly Report All data including termination records

27) X009 Confirm Report Weekly report done

28) E010 Terminate Employee Enter termination control flag

29) X010 Block IT Access Block access to IT

30) W004 Terminate Employment Update an employee record

31) X011 Confirm Termination Confirmation or error message

Add Row Ins Row Del Row Validate Extract SNAP

1.22

Effe

ct w

hen

Priv

ate

Dat

a is

Dis

clos

ed

Priv

acy

Pro

tect

ion

Leve

l

Pri

vacy

Pro

babi

lity

Impa

ct o

n

Saf

ety

Saf

ety

4: Makes guilty 2: Strong 2.4 10% 2: Little 0.2

4: Makes guilty 1: Weak 1.5 5% 2: Little 0.1






4: Makes guilty 1: Weak 1.5 5% 4: Quite 0.2

2: Harmful 3: Two-way 2.0 20% 3: Medium 0.6

1: Inconvenient 3: Two-way 1.2 10% 3: Medium 0.3

1: Inconvenient 4: Enclosed 1.5 80% 4: Quite 3.2

1: Inconvenient 5: Internal 1.8 50% 4: Quite 2.0

1: Inconvenient 2: Strong 0.9 30% 4: Quite 1.2

1: Inconvenient 5: Internal 1.8 40% 2: Little 0.8

2.4 0.2

0.9 3.2

Safety Index:

Minimum Privacy: Maximum Risk:

Privacy Index:

Assessing Privacy & Safety


AssessingImplementation

Assessingthe Model

Privacy of Data Functions & Transactions

▪ How private is Data / are Transactions?

▪ Can it be accessed?

▪ Not really if machine-internal

▪ Quite easy with password

▪ Less easy with two-way authentication

▪ Most difficult with private key encryption

▪ ISO/IEC 20926 IFPUG provides a model for measuring privacy that is well suited for web pages and transactional systems


Software Boundary

ILF

User

EIF

EI EO EQ

Assessing Privacy & Safety


Name Label Description of Elementary Data Function ILF EIF Type

1) D001 Employee Data A persistent logical entity maintained by the application 1) ILF

2) D002 Active Directory The active directory provides the unique username and the account list where he or she has access 1) EIF

1

EIF: 1

Elementary Data Functions

ILF:Add Row Ins Row Del Row Validate Extract SNAP

1.22

Effe

ct w

hen

Priv

ate

Dat

a is

Dis

clos

ed

Exp

osur

e to

Priv

acy

Vio

latio

n

Pri

vacy

Pro

babi

lity

Impa

ct o

n

Saf

ety

Saf

ety

4: Makes guilty 2: Strong 2.4

3: Costs money 3: Two-way 2.6 5% 1: Low 0.1

SNAP 2.4 0.2

0.9 3.2

Privacy Index: Safety Index:

Minimum Privacy: Maximum Risk:

Name Label Description of Elementary Transaction Type1) T001 Add Employee Primary intent is to maintain data in the ILF EI

2) T002 View Employees Primary intent is to present data to the user, data is retrieved both from an ILF and an EIF, and there are none of the following involved: calculations, derived data, ILF update or alteration of system behavior EQ

3) T003 Merge Employees Primary intent is to maintain data in the ILF EI

4) T004 Weekly Report Show total employment status with liabilities by the salaries EO

5) T005 Update Employee Primary intent is to maintain data in the ILF EI

6) T006 Terminate Employee Stops access to the IT and terminates employment EI

EI: 4

EO: 1

EQ: 1

Elementary Transactions

Add Row Ins Row Del Row More FTR Validate Extract SNAP

1.22

Effe

ct w

hen

Priv

ate

Dat

a is

Dis

clos

ed

Exp

osur

e to

Priv

acy

Vio

latio

n

Pri

vacy

Pro

babi

lity

Impa

ct o

n

Saf

ety

Saf

ety


4: Makes guilty 4: Enclosed 3.8 10% 3: Medium 0.3

4: Makes guilty 2: Strong 2.4 20% 1: Low 0.2

4: Makes guilty 3: Two-way 3.2 10% 4: Quite 0.4


1: Inconvenient 2: Strong 0.9 80% 4: Quite 3.2

2.4 0.2

0.9 3.2

21

Maximum Risk:

Privacy Index:

Minimum Privacy:

Safety Index:

Metrics for Software Contracts

▪ Functional Size

▪ COSMIC

▪ IFPUG, NESMA

▪ …

▪ Test Size

▪ Test Intensity

▪ Acceptable Defect Density

▪ Privacy Index & Minimum Privacy

▪ For Data Movements

▪ For Data Functions & Transactions

▪ Security Index & Maximum Risk




Metrics for System Contracts

▪ Hardware Vulnerability

▪ Software might destroy itself when Hardware is compromised

▪ Functional Size

▪ COSMIC

▪ IFPUG, NESMA

▪ Test Size

▪ Test Intensity

▪ Acceptable Defect Density

▪ Privacy Index & Minimum Privacy



▪ Security Index & Maximum Risk




Conclusions

▪ Contracts for safety-critical software must specify how well we protect privacy and how safe the software behaves

▪ Based on a functional size model

▪ Contracts without specifying test metrics are dangerous and bound to later arbitration

▪ Currently, test metrics work with ISO/IEC 19761 COSMIC only

▪ Make ISO/IEC 20926 & 24570 compliant with the VIM and the GUM

▪ Consumers need such test metrics, privacy and safety metrics when purchasing software or autonomous systems


Logos Press

Berlin 2016

Questions?


http://www.logos-verlag.de/cgi-bin/buch?isbn=4406


New Book on Six Sigma Transfer Functions

http://www.logos-verlag.de/cgi-bin/buch?isbn=4406

Ac2017 8. metrics forprivacysafety-slides

Software

Transcript of Ac2017 8. metrics forprivacysafety-slides