Factors Impacting the Effort Required to Fix Security

Vulnerabilities

Lotfi ben Othmane, Golriz Chehrazi, Eric Bodden, Petar Tsalovski, Achim Bruker,

Philip Miseldine

09 September 2015

Fixing Processes

Released software SAP security response process

Under development software Fixing process for security testing

Participants include Central security teams IMS maintenance organization Security experts Developers ….

Vulnerabilities Fixing Process at SAP

2

Introduction - The Problem

Goal: Predict the time to spend on analyzing and fixing a given vulnerability?

Let t = f (x1, ……xj)

What are x1……xj ?

3

Vulnerabilities Average fix time (min)

Dead Code (unused methods) 2.6

Poor logging: system output stream 2.9

XSS (stored) 9.6

Lack of authorization check 6.9

Unsafe threading 8.5

Null dereference 10.2

SQL injection 97.5

4

Introduction - Motivation

Cornell, RSA 2012

The only factor considered is vulnerability type What about the others?

Cost of implementing security fixes

Goal: Identify the factors that impact the fixing time

Method: Interview participants in the vulnerability fixing process

Result: The major factors that impact the fixing time

5

The Study - Scope

Interviews were conducted from 8 to 12 Dec. 2014

Number of participants 12 (12 hours)• 9 from Germany and 3 from India• Security experts, developers,

coordinators, project leaders• NetWeaver experts, custom

application experts, application

experts

6

The Study - Conduct of the Study

Transcribe the interviews

Code the interviews

Consolidate the data

Analyze the results

Preparation of the questions

Selection of participants

Interviews

Each interview is transcribed into about

16 pages

Identified 21 code classes from 3 sample

interviews

Coded each transcript in a report of 4

pages

Each interviewee is asked to review the

report of his interview

7

The Study - Conduct of the Study – Cont.

Preparation of the questions

Selection of participants

Interviews

Transcribe the interviews

Code the interviews

Consolidate the data

Analyze the results

Coding examples

-“Code injections are difficult to fix”

vulnerability type

Vulnerability characteristics

- “If the function module is the same in all these 12 or 20 releases […] , then I just have to do one correction”

Similarity of code in the different releases

Software structure

The Study - Conduct of the Study - cont.

8

Factors that Impact the Vulnerability Fix time

Factor categories # of factors Freq.

Vulnerabilities characteristics 6 9

Software structure 19 10

Technology diversification 3 5

Communication and collaboration 7 8

Availability and quality of information and documentation 9 9

Experience and knowledge 12 11

Code analysis tool 4 4

Other 4 4

9

10

Observed Fixing Process

Pre-analysis

Case 1: Analysis and

design of global solution

Case 2: Analysis and design area

solution

Case 3: Analysis and design local

solution

Test Release

Test

Release

Iterations among successive steps are performed implicitly / not marked

Implemen-tation

Implemen-tation

Vulnerability type is one among many factors (65) that impact the vulnerability fix time

The 8 factor categories reflect the main areas for improving the vulnerability fixing processes

- E.g., software structure, training, etc.

11

Take-Away

Control of the threats to the validity of the results• The interviewees are diversified• 2 researchers coded each interview and the results are consolidated • The participants validated the reports of their interviews

Weaknesses• Used one method to identify the factors—interviewing experts• Interviewed only 2 developers

External use• Diversity of product areas• Distribution of development teams

12

Threats to Validity

The main interview questions shall help the interviewees to tell their own stories

- “What” questions are inefficient to enumerate elements

The participants sometimes have their own messages to deliver

Vulnerability fixing processes are as many as the process participants

- Do not try to base the fix effort estimate on “a process”

Lessons learned

13

Thank you

lotfi.ben.othmane@sit.fraunhofer.de

14