Bug Injection in SATE VI

Aurelien DelaitreLead, SATE VI Classic TrackPrometheus Computing LLC

SATE VI Workshop - September 19, 2019 - MITRE, McLean VA

https://samate.nist.gov/SATE.html

Why Bug Injection?

Relevance

GroundTruth

StatisticalSignificance

2

Why Bug Injection?

Relevance

GroundTruth

StatisticalSignificance

ProductionSoftware

Common Vulnerabilities and Exposures (CVE)

SyntheticTest Suites

3

Why Bug Injection?

Relevance

GroundTruth

StatisticalSignificance

BugInjection

4

Ways to “Get” Bugs

● Bug Injectors● Manual & Semi-Automated Injection● Specifically Developed Test Suites● Existing Bugs

○ Discovered○ Undiscovered

5

Bug Types in SATE VI

C: Undefined Behavior

● Pointers● Buffers● Initialization

Java: Code Injection

● Cross-Site Scripting (XSS)● SQL Injection

▶ High-Impact▶ Easy to Prove

6

Proof of Vulnerability (PoV)

Why?

● Proves Bug Matters● Retrieve Bug Trace

How?

● Fuzzing● Bug Tracker● Manual

7

Bug Traces

● Based on PoVs○ C: GDB / Valgrind / ASAN○ Java: Flow

● Manual Analysis○ Doc Review○ Code Review

8

What Went Wrong?

9

Cheap but Hard Bugs

10

packet-arp.c

▶ Almost Never Found by Tools

Asymmetrical Bug/Fix Pairs

11

SimplePageBean.java

Buggy

Fixed

Buggy Bugs

▶ Implementation-dependent▶ Unknown Sink

▶ Tainted Data Questionable▶ Unintended Bug Type

fts3_write.c

global.c

12

fts3_write.c

Buggy Fixes

▶ Tainted Data Questionable▶ Condition Always False

global.c

13

pragma.c

Buggy Bugs

▶ Tainted Data Questionable▶ Condition Always True

global.c

14

pragma.c

Buggy Fixes

▶ Tainted Data Questionable▶ Condition Always False

global.c

15

fts3_tokenize_vtab.c

Buggy Fixes

▶ Tainted Data Questionable▶ Condition Always False Due to Programming Error

global.c

16

Sink Separationdate.c insert.c

17

Shadowing

18

Shadowing

19

Shadowing

20

Take Away

21

AutomatedBug Injection Curation Test Suites

StrongerBetterFaster