Farewell Mr Crosse! Things have changed since Mr. Crosse first started teaching…
Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies...
Transcript of Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies...
![Page 1: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/1.jpg)
Have Things Changed Now?- An Empirical Study of Bug Characteristics in Modern Open
Source Software
Zhenmin Li, Lin Tan, Xuanhui Wang, Shan LuYuanyuan Zhou and Chengxiang Zhai
University of Illinois, Urbana-Champaign
![Page 2: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/2.jpg)
Motivation
Bugs account for 40% of system failures.
Designing debugging tools requires a good understanding of bug characteristics.
Many empirical studies have been performed.
Why another empirical study?
2
![Page 3: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/3.jpg)
Things have changed
Many empirical studies have been performed over 10 years ago
3
![Page 4: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/4.jpg)
Things have changed
Many empirical studies have been performed over 10 years ago
More effective modern debugging tools - Valgrind, Purify, Coverity, etc
3
![Page 5: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/5.jpg)
Things have changed
Many empirical studies have been performed over 10 years ago
More effective modern debugging tools - Valgrind, Purify, Coverity, etc
Fewer memory bugs in release code?
3
![Page 6: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/6.jpg)
Things have changed
Many empirical studies have been performed over 10 years ago
More effective modern debugging tools - Valgrind, Purify, Coverity, etc
Fewer memory bugs in release code?
Rising security concerns
3
![Page 7: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/7.jpg)
Things have changed
Many empirical studies have been performed over 10 years ago
More effective modern debugging tools - Valgrind, Purify, Coverity, etc
Fewer memory bugs in release code?
Rising security concerns
More security-related bugs?
3
![Page 8: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/8.jpg)
Things have changed Emphasis on user friendly interfaces
4
![Page 9: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/9.jpg)
Things have changed Emphasis on user friendly interfaces
Have GUI bugs become more pervasive?
4
![Page 10: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/10.jpg)
Things have changed Emphasis on user friendly interfaces
Have GUI bugs become more pervasive?
Software architecture shift - Multithreaded/Multiprocessed
4
![Page 11: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/11.jpg)
Things have changed Emphasis on user friendly interfaces
Have GUI bugs become more pervasive?
Software architecture shift - Multithreaded/Multiprocessed
More concurrency bugs?
4
![Page 12: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/12.jpg)
Things have changed Emphasis on user friendly interfaces
Have GUI bugs become more pervasive?
Software architecture shift - Multithreaded/Multiprocessed
More concurrency bugs?
Need a new empirical study to answer these questions
4
![Page 13: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/13.jpg)
Our Contribution Studied two large and popular OSS projects
5
Mozilla Apache
![Page 14: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/14.jpg)
Our Contribution Studied two large and popular OSS projects
5
Mozilla ApacheProject Type Client Server
![Page 15: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/15.jpg)
Our Contribution Studied two large and popular OSS projects
5
Mozilla ApacheProject Type Client Server
Major Language C & C++ C
![Page 16: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/16.jpg)
Our Contribution Studied two large and popular OSS projects
5
Mozilla ApacheProject Type Client Server
Major Language C & C++ CCode Size (MLOC) 4 0.3
![Page 17: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/17.jpg)
Our Contribution Studied two large and popular OSS projects
5
Mozilla ApacheProject Type Client Server
Major Language C & C++ CCode Size (MLOC) 4 0.3
No. of Releases ~90 ~90
![Page 18: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/18.jpg)
Our Contribution Studied two large and popular OSS projects
5
Mozilla ApacheProject Type Client Server
Major Language C & C++ CCode Size (MLOC) 4 0.3
No. of Releases ~90 ~90Bug DB start time 1998 2001
![Page 19: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/19.jpg)
Our Contribution Studied two large and popular OSS projects
5
Mozilla ApacheProject Type Client Server
Major Language C & C++ CCode Size (MLOC) 4 0.3
No. of Releases ~90 ~90Bug DB start time 1998 2001
Manually collected and classified 709 bugs
![Page 20: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/20.jpg)
Our Contribution Studied two large and popular OSS projects
5
Mozilla ApacheProject Type Client Server
Major Language C & C++ CCode Size (MLOC) 4 0.3
No. of Releases ~90 ~90Bug DB start time 1998 2001
Manually collected and classified 709 bugs
Automatically classified around 29,000 bugs
![Page 21: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/21.jpg)
Our Bug SourcesMozilla Apache
6
![Page 22: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/22.jpg)
Our Bug SourcesMozilla Apache
GeneralRandomly sampled
548 209
6
![Page 23: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/23.jpg)
Our Bug SourcesMozilla Apache
GeneralRandomly sampled
548 209
After filtering 264 98
6
After filtering: only fixed run-time bugs with known root causes
![Page 24: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/24.jpg)
Our Bug SourcesMozilla Apache
GeneralRandomly sampled
548 209
After filtering 264 98
Security All 193 64
6
After filtering: only fixed run-time bugs with known root causes
![Page 25: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/25.jpg)
Our Bug SourcesMozilla Apache
GeneralRandomly sampled
548 209
After filtering 264 98
Security All 193 64
6
After filtering: only fixed run-time bugs with known root causes
We use information retrieval techniques to collect concurrency bugs and manually classify 90 of them.
![Page 26: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/26.jpg)
Our Bug SourcesMozilla Apache
GeneralRandomly sampled
548 209
After filtering 264 98
Security All 193 64
6
After filtering: only fixed run-time bugs with known root causes
Total=709We use information retrieval techniques to collect concurrency bugs and manually classify 90 of them.
![Page 27: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/27.jpg)
Classification Dimensions
Root CauseC
ompo
nent
Impact
7
![Page 28: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/28.jpg)
Classification Dimensions
Root CauseC
ompo
nent
Impact
7 Every bug is classified in three dimensions.
![Page 29: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/29.jpg)
Classification Dimensions
Root CauseC
ompo
nent
Impact
MemoryConcurrency
Semantic
7 Every bug is classified in three dimensions.
![Page 30: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/30.jpg)
Classification Dimensions
Root CauseC
ompo
nent
Impact
MemoryConcurrency
Semantic
Performance DegradationIncorrect Functionality
Data CorruptionCrashHang
7 Every bug is classified in three dimensions.
![Page 31: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/31.jpg)
Classification Dimensions
Root CauseC
ompo
nent
Impact
MemoryConcurrency
Semantic
CoreGUI
NetworkI/O
Performance DegradationIncorrect Functionality
Data CorruptionCrashHang
7 Every bug is classified in three dimensions.
![Page 32: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/32.jpg)
Major FindingsConcurrency
Semantic
Memory
8
1992 [Sullivan92]
![Page 33: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/33.jpg)
Major FindingsConcurrency
Semantic
Memory
Memory
8
Memory bugs are decreasing.
1992 [Sullivan92] 2005
![Page 34: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/34.jpg)
Major FindingsConcurrency
Semantic
Memory
Semantic
Memory
8
Memory bugs are decreasing.
Semantic bugs are becoming increasingly dominant.
1992 [Sullivan92] 2005
![Page 35: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/35.jpg)
Major FindingsConcurrency
Semantic
Memory
Concurrency
Semantic
Memory
8
Memory bugs are decreasing.
Semantic bugs are becoming increasingly dominant.
A few concurrency bugs probably because of underreporting.
1992 [Sullivan92] 2005
![Page 36: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/36.jpg)
Impact
9
Crash
FuncH
ang
Corru
ptPer
f
Oth
ers
unknown
0%
25%
50%
75%Memory Concurrency Semantic
Apache
Results are similar for both
Mozilla and Apache
![Page 37: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/37.jpg)
Impact
57.1% of crashes are caused by memory bugs.
9
Crash
FuncH
ang
Corru
ptPer
f
Oth
ers
unknown
0%
25%
50%
75%Memory Concurrency Semantic
Apache
14.3%
Results are similar for both
Mozilla and Apache
![Page 38: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/38.jpg)
Impact
57.1% of crashes are caused by memory bugs.
9
Crash
FuncH
ang
Corru
ptPer
f
Oth
ers
unknown
0%
25%
50%
75%Memory Concurrency Semantic
Apache
69.4%
14.3%
Results are similar for both
Mozilla and Apache
![Page 39: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/39.jpg)
Impact
57.1% of crashes are caused by memory bugs.
Incorrect functionality dominates.
9
Crash
FuncH
ang
Corru
ptPer
f
Oth
ers
unknown
0%
25%
50%
75%Memory Concurrency Semantic
Apache
69.4%
14.3%
Results are similar for both
Mozilla and Apache
Compare at 35%
![Page 40: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/40.jpg)
Impact
57.1% of crashes are caused by memory bugs.
Incorrect functionality dominates.
9
Crash
FuncH
ang
Corru
ptPer
f
Oth
ers
unknown
0%
25%
50%
75%Memory Concurrency Semantic
Apache
69.4%
14.3%4.1% 1.0% 6.1% 1.0% 4.1%
Results are similar for both
Mozilla and Apache
Compare at 35%
![Page 41: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/41.jpg)
Impact
57.1% of crashes are caused by memory bugs.
Incorrect functionality dominates.
9
Crash
FuncH
ang
Corru
ptPer
f
Oth
ers
unknown
0%
25%
50%
75%Memory Concurrency Semantic
Apache
69.4%
14.3%4.1% 1.0% 6.1% 1.0% 4.1%
Results are similar for both
Mozilla and Apache
Compare at 35%
See our paper for numbers
with error range with 95%
confidence level
![Page 42: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/42.jpg)
Component
10
Mozilla ApacheCore GUI Net I/O Others Core GUI Net I/O Others
0%
20%
40%
60%
80%
![Page 43: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/43.jpg)
Component
10
Mozilla ApacheCore GUI Net I/O Others Core GUI Net I/O Others
0%
20%
40%
60%
80%
3.0%2.3%1.5%
52.7%
40.5%
![Page 44: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/44.jpg)
Component
10
Mozilla ApacheCore GUI Net I/O Others Core GUI Net I/O Others
0%
20%
40%
60%
80%
3.0%2.3%1.5%
52.7%
40.5%
1.0%
12.2%8.2%2.0%
76.5%
![Page 45: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/45.jpg)
Component
Client and server software have different bug characteristics.
10
Mozilla ApacheCore GUI Net I/O Others Core GUI Net I/O Others
0%
20%
40%
60%
80%
3.0%2.3%1.5%
52.7%
40.5%
1.0%
12.2%8.2%2.0%
76.5%
![Page 46: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/46.jpg)
Security Bugs -Trend
11
0
20
40
60
80
100
99 00 01 02 03 04 05
Mozilla Apache
Number of bugs
![Page 47: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/47.jpg)
Security Bugs -Trend
11
0
20
40
60
80
100
99 00 01 02 03 04 05
Mozilla Apache
Number of bugs
Only contain half a year’s data
![Page 48: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/48.jpg)
Security Bugs -Trend
11
0
20
40
60
80
100
99 00 01 02 03 04 05
Mozilla Apache
Number of bugs
99 00 01 02 03 04 050%
2%
4%
6%
8%
10%
12%
Mozilla Apache
Normalized Percentage
Only contain half a year’s data
![Page 49: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/49.jpg)
Security Bugs -Trend
11
0
20
40
60
80
100
99 00 01 02 03 04 05
Mozilla Apache
Number of bugs
99 00 01 02 03 04 050%
2%
4%
6%
8%
10%
12%
Mozilla Apache
Normalized Percentage
Only contain half a year’s data
![Page 50: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/50.jpg)
Security Bugs -Trend
11
0
20
40
60
80
100
99 00 01 02 03 04 05
Mozilla Apache
Number of bugs
99 00 01 02 03 04 050%
2%
4%
6%
8%
10%
12%
Mozilla Apache
Normalized Percentage
Only contain half a year’s data
![Page 51: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/51.jpg)
Security Bugs -Trend
11
Security bugs are increasing.
0
20
40
60
80
100
99 00 01 02 03 04 05
Mozilla Apache
Number of bugs
99 00 01 02 03 04 050%
2%
4%
6%
8%
10%
12%
Mozilla Apache
Normalized Percentage
Only contain half a year’s data
![Page 52: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/52.jpg)
Security Bugs
12
0%
25%
50%
75%
ApacheMem Sem Con Others
9.4%1.6%
71.9%
17.2%
Results are similar for both
Mozilla and Apache
![Page 53: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/53.jpg)
Security Bugs
AGAINST the belief that buffer overflows are the most common form of security vulnerabilities
12
0%
25%
50%
75%
ApacheMem Sem Con Others
9.4%1.6%
71.9%
17.2%
Results are similar for both
Mozilla and Apache
![Page 54: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/54.jpg)
Automatic Classification Do the previous distributions hold for all bugs?
Do we see the same trend across software evolution?
13
![Page 55: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/55.jpg)
Automatic Classification Do the previous distributions hold for all bugs?
Do we see the same trend across software evolution?
Require more bugs or all of the reported bugs, which could be around 29,000
13
![Page 56: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/56.jpg)
Automatic Classification Do the previous distributions hold for all bugs?
Do we see the same trend across software evolution?
Require more bugs or all of the reported bugs, which could be around 29,000
Can NOT be done MANUALLY
13
![Page 57: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/57.jpg)
Automatic Classification Do the previous distributions hold for all bugs?
Do we see the same trend across software evolution?
Require more bugs or all of the reported bugs, which could be around 29,000
Can NOT be done MANUALLY
Use machine learning techniques to automatically classify all 29,000 or so fixed run-time bugs in Mozilla (by June 2005)
13
![Page 58: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/58.jpg)
Auto Classification Results Automatic classification confirmed the distribution results
of root causes and impacts (see our paper).
14
![Page 59: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/59.jpg)
Auto Classification Results Automatic classification confirmed the distribution results
of root causes and impacts (see our paper).
14
99 00 01 02 03 04 050%
20%
40%
60%
80%
100%
Semantic Memory
Mozilla
![Page 60: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/60.jpg)
Auto Classification Results Automatic classification confirmed the distribution results
of root causes and impacts (see our paper).
14
99 00 01 02 03 04 050%
20%
40%
60%
80%
100%
Semantic Memory
Mozilla
![Page 61: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/61.jpg)
Auto Classification Results Automatic classification confirmed the distribution results
of root causes and impacts (see our paper).
14
Semantic bugs increase over time
99 00 01 02 03 04 050%
20%
40%
60%
80%
100%
Semantic Memory
Mozilla
![Page 62: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/62.jpg)
Auto Classification Results Automatic classification confirmed the distribution results
of root causes and impacts (see our paper).
14
Semantic bugs increase over time
99 00 01 02 03 04 050%
20%
40%
60%
80%
100%
Semantic Memory
Mozilla
![Page 63: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/63.jpg)
Auto Classification Results Automatic classification confirmed the distribution results
of root causes and impacts (see our paper).
14
Semantic bugs increase over time
Memory bugs decrease over time
99 00 01 02 03 04 050%
20%
40%
60%
80%
100%
Semantic Memory
Mozilla
![Page 64: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/64.jpg)
Conclusions Manually studied 709 bugs and automatically
classified 29,000 bugs
15
![Page 65: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/65.jpg)
Conclusions Manually studied 709 bugs and automatically
classified 29,000 bugs
Memory bugs in release code decreased.
Detection tools have a positive impact.
15
![Page 66: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/66.jpg)
Conclusions Manually studied 709 bugs and automatically
classified 29,000 bugs
Memory bugs in release code decreased.
Detection tools have a positive impact.
Semantic bugs are a major source of bugs, accounting for 81.1-86.7% of all classified bugs.
More effort should be applied to detecting and fixing semantic bugs.
15
![Page 67: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/67.jpg)
Conclusions Most security vulnerabilities are caused by
semantic bugs
16
![Page 68: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/68.jpg)
Conclusions Most security vulnerabilities are caused by
semantic bugs
Client and server have different bug characteristics.
Need different testing support
16
![Page 69: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/69.jpg)
Conclusions Most security vulnerabilities are caused by
semantic bugs
Client and server have different bug characteristics.
Need different testing support
Concurrency bugs are hard to reproduce.
Need tools, e.g. FDR and BugNet, to help replay
16
![Page 70: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/70.jpg)
Conclusions Most security vulnerabilities are caused by
semantic bugs
Client and server have different bug characteristics.
Need different testing support
Concurrency bugs are hard to reproduce.
Need tools, e.g. FDR and BugNet, to help replay
Correlation results and minor findings are shown in our paper.
16
![Page 71: Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,](https://reader031.fdocuments.net/reader031/viewer/2022011906/5f3b47757b2e1027477b6fea/html5/thumbnails/71.jpg)
Questions? Thank you!