Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software


Moritz Beller @Inventitech

https://twitter.com/Inventitech



Radjino Bholanath, Andy Zaidman




Radjino Bholanath, Andy Zaidman

Shane McIntosh


Automatic Static Analysis Tools (ASATs)

Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

RQ1: How Prevalent Are ASATs?

Image: http://www.valueinvestasia.com/wp-content/uploads/2015/03/odd-one-out.jpg

122 popular OSS projects


122


122122


122

36

122

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT

122 59% 23% -

36 77% 36% 36%



122 59% 23% -

36 77% 36% 36%

1) (Automated) investigation of repository information is an approximation of real ASAT use



122 59% 23% -

36 77% 36% 36%

1) (Automated) investigation of repository information is an approximation of real ASAT use

2) We cannot infer how a project uses ASATs from a repository analysis alone


RQ2: How are ASATs configured?


checkstyle.xml


filename checkstyle.xml



parse r

ules



enable

parse r

ules



enable

re-configure

parse r

ules



enable

re-configure

parse r

ules

custom

General Defect Classification (GDC)


RQ1: 9


RQ1: 9

Checkstyle, FindBugs, PMD, ESLint, JSCS, JSHint, JSL, PYLint, RuboCop


1,825

RQ1: 9

Checkstyle, FindBugs, PMD, ESLint, JSCS, JSHint, JSL, PYLint, RuboCop

168,425


RQ2.1: How Popular Are Certain ASATs?

Tool Language Configuration Files

Checkstyle Java 18,785

FindBugs Java 2,090

PMD Java 7,458

ESLint JavaScript 4,435

JSCS JavaScript 11,677

JSHint JavaScript 108,770

JSL JavaScript 862

Pylint Python 4,071

RuboCop Ruby 10,066

Total - 168,405

RQ 2.2: Which Rules Do Developers Enable?

65%


35% 65%


35% 65%

ASATs perform poorly at finding functional defects. Wagner et al.



0%

10000000%

20000000%

30000000%

40000000%

50000000%

60000000%

70000000%

80000000%

90000000%

100000000%

Checkstyle ESLint FindBugs JSCS JSHint JSL PMD Pylint RuboCop

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

0%

10000000%

20000000%

30000000%

40000000%

50000000%

60000000%

70000000%

80000000%

90000000%

100000000%

Checkstyle ESLint FindBugs JSCS JSHint JSL PMD Pylint RuboCop


100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%


We: This is great!

Image: Phil Skipper, The Potentialist,http://i2.wp.com/thepotentiality.com/assets/media/Kurt-Enthusiasm.jpg


ASAT Developers*: Don't care.

Image: NiBiS, http://bidab.nibis.de/PICT/traurig.jpg

RQ 2.3: How Good Is The Default?


Most ASAT configurations deviate from the default.


But, typically only have one change from the default …



But, typically only have one change from the default … ● - Addition



But, typically only have one change from the default … ● - Addition● - Deletion



But, typically only have one change from the default … ● - Addition● - Deletion● - Re-configuration/Custom analysis


RQ2: Open Questions

● Why do projects favor certain GDC rule categories from ASATs?

RQ2: Open Questions


● Can ASAT developers better fit their default configurations to their users' needs?

RQ2: Open Questions


● Can ASAT developers better fit their default configurations to their users' needs?

● Do 'dynamic' languages require more ASAT use?

RQ2: Open Questions

RQ3: How Do ASAT Configurations Evolve?

Image: Daimler AG, http://5komma6.mercedes-benz-passion.com/wp-content/uploads/2013/06/s-class-lineup.jpg

RQ 3.1: How Often Do Changes Occur?

RQ 3.1: How Often Do Changes Occur?

>80%

“never”

RQ 3.2: When Do Changes Occur?

RQ 3.2: When Do Changes Occur?

<20%

of files

RQ 3.3: How Big Are The Changes?

RQ 3: Open Questions

● Why do ASAT configurations not typically evolve?


● Why do ASAT configurations not typically evolve?

● How are ASATs used in a CI-environment?


Moritz Beller

@Inventitech



Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software

Software

Transcript of Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software