Static Analysis versus Software Model Checking for bug...
Transcript of Static Analysis versus Software Model Checking for bug...
Static Analysis versus Software Model Checking for bug finding
Dawson Englers, Madanlal MusuvathiStanford University, CA, USA
Presented By:Prateek Agarwal
ETH, Zürich
Agenda
● Goal● Aim● Scope● Methodologies used
– Meta Compilation
– C Model Checker● Case Studies● Conclusions
Goal
Checking validity of general perception
Static Analysis is easy to apply, finds shallow bugs
Model Checking is harder but strictly better
Aim
● Documentation of experience● Set of case studies● Comparison of two approaches, based on
– Bugs/Lines of Code
– False positives vs Actual Bugs
– Time & Effort required
– Types of Bugs: generic vs protocol specific● Find as many bugs as possible
Scope
● Verification of system/protocol software– Critical
– Hard to test/inspect/reason about manually
– Highly optimized● Protocol codes follow event driven model
Meta Compilation (MC)
● Extension of compiler● Used for Static Analysis & Model Checking● Metal: Language for
– Slice specification by pattern matching
– Code translation
– State machines● xg++ over g++ compiler
Meta Compilation: Key Features
● Code should be highly structured– e.g. special MACRO/functions for read/write
● No source code annotation needed● Emphasis on reducing false positives● No Simulation of code
– No model of heap, tracking variable values
– Simulation tends to become model checking
Static Analysis using MC● Patterns & Actions● Small State machines for checking
Example Rule: Sync before read...start: {WAIT_FOR_DB_FULL(addr);} ==> stop|{MISCBUS_READ_DB(addr);} ==>
{err(“Buffer read not sync");};....
Start
ErrorStop
misc_bus_read_db
wait_for_db_full
Engler & Masuvathi [1]
Model Checking using MC
● Extraction & Translation– protocol code Mur→ Φ Model
● Eliminates handcrafted system models● MurΦ
– Explicit state enumeration model checker
– Each reachable state visited once
Model Checking using MCState Variable(Metal Slicer)
Translation Pattern(Metal Printer)
CorrectnessProperties
Protocol Model
Protocol Code(Implementation)
Start StateHardware Model
Error List
xg++
MurØ
Automatically Generated
Hand Crafted by User
Program
Lie, Chou, Engler, Dill [2]
Legend:
Unedited source code
Example: Model Checking using MCMetal Slicer:.../* Patterns for n/w and proc mesgs, which use length field */pat sends =
{ NI_SEND (type, data, keep,swp, wait,nl);} | { P1_SEND(type, data, keep,swp, wait,nl);} ...Metal Printer...
/* Automatically insert length assertion before send */
{N1_SEND(type, data, keep, swap, wait, null);} ==>{
if(mgk_int_cst(data) != 0)amgk_e(“assert(nh.len = len_data);”);
elsemgk_e(“assert (nh.len = len_nodata);”);
mgk_e(“ni_send(%t, %t, procNum,nh”,type,swap);...
Lie, Chou, Engler, Dill [2]
Model Checking : C Model Checker
● Motivation– xg++ user needs to know system intricacies
– MurΦ lacks many C constructs
– Eliminate need of higher level system model● Use the implementation code itself● Process scheduling & execution by CMC
CMC Key Features● Searches for all transitions● Complete system state saved and restored● Heuristics & Optimizations for state space
explosion● Correctness properties
– Assertions from implementation code
– Memory leaks etc. in built
– User specified invariants● Environment Model still handcrafted
Case Study 1: FLASH
● Cache coherence protocol for FLASH– Multiprocessor architecture
– Code runs on each cache miss● 5 FLASH protocols● Code Size: 10K-18K
John Hennessy [3]
Results
● Static Analysis– Bugs Found: 33, False Positives: 28
– Most rules on buffer management
– e.g. allocation before use, deallocation after use, not used after deallocation
● Model Checking– 8 bugs
– e.g. queue overflows, no invalidation of an exclusive line
– Handwritten model lead to delays
Summary● Static Analysis Advantage:
– Better for same set of properties
– Works best for code visible rules● Model Checking Advantage:
– Can cover complex properties, invariants
– e.g. counter overflow missed by SA● Model Checking Disadvantage:
– Simple models Less Coverage→– Detailed model Increased Time & Effort→– Erroneous model False Positives→
Case Study 2: AODV Protocol● Ad-hoc On-demand Distance Vector Protocol● Routing protocol● Simplified environment
– Input: route request, timer interrupt, packet receipt● 3 publicly available implementations used● Average 6 KLOC● Model Checking using CMC
Results
● Model Checking– 42 unique bugs, 1 bug in AODV specification
● Static Analysis: 34 bugs
Both Model Checking Static Analysis
Generic Properties 21 1 13
Protocol Specific 0 20 0
Total 21 21 13
comparison of bug count classified by properties
Comparison
● Generic Properties:– CMC found only one bug not found by SA
– SA found 13 bugs not found by CMC● Protocol Specific Properties:
– Difficult to check statically 0 bugs→● SA hit more code but CMC hit more properties!
Summary
● Static Analysis Advantage:– Wins when checking same set of properties (again)
● Model Checking Advantage: – Simulates execution of 'entire system'
– Looks for actual errors (not 'causes' of error)
– Gives actual execution trace● More checks more bugs→
– CMC executed the code
Case Study 3: TCP
● TCP Code size ~ 10 * (AODV code size)● Frequently audited, Heavily Tested code● Only Model Checking done (using CMC)● System tightly coupled with Environment
– System: TCP stack
– Environment: Linux Kernel, Kernel Modules● Incremental refinement of model
– Start simple, keep adding complexity
Case Study 3: TCP● External Functions can be included in System● Advantage
– Model Checker will find errors in them
– No effort to create stubs● Disadvantage
– Increase in state space
– Dependencies on other functions● Essential Question
– Where to draw the line between system and environment?
System Separation● Cut on narrowest interface● TCP – Kernel module
– Poorly documented
– Effectively simulating kernel
– Months of effort!
– Difficult to determine false positives● New approach: Use well defined interface
– System calls interface
– Hardware abstraction layer
– Effectively running entire kernel in CMC!
Results
● 4 bugs found● Metric to measure coverage:
– Line Coverage: Lines of code covered
– Protocol Coverage: %age of abstract protocol behavior covered
Description Line Coverage Protocol Coverage Bugs
Standard server, client 47.4 % 64.7 % 2
+ Simultaneous connect 51.0 % 66.7% 0
+ Partial Close 52.7 % 79.5% 2
+ Message corruption 50.6% 84.3% 0
Combined Coverage 55.4% 92.1%
Conclusions
● No model is as good as implementation itself– Simplification , omission missing critical errors→
● Manual work scale badly for complex systems● Use well defined interfaces● More code coverage More bugs found→
My Opinion
● Usage of live software project is a big plus● Meta-Compilation & CMC give huge advantage● Better comparison if same tools used across
case studies● Time & Effort not measured accurately● No comparison with other existing tools
Future Scope
● How to increase code coverage in meta compilation and CMC?
● Analysis of class of properties checked
References
1)Experiences using static analysis & model checking for bug finding – Dawson Engler and Madanlal Masuvathi
2)A simple method for extracting models from protocol code – David Lie, Andy Chou, Dawson Engler, David L. Dill
3)Stanford FLASH Multiprocessor: Status, Some Lessons and Plans – John Hennessy
Q & A