Post on 22-Dec-2015
/* iComment: Bugs or Bad Comments? */Lin Tan, Ding Yuan, Gopal Krishna, Yuanyuan
ZhouPublished in SOSP 2007
Presented by Kevin Boos
In a Nutshell• iComment: static analysis + NLP• Detects code-comment mismatches• Uses both source code and comments
Roadmap• iComment Paper
o Motivationo Challenges o Contributionso Approach & Methodologyo Resultso Related Work
• Complexity• Authors’ other works
Motivation• Software bugs affect reliability.
o Mismatches between code and developer assumptions
// Caller must acquire lock.static int reset_hardware(...) {
//access shared data. }
static int in2000_bus_reset(...) {reset_hardware(...);
}
Prevalence of Comments
• Comments = developer assumptionso Must hold locks, interrupts must be disabled,
etc.
• Other tools do not utilize comments! o Ignore valuable information (dev. intentions)
Software Lines of Code
Lines of Comments
Linux 5 million 1 million
Mozilla 3.3 million 0.5 million
Code vs. CommentsCode Comment Implication Precise Imprecise Comments are harder to analyze.
Can be tested
Can NOT be tested
Software evolution makes comments less reliable.
Harder to understand
Easier to understand
Developers read comments before code.Wrong comments mislead programmers.
• Developer assumptions can’t always be inferred from source code
• Comments and code are redundanto or should be…
Inconsistencies• What’s wrong: comments or code?
o Developer mistakeo Out of date o Copy and paste error (clone detection)
• Bad code might be bugs• Bad comments cause future bugs
Challenges• Parsing and understanding
commentso Natural language is ambiguous and varying
/* We need to acquire the IRQ lock before calling … */
/* Lock must be acquired on entry to this function. *//* Caller must hold instance lock! */
• NLP only captures sentence structureo No concept of understandingo Decent accuracyo Comments may be grammar disasters…
Contributions• First step towards automatically
analyzing comments o Combines NLP, machine learning, static
analysis
• Identifies inconsistent code & comments
• Real-world applicabilityo Discovered 60 new bugs or bad comments
• Only two topics: locks & calls
Approach• Two types of comments
o Explanatory: /* set the access flags */o Assumptions/Rules: /* don’t call with lock held
*/
• Check comment rules topic-by-topico General frameworko Users choose the hot topics
Rule Templates• <Lock L> must be held before entering <Function F>. • <Lock L> must NOT be held before entering <Function F>. • <Lock L> must be held in <Function F>. • <Lock L> must NOT be held in <Function F>. • <Function A> must be called from <Function B> • <Function A> must NOT be called from <Function B>
• Other templates exist (see paper)• User can add more templates
Handling Comments• Extract comments
o NLP, keyword filters, correlated word filters
• Classify comments (rule generation)o Manually label small subseto Create decision tree with machine learningo Decision tree matches comments to templateso Fill template parameters with actual variables
• Training is optional for users
Rule Checker• Static analysis
o Flow sensitive and context sensitiveo Scope of comments
• Display the inconsistencieso Sorted by ranking (support probability)
EvaluationSoftware SLOC #Cmts. Languag
e Description
Linux 5.0 M 1.0 M C OS
Mozilla 3.3 M 0.51M C, C++ Browser Suite
Wine 1.5 M 0.22 M C Runs Windows Apps in Linux
Apache 0.27 M 0.06 M C Web Server
• Four large software projects• Two topics: locks and function calls• Average training data: 18%
ResultsSoftware Mismatch Bugs Bad
Cmts. FP Rules
Linux 51 (14) 30 (11) 21 (3) 32 1209
Mozilla 6 (5) 2 (1) 4 (4) 3 410
Wine 2 1 1 3 149
Apache 1 0 1 0 64
Total 60 (19) 33 (12) 27 (7) 38 1832 • Automatically detected 60 new bugs and bad
commentso 19 new bugs and bad comments already confirmed by
developers
• False positives exist (38%)o Incorrectly generated ruleso Inaccuracy of checking rule
Training Accuracy
Linux Mozilla Wine Apache
90.8% 91.3% 96.4% 100%
• Accuracy: % of correct mismatches
Training SW Mozilla Wine Apache
Linux 81.5% 78.6% 83.3%
Linux+Mozilla —— 89.3% 88.9%
—— Software-specific training ——
—— Cross-software training ——
Related Work• Extracting rules from source code
o iComment employs static analysis but not dynamic traces
• Annotations o Poor adoption rateso Requires manual effort per comment
• Documentation generationo No usage of NLPo iComment also analyzes unstructured
comments
Complexity• Detecting inconsistencies
o NLP• Abstracted away by tools
o Machine learning• Simple manual training rules
• Code maintenanceo Developers may forget to be thorough
• Automatic bug detectiono Locking errors are extremely complex
Author Bio• Primary author: Lin Tan• Improving software reliability
o Commentso Source codeo Execution traceso Manual input
• HotComments – prior ideas paper
Author Bio• Secondary author: Ding Yuan• Reliability of large software systems• Better logging
o Enhanced output
Author Bio• Professor: Yuanyuan Zhou• Better debuggers, software reliability
• Founded PatternInsight
PatternInsight Startup
• http://patterninsight.com/