Which Configuration Option Should I Change?
-
Upload
cedric-wilson -
Category
Documents
-
view
31 -
download
0
description
Transcript of Which Configuration Option Should I Change?
![Page 1: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/1.jpg)
Which Configuration Option Should I Change?
Sai Zhang, Michael D. ErnstUniversity of Washington
Presented by: Kıvanç Muşlu
![Page 2: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/2.jpg)
2
Developers
Users
I have released a new software version …
I cannot get used to the UI
I do not know how to configure it
…
![Page 3: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/3.jpg)
3
Diagnosis of User-Fixable Software Errors
• Goal: – enable users to fix software errors
• Challenges:– Errors can be crashing or non-crashing– Users much less understand source code– Developer tools are of little use
![Page 4: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/4.jpg)
4
A new software versionOur previous work [ISSTA’13]
Help users adapt to the new UI
Users
I cannot get used to the UI
![Page 5: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/5.jpg)
5
A new software version
This paper: How to help users configure the new software version (i.e., diagnosis of configuration errors)
Users
I do not know how to configure it
![Page 6: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/6.jpg)
6
Software system often requires configuration
Configuration options… Configuration errors:
- Users use wrong values for options - The software exhibits unintended behaviors
Example: --port_num = 100.0
Should be a valid integer
![Page 7: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/7.jpg)
7
Configuration errors are common and severe
Root causes of high-severity issues ina major storage company [Yin et al, SOSP’11]
Configuration errors can havedisastrous impacts(downtime costs 3.6% of revenue)
![Page 8: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/8.jpg)
8
Configuration errors are difficult to diagnose• Error messages are absent or ambiguous
– e.g.,
• Infeasible to automatically search for a good configuration– Need to know the spec of a valid configuration option value (e.g., regex, date time, integer value range)– Huge search space− Need to specify a testing oracle for automation
• Cannot directly use existing debugging techniques[Zhang et al., ICSE’13]
(after setting --port_num = 100.0 in webs server)
![Page 9: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/9.jpg)
9
Goal: diagnosing configuration errors for evolving software
To maintain the desired behavior on the new versionWhich configuration option should I change?
Old version New version
Requires configuration!
a different output
![Page 10: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/10.jpg)
10
Old version New version
a different output
Diagnosing configuration errors with ConfSuggester
Our technique: ConfSuggester
Suspicious configuration options…
Key idea: The execution traceon the old version asthe “intended behavior”
![Page 11: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/11.jpg)
11
Design constraints for ConfSuggester
• Accessible: no assumption about user background (e.g., users cannot read or write code annotations)
• Easy-to-use: fully automated
• Portable: no changes to OS or runtime environment
• Accurate: few false positives
![Page 12: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/12.jpg)
12
Outline
• Example
• A Study of Configuration Evolution
• The ConfSuggester Technique
• Evaluation
• Related Work
• Contributions
![Page 13: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/13.jpg)
13
Outline
• Example
• A Study of Configuration Evolution
• The ConfSuggester Technique
• Evaluation
• Related Work
• Contributions
![Page 14: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/14.jpg)
14
A popular performance testing tool
Use Jmeter to monitor a website’s performance
Managers
![Page 15: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/15.jpg)
15Use Jmeter to monitor a website’s performance
Managers
Version 2.8 Version 2.9All regression tests passed
![Page 16: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/16.jpg)
16
Version 2.8 Version 2.9All regression tests passed
The new version behaves as designed,but differently from a user expects.
No regression bugs.Causes XML parsing error
![Page 17: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/17.jpg)
17
Version 2.8 Version 2.9All regression tests passed
ConfSuggester
Suspicious configuration options…
output_format
Resolve the problem: set output_format = XML
![Page 18: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/18.jpg)
18
Version 2.8 Version 2.9All regression tests passed
![Page 19: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/19.jpg)
19
Outline
• Example
• A Study of Configuration Evolution
• The ConfSuggester Technique
• Evaluation
• Related Work
• Contributions
![Page 20: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/20.jpg)
20
Do configuration changes arise in software evolution?• 8 open-source programs
• 40 versions released in the past 6 years
• Searched for “configuration changes”-related messages in 7022 commits and 28 change logs
‒ Count the number of changes made to configuration options
![Page 21: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/21.jpg)
21
Results
• Configuration changes arise in every version of all software systems
• Configuration change can lead to unexpected behaviors (details later)
(394 configuration changes in total)
AddedOptions
ModifiedOptions
DeletedOptions
Enhance features
Fix bugs
Renaming
Reliability
![Page 22: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/22.jpg)
22
Outline
• Example
• A Study of Configuration Evolution
• The ConfSuggester Technique
• Evaluation
• Related Work
• Contributions
![Page 23: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/23.jpg)
23
Key insights of ConfSuggester• Control flow propagates most configuration options’ effects
• The execution traces on the old version can serve as the “intended behavior”– The control flow difference and their impacts provides diagnosis clues
/* a configuration option in JMeter */String output_format = readFromCommandLine(); ...if (output_format == “XML”) { saveAsXML();} else { saveAsCSV();} The evaluation result of this predicate affects the
next 1000+ instructions
![Page 24: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/24.jpg)
24
Workflow of ConfSuggester
Old version
New version A new trace
An old traceTrace
Comparison
Deviated execution parts(at the predicate-level)
Root CauseAnalyzer1.
2.3. …
Report
![Page 25: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/25.jpg)
25
Workflow of ConfSuggester
Old version
New version A new trace
An old traceTrace
Comparison
Deviated execution parts(at the predicate-level)
Root CauseAnalyzer1.
2.3. …
Report
User demonstration: show the error
Dynamic analysis: understand the
behaviorStatic analysis:
compute the solution
![Page 26: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/26.jpg)
26
Workflow of ConfSuggester
![Page 27: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/27.jpg)
27
User demonstration
Old version
New version A new trace
An old trace
Code instrumentation, monitoring:1. predicate execution frequency and result2. execution of each other instruction
![Page 28: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/28.jpg)
28
Execution trace comparison
An old trace
A new trace
: a predicate : a deviated predicate
Rankingdeviatedpredicates
Identifyingdeviatedpredicates
Matchingpredicates
![Page 29: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/29.jpg)
29
Matching predicate across traces
• JDiff algorithm [Apiwattanapong’07]− Tolerate small changes between versions
...if (output_format == “XML”) { saveAsXML();} else { saveAsCSV();}...
Old version
...if (isValidFormat(output_format) { //check validity }
if (output_format == “XML”) { checkXMLParser(); saveAsXML();} else { saveAsCSV();}...
New version
![Page 30: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/30.jpg)
30
Identifying deviated predicates
…An old trace
A new trace
: a predicate : a deviated predicate
a predicate p’s behavior in an execution trace t:
ϕ (p, t) =
a predicate p’s behavior difference across executions:
deviation(p, told, tnew) = | ϕ (p, told) - ϕ (p, tnew) |
p is a deviated predicate, if deviation(p, told, tnew) > δ
21
𝑒𝑥𝑒𝑐 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 +1
𝑡𝑟𝑢𝑒𝑟𝑎𝑡𝑖𝑜
Goal:
![Page 31: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/31.jpg)
31
Ranking deviated predicates
Rank predicates by their impacts A predicate p’s deviation impact = deviation(p, told, tnew) × (controlled_instructions(p, told) + controlled_instructions(p, tnew) )
...if (output_format == “XML”) { saveAsXML();} else { saveAsCSV();}...
Old trace
if(..)
saveAsXML() saveAsCSV()
Old trace# of instructionsexecuted
Defined in the previous slide
predicate p:
![Page 32: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/32.jpg)
32
Ranking deviated predicates
...if (output_format == “XML”) { saveAsXML();} else { saveAsCSV();}...
if(..)
saveAsXML() saveAsCSV()
New trace # of instructionsexecuted
New tracepredicate p:
Rank predicates by their impacts A predicate p’s deviation impact = deviation(p, told, tnew) × (controlled_instructions(p, told) + controlled_instructions(p, tnew) )
![Page 33: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/33.jpg)
33
Ranking deviated predicates
...if (output_format == “XML”) { saveAsXML();} else { saveAsCSV();}...
if(..)
saveAsXML() saveAsCSV()
New trace # of instructionsexecuted
New tracepredicate p:
Rank predicates by their impacts A predicate p’s deviation impact = deviation(p, told, tnew) × (controlled_instructions(p, told) + controlled_instructions(p, tnew) )
Approximate the impact of a predicate’s behavior change to the subsequent program execution.
![Page 34: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/34.jpg)
34
Root Cause Analyzer
Find configuration options affecting the deviated predicate− Using static thin slicing [Sridharan ’07]
//a configuration option in JMeterString output_format = ...; ...if (output_format == “XML”) { saveAsXML();} else { saveAsCSV();}
The behavior of this predicate deviates
Compute a backward thin slice from here
Find the affecting predicate
1.2.3. …
Reportoutput_format
![Page 35: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/35.jpg)
35
Outline
• Example
• A Study of Configuration Evolution
• The ConfSuggester Technique
• Evaluation
• Related Work
• Contributions
![Page 36: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/36.jpg)
36
8 configuration errors from 6 subjects
Subject LOC #Options ∆LOC #Config errorsRandoop 18587 57 1893 1
Weka 275035 14 1458 1
Synoptic 19153 37 1658 2
JChord 26617 79 3085 2
JMeter 91797 55 3264 1
Javalanche 25144 35 9261 1
Non-trivial code changes
Reproduced from change logs and user reports.
![Page 37: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/37.jpg)
37
ConfSuggester’s accuracy• Measure accuracy by the rank of the actual root
cause in ConfSuggester’s output1.2.3. …
![Page 38: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/38.jpg)
38
ConfSuggester’s accuracy• Measure accuracy by the rank of the actual root
cause in ConfSuggester’s output1.2.3. …
Technique Average Root Cause RankBaseline 23.3ConfAnalyzer [Rabkin’11] 22ConfDiagnoser [Zhang’13] 15.3ConfSuggester 1.9• Baseline:
‒ Users select options in an arbitrary order‒ Half of the total number of available options
![Page 39: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/39.jpg)
39
ConfSuggester’s accuracy• Measure accuracy by the rank of the actual root
cause in ConfSuggester’s output1.2.3. …
Technique Average Root Cause RankBaseline 23.3ConfAnalyzer [Rabkin’11] 22ConfDiagnoser [Zhang’13] 15.3ConfSuggester 1.9• ConfAnalyzer:
‒ Use program slicing for error diagnosis
![Page 40: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/40.jpg)
40
ConfSuggester’s accuracy• Measure accuracy by the rank of the actual root
cause in ConfSuggester’s output1.2.3. …
Technique Average Root Cause RankBaseline 23.3ConfAnalyzer [Rabkin’11] 22ConfDiagnoser [Zhang’13] 15.3ConfSuggester 1.9• ConfDiagnoser:
‒ Use trace comparison (on the same version) for error diagnosis
![Page 41: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/41.jpg)
41
ConfSuggester’s accuracy• Measure accuracy by the rank of the actual root
cause in ConfSuggester’s output1.2.3. …
Technique Average Root Cause RankBaseline 23.3ConfAnalyzer [Rabkin’11] 22ConfDiagnoser [Zhang’13] 15.3ConfSuggester (this paper) 1.9
• ConfSuggester:- 6 errors: root cause ranks 1st
- 1 error: root cause ranks 3rd
- 1 error: root cause ranks 6th
![Page 42: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/42.jpg)
42
ConfSuggester’s efficiency
• User demonstration– 6 minutes per error, on average
• Error diagnosis– 4 minutes per error, on average
![Page 43: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/43.jpg)
43
Outline
• Example
• A Study of Configuration Evolution
• The ConfSuggester Technique
• Evaluation
• Related Work
• Contributions
![Page 44: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/44.jpg)
44
Related work on configuration error diagnosis
• Tainting-based techniques– Dynamic tainting [Attariyan’08], static tainting [Rabkin’11]Focuses exclusively on crashing errors
• Search-based techniques– Delta debugging [Zeller’02], Chronus [Whitaker’04]Requires a correct state for comparison, or OS-level support
• Domain-specific techniques– PeerPressure [Wang’04], RangeFixer [Xiong’12]Targets a specific kind of configuration errors, and does not support a general language like Java
A common limitation: do not support configuration error diagnosis in software evolution.
![Page 45: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/45.jpg)
45
Outline
• Example
• A Study of Configuration Evolution
• The ConfSuggester Technique
• Evaluation
• Related Work
• Contributions
![Page 46: Which Configuration Option Should I Change?](https://reader035.fdocuments.net/reader035/viewer/2022081520/5681388f550346895da03f5a/html5/thumbnails/46.jpg)
46
• A technique to diagnose configuration errors for evolving softwareCompare relevant predicate behaviors between executions from two versions
• The ConfSuggester tool implementation http://config-errors.googlecode.com
Accessible: no assumption about user background Easy-to-use: fully automated Portable: no changes to OS or runtime environment Accurate: few false positives
ContributionsConfiguration errors
ConfSuggester1.2.3. …
Report