Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining...
Transcript of Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining...
![Page 1: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/1.jpg)
Predicting Failures at SAP
A talk by Kim Herzig
![Page 2: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/2.jpg)
ReproductionMining Metrics to Predict Component Failures
Zeller et al.
![Page 3: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/3.jpg)
Structure of this Talk
1. Building failure predictor
2. Working at SAP and initial problems
3. Current status
4. Future work
![Page 4: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/4.jpg)
Hypotheses
Hypothesis
Increase in complexity metrics of an entity E correlates with the number of post-release defects of E
There is a common subset of metrics for which H1 applies in all projects
There is a combination of metrics which significantly predicts the post-release defects of new entities within a
project
Predictors obtained using H3 from one project also predict failure-prone entities in other projectssimilar
H1
H2
H3
H4
![Page 5: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/5.jpg)
RoadmapStep 1: Map bug report to code changes
get reported bugs(post release)
get version datafor reported bugs
get source code
source of graphics: Mining Metrics to Predict Component Failures, Zeller et al.
BugDatabase
VersionDatabase
SourceCode
![Page 6: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/6.jpg)
RoadmapStep 2: Map failures to defects in entities
number of failures found
source of graphics: Mining Metrics to Predict Component Failures, Zeller et al.
BugDatabase
VersionDatabase
SourceCode
Entity Entity Entity
![Page 7: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/7.jpg)
RoadmapStep 3: Predict failure for new entities
How to build this?
source of graphics: Mining Metrics to Predict Component Failures, Zeller et al.
Entity Entity Entity
Entity Predictor FailureProbability
BugDatabase
VersionDatabase
SourceCode
![Page 8: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/8.jpg)
Building Failure Predictor
compute complexity metrics for each entity
e: m(e)m(e)
metrics data
Entity
EntityEntity
![Page 9: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/9.jpg)
Building Failure Predictor
find metrics m(e) correlating to
quality(e): failure predictor
metrics data
Failure Predictor
Entity
EntityEntity
![Page 10: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/10.jpg)
To Be Reproduced !For more details read paper
![Page 11: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/11.jpg)
SAP vs. Microsoft• Other programming language (ABAP)‣ no metric tools!‣ ABAP highly database oriented
• Customers involved in processes‣ not all processes are in house (communication)‣ bug database more like ticket system‣ many different releases with different features‣ Service packs contain new feature
• New terrain for SAP‣ pure future work‣ many current projects but lack of experience
![Page 12: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/12.jpg)
Goals for SAP
• Global Quality Governance:
‣ Help to improve SAP quality strategy
‣ Make development processes more efficient
‣ Integrate into product standard framework
• Development Teams:
‣ Which module to test most?
‣ Identifying critical components!
![Page 13: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/13.jpg)
How does SAP work?online!
![Page 14: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/14.jpg)
R/3
• Application Server
‣ programs written in ABAP/4
‣ programs executed on server
• Development Environment
‣ completely online (no data local available)
‣ integrated version repository
![Page 15: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/15.jpg)
Initial Problems
![Page 16: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/16.jpg)
Limited Access
• Working on Maintenance Servers
‣ Code frequently shipped
‣ Limited data access
‣ Limited development permission
![Page 17: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/17.jpg)
Bug Data
• Ticket System
• separate system (no integration)
• used by clients and internally
• difficult data access
• Version Repository
• integrated in development system
• no direct connection to ticket system
![Page 18: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/18.jpg)
ABAP/4
• Source code stored online (database)‣ how to get code? see later!
• Mixture between functional and object-oriented programming‣ What metrics to be used?
• Integration of SQL‣ Additional metrics?
• Gimmicks like ‘implicit function calls’‣ Hard to detect!
*
* Allgemeiner Berichtsaufbereitungsprozessor
![Page 19: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/19.jpg)
What can be reused?
![Page 20: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/20.jpg)
Basic Framework
• Maps package name to code objects
• Gets source code (incomplete)
• Computes McCabe complexity
• Gets code deltas from repository (untested)
![Page 21: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/21.jpg)
Structure of this Talk
1. Building failure predictor
2. Working at SAP and initial problems
3. Current status
4. Future work
![Page 22: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/22.jpg)
Initial Summary
we have we need
bug data source code
version data metrics data
![Page 23: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/23.jpg)
Requirements
• SAP wants to reuse code‣ programming ABAP/4
• Keep adoption effort as low as possible‣ programming JAVA
• Use as many code information as possible
• Compute metrics efficiently
![Page 24: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/24.jpg)
Let’s get started!
![Page 25: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/25.jpg)
identify code objects
Getting Source
≥ 20 TABLES
initial package
compute dependencies
dependent packages
Source Code
Meta Data
![Page 26: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/26.jpg)
Connectto R/3
The Hybrid
Start GUI
RFC call collect sourcecode
compute basicmetrics
CodeObjects,Metrics/Meta- Data
persistentDB
store results
compute metametrics
![Page 27: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/27.jpg)
SAP Reuse
collect sourcecode
compute basicmetrics
ABAP/4 GUI
ABAP/4 tables
compute meta metrics(to be implemented)
![Page 28: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/28.jpg)
Compute Basic Metrics
Definition: can be computed by simply scanning code
• While assembling code objects
• Granularity: Function/Method - Level
• Collect meta data for later processing(parameter types, x calls y, etc.)
![Page 29: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/29.jpg)
Definition: can not be computed by simply scanning code
• Uses meta data collected by basic metrics
• Computed on database (MSSQL)
• Never computed at SAP
Compute Meta-Metrics
![Page 30: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/30.jpg)
The Metrics
![Page 31: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/31.jpg)
The Metrics- Module related metrics- Counting number of classes, functions, etc.- Detect database connectivity- Look at include statements
- Function related metrics- McCabe, FanIn, FanOut etc- Usage of global variables
- Class related metrics- Inheritance, Coupling
![Page 32: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/32.jpg)
The Metrics
~50% of metrics implemented
![Page 33: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/33.jpg)
metrics data available byend of February
![Page 34: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/34.jpg)
Work to Come
• Measure entity quality‣ Assumption: every delta is bug fix‣ filtering bug data (reopened bugs, delta size)
• Build the failure predictor‣ which metrics correlate to quality
• Statistic analysis‣ try to merge with JAVA approach (Tilman)
• Reproduction on other systems‣ Let’s get excited!
![Page 35: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/35.jpg)
Thank you!Questions?
![Page 36: Predicting Failures at SAP · Predicting Failures at SAP A talk by Kim Herzig. Reproduction Mining Metrics to Predict Component Failures Zeller et al. Structure of this Talk 1. Building](https://reader034.fdocuments.net/reader034/viewer/2022052000/601220faa5d8350745077b34/html5/thumbnails/36.jpg)
Questions?Thank you!