Saner presentation nk_march16
-
Upload
noureddine-kerzazi -
Category
Science
-
view
38 -
download
0
Transcript of Saner presentation nk_march16
![Page 1: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/1.jpg)
Botched Releases: Do we Need to Roll Back?
Noureddine Kerzazi Bram Adams&
![Page 2: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/2.jpg)
Agenda
• Context• RQs
▸RQ1. How often do botched releases cause crashes, hangs or performance slowdown?▸RQ2. Can we build a good explanatory model of botched releases?▸RQ3. What are the most important indicators of botched releases?
• Methodology: Data Collection• Results• The Next Step
2
![Page 3: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/3.jpg)
Context3
• When it breaks you got alerted before you know
![Page 4: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/4.jpg)
4
What looks like a Crash?
![Page 5: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/5.jpg)
5
![Page 6: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/6.jpg)
You might think that it's easier to roll back the release
How about Db Scripts that have been applied? How about Configuration Files
that have been changed?How about The Cache Clear?Provisioning of prod. Servers….….
![Page 7: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/7.jpg)
Place your screenshot here
7
What looks like a Hang?
• How many hangs we faced in 15 months?
![Page 8: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/8.jpg)
Place your screenshot here
8
What looks like a performance issue?
![Page 9: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/9.jpg)
Post-Release failures trend9
![Page 10: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/10.jpg)
A Botched Release is ….
10
a release experiencing abnormal
system behaviour (e.g., crashes, hangs or
poor performance) few minutes after being
deployed into the production environment.
![Page 11: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/11.jpg)
Overview of the Release Process
• Worth noticing that:▸We are in the context of parallel development (72+ active branches).▸Relatively large scale deployment (16 + servers)
11
Pre-Release
HotFix
MustFix
Pipeline
Staging Env.
Production Env.
![Page 12: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/12.jpg)
Context
• Web-based financial system▸192 countries and 42 currencies (? laws)
• .Net platform : C#, VB, C, HTMl 5, Angular, …• 1.6 million LOC• 8.524 source code files• 40 projects within the solution• 200+ Devs, BA, DBA, Testers, IT, 1 Architect, 1 Integrator
12
![Page 13: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/13.jpg)
Populating a Release History Database13
![Page 14: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/14.jpg)
2. Results
.
14
![Page 15: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/15.jpg)
15
RQ1. How often do botched
releases cause crashes, hangs
or performance slowdown?
![Page 16: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/16.jpg)
How Often do Botched Releases Cause Crashes or Performance Issues?16
CrashesHang
Performance
![Page 17: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/17.jpg)
RQ1
• 22.5% (72 out of 320) of release were botched.• ~50% of post-release failures that cause system crashes
pertain to the source code• ~20% of system crashes and hangs relate to Configuration
failures • ~25% of hangs and ~12% of crashes are due to problems
with database scripts.• ~19% of hangs and ~12% of crashes are due to platform
changes
17
![Page 18: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/18.jpg)
RQ118
▹Botched Releases are not only related to source code issues.
![Page 19: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/19.jpg)
19
RQ2. Can we build an explanatory
model of botched releases?
![Page 20: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/20.jpg)
Populating a Release History Database20
![Page 21: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/21.jpg)
21
RQ3. What are the most important
indicators of botched releases?
![Page 22: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/22.jpg)
What are the Most Important Indicators
of Botched Releases ?
22 RQ2 & RQ3
#Devs Changing
the code
Purp. of the Branch
Modified Subsystem
#Branch syncs
#rev. to work items
Interval Broken Buil.
Trust
Magnitude
# sub Systems
Log10 Files
Log10 Churn
![Page 23: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/23.jpg)
FactorsWhat matters more is not how codeflows into a release, but rather theamount of development work thatwent into it, the kind of release andwho was responsible for it.
23
![Page 24: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/24.jpg)
Some Threats to Validity
• Mapping data • Metrics chosen to build our explanatory model of botched
releases are based on the experience of release engineers.
24
![Page 25: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/25.jpg)
LET’S REVIEW
YBotched Releases are not onlyrelated to source code issues.
MWhat matters more is not how code flows into a release, but rather the amount of development work that went into it, the kind of release and who was responsible for it.
TTrust is one of the most important factors.
WWhen it breaks, find out why?
RThe random forest model obtains a precision of 88.3% (83/94) and recall of 84.7% (83/98).
NNever compromise in your release practices, no matter how small your release gets.
25Botched Releases
Do we Need to Rollback
![Page 26: Saner presentation nk_march16](https://reader031.fdocuments.net/reader031/viewer/2022030312/58edc5f71a28ab784d8b4637/html5/thumbnails/26.jpg)
WANT BIG IMPACT?USE BIG IMAGE.
26