you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 ·...
Transcript of you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 ·...
![Page 1: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/1.jpg)
“It is wrong to suppose that if you can’t measure it, you can’t
manage it - a costly myth”
W. Edwards Deeming
![Page 2: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/2.jpg)
“If you can’t measure it, you can’t improve it”
Lord Kelvin
![Page 3: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/3.jpg)
Technical Debt for Linux-based distributions:Estimating what you are missing
Jesus M. Gonzalez-Barahona (URJC & Bitergia)Paul Sherwood (Codethink)speakerdeck.com/bitergia
Linux Foundation Open Source Leadership SummitTahoe, CA (USA)February 14th 2017
![Page 4: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/4.jpg)
OutlineSome context
Why debt for distros
Approach
Current results
Next steps
![Page 5: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/5.jpg)
Some context
![Page 6: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/6.jpg)
/Jesus
Like five years ago I was having coffees with the gang of Bitergia founders
Involved in the company since then
bitergia.com
I work at Universidad Rey Juan Carlos...
...researching about software development
gsyc.es/~jgb
My two hats:
![Page 7: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/7.jpg)
/PaulCurrently…
Codethink CEO and shareholder
Consultant + troubleshooter
Baserock contributor
Previously...
Teleca Founder
cmdline tools + VCS
Project Manager
“The Software Commandments”
![Page 8: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/8.jpg)
Why debt for distros
![Page 9: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/9.jpg)
Context
(Paul’s POV)
● Develop/integrate/test software● Employ/fund others to do that too● Offer teams to large customers
● Advise on business impacts of FOSS● Recommend *using* FOSS● See lots of projects *misusing* FOSS
○ EOL versions
○ Long local forks, not upstreamed
● Notice Year 1 practices hurt Year 2..Year 20● Wonder why… maybe because
○ Year 1 metrics are obvious (developer costs vs delivery date)
○ Later metrics are a mystery...
![Page 10: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/10.jpg)
Unanswered:when shouldwe update?
![Page 11: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/11.jpg)
Unanswered:when shouldwe update?
![Page 12: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/12.jpg)
We’re not talkingabout updating just a few components...
![Page 13: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/13.jpg)
Typical IVI project approaching 1000…
Which ones do we need to upgrade?
How often do we need to re-decide?
![Page 14: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/14.jpg)
Example ● Project started on 3.8.x kernel in 2012○ Plus custom drivers
● Went live three years later on same 3.8.x○ Plus custom functionality○ Plus thousands of fixes backported
● As the years go by○ Developers move on - no-one understands the
custom stuff○ Cost of backporting increases
● New variants need new features (eg virtualization)○ Cost of backporting from later kernels increases
Eventually one of the releases DEMANDS an update
![Page 15: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/15.jpg)
Example continued
Development
Maintenance
![Page 16: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/16.jpg)
When to update
What you risk by upgrading
What you risk or lose by not upgrading
![Page 17: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/17.jpg)
When to update
The balance may change suddenlyover time
![Page 18: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/18.jpg)
Rationale ● Technical debt is a popular concept● … but not for third-party software● … and not for FOSS
● Distros are large third-party software sets● Distros update constantly● Distro users often do not
● Cost of updating is perceived high● Cost of not updating is unknown
Can we even **find** metrics for this?
![Page 19: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/19.jpg)
ApproachWhat to measure?
- Delta vs mainline- For individual components,
and- For whole stack:
- distros- custom
assemblies/stacks
![Page 20: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/20.jpg)
Defining“Gold standard”
![Page 21: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/21.jpg)
The differentkinds of gold(examples)
Goals Scenarios Candidates
Stability Isolated system, frozen functionality
Debian stable
Functionality Cloud application
Latest upstream
Security Upgradable embedded
Stable upstream
![Page 22: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/22.jpg)
Comparing with upstream
Upstream master
Upstream 2.x
Deployed packages
Distro packages
1.4 2.0 2.1
![Page 23: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/23.jpg)
Comparing with upstream(no updates)
Upstream master
Upstream 2.x
Deployed packages
Distro packages
1.4 2.0 2.1
![Page 24: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/24.jpg)
Comparing with upstream(late updates)
Upstream master
Upstream 2.x
Deployed packages
Distro packages
1.4 2.0 2.1
![Page 25: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/25.jpg)
Comparing with upstream(new package)
Upstream master
Upstream 2.x
Deployed packages
Distro packages
1.4 2.0 2.1 3.0
??
![Page 26: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/26.jpg)
Compare “most likely upstream equivalent”
1.4 2.0 2.1 3.0
??
![Page 27: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/27.jpg)
Compare “most likely upstream equivalent”with HEAD 1.4 2.0 2.1 3.0
??
![Page 28: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/28.jpg)
Difference is “technical lag”with“gold standard”
1.4 2.0 2.1 3.0
??
![Page 29: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/29.jpg)
How to measure difference 1.4 2.0 2.1 3.0
Lines of codeNumber of functions, classesNumber of bugs fixedNumber of security bugs fixedNumber of issues closedTime for benchmark runsUnit test coverageResults in integration tests...
![Page 30: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/30.jpg)
Current results
![Page 31: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/31.jpg)
Debian Git releases,lag in November(lines, files)
![Page 32: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/32.jpg)
Debian Git releases,lag in Nov.(commits)
![Page 33: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/33.jpg)
Normalizedeffort (in days)
For each developer:number of days with at least
one commit
For a project:sum for all developers
![Page 34: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/34.jpg)
Debian Git releases,lag in Nov.(normalizedeffort)
![Page 35: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/35.jpg)
Next steps
![Page 36: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/36.jpg)
Application to many domains
Debian packages in a virtual machine
Python pip packages in a deployed container
JavaScript npm modules in a web app
Yocto packages in an embedded system
![Page 37: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/37.jpg)
Definition of details,according torequirements
Different “golden standards”
Different metrics for lag
Different aggregations
Software for automated computation of lag per component
(and dependencies?)
![Page 38: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/38.jpg)
Credits
![Page 39: you can’t measure it, you can’t manage it - a costly myth” “It is … · 2017-12-14 · Plus custom drivers Went live three years later on same 3.8.x Plus custom functionality](https://reader034.fdocuments.net/reader034/viewer/2022042319/5f08e0387e708231d4242635/html5/thumbnails/39.jpg)
Images “Gold”, by Michael MandibergCC Attribution-ShareAlike 2.0https://flic.kr/p/6feTT2
“Gold philarmonic”, by Eric GolubCC Attribution-ShareAlike 2.0https://flic.kr/p/7csHXG
“Plymouth”, by Dennis JarvisCC Attribution-ShareAlike 2.0https://flic.kr/p/5pqT72
“Jenga distorted”, by Guma89 at WikiMedia Commons
CC Attribution-ShareAlike 3.0
https://commons.wikimedia.org/wiki/File:Jenga_distorted.jpg
“Balance scale”, by winnifredxoxo at FlickrCC Attribution-ShareAlike 2.0https://flic.kr/p/9LdVCR