Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it...
Transcript of Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it...
![Page 1: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/1.jpg)
![Page 2: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/2.jpg)
Agile and DevOps from the trenches at ASTRON
ADASS XXVIII, 11-15 November 2018
![Page 3: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/3.jpg)
How it all started
• In 2011, LOFAR software development was in crisis• Focus had been on getting the instrument to work• Little time was spent to make it ready for operations• Pressure on the software team to deliver new features
![Page 4: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/4.jpg)
How it all started
• In 2011, LOFAR software development was in crisis• Focus had been on getting the instrument to work• Little time was spent to make it ready for operations• Pressure on the software team to deliver new features
• But• Features lacked requirements• Huge Technical Debt
![Page 5: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/5.jpg)
How it all started
• In 2011, LOFAR software development was in crisis• Focus had been on getting the instrument to work• Little time was spent to make it ready for operations• Pressure on the software team to deliver new features
• But• Features lacked requirements• Huge Technical Debt
• Something needed to change ...
![Page 6: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/6.jpg)
Traditional Software Development
Waterfall method
Royce 1970, © IEEE
![Page 7: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/7.jpg)
Why does Waterfall not work?
![Page 8: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/8.jpg)
Why does Waterfall not work?
Complexity!
![Page 9: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/9.jpg)
Why does Waterfall not work?
Complexity!
technology
requ
irem
ents
Stacey matrix
![Page 10: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/10.jpg)
Why does Waterfall not work?
Complexity!
• Waterfall is good for Simple* projects• Agile works better for Complicated and
Complex projects
* Simple does not mean Easytechnology
requ
irem
ents
Stacey matrix
![Page 11: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/11.jpg)
Why does Waterfall not work?
Complexity!
• Waterfall is good for Simple* projects• Agile works better for Complicated and
Complex projects
• Anarchy should be avoided whereverpossible
* Simple does not mean Easytechnology
requ
irem
ents
Stacey matrix
![Page 12: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/12.jpg)
Waterfall vs Agile
![Page 13: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/13.jpg)
Waterfall vs Agile
• Cyclic approach• You still use Waterfall
but with (very) short iterations• This makes you Agile,
because you can easily adapt to change.
![Page 14: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/14.jpg)
Waterfall vs Agile
• Cyclic approach• You still use Waterfall
but with (very) short iterations• This makes you Agile,
because you can easily adapt to change.
• But Agile is more ...
![Page 15: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/15.jpg)
Agile Software Development & Scrum
Manifesto for Agile Software Development
![Page 16: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/16.jpg)
Agile Software Development & Scrum
Manifesto for Agile Software Development
• Individuals and interactions over processes and tools• Working software over comprehensive documentation• Customer collaboration over contract negotiation• Responding to change over following a plan
![Page 17: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/17.jpg)
Agile Software Development & Scrum
Manifesto for Agile Software Development
• Individuals and interactions over processes and tools• Working software over comprehensive documentation• Customer collaboration over contract negotiation• Responding to change over following a plan
Scrum is a framework that aims to implement these Agile principles
![Page 18: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/18.jpg)
Agile Software Development & Scrum
Does Agile/Scrum work in a scientific environment?
![Page 19: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/19.jpg)
Agile Software Development & Scrum
Does Agile/Scrum work in a scientific environment?
• Reasons why it could work:• Projects are generally complex• Requirements constantly change (both user and system)
![Page 20: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/20.jpg)
Agile Software Development & Scrum
Does Agile/Scrum work in a scientific environment?
• Reasons why it could work:• Projects are generally complex• Requirements constantly change (both user and system)
• So, we gave it a try ...
![Page 21: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/21.jpg)
Agile/Scrum at ASTRON
![Page 22: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/22.jpg)
Agile/Scrum at ASTRON
![Page 23: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/23.jpg)
Agile/Scrum at ASTRON
![Page 24: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/24.jpg)
First Lessons Learned● Not having a Product Owner is really problematic
(Even if you have involved users)
![Page 25: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/25.jpg)
First Lessons Learned● Not having a Product Owner is really problematic
(Even if you have involved users)
➔ It is one of the main reasons for not having Sprint Reviews(We sometimes give demos, but not on a regular basis)
![Page 26: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/26.jpg)
First Lessons Learned● Not having a Product Owner is really problematic
(Even if you have involved users)
➔ It is one of the main reasons for not having Sprint Reviews(We sometimes give demos, but not on a regular basis)
➔ It results in bad User Stories(Can be a big issue)
![Page 27: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/27.jpg)
Agile/Scrum at ASTRON
In practice
![Page 28: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/28.jpg)
Agile/Scrum at ASTRON
In practice• Three-week Sprints• Sprint Planning based on Product Backlog• Development on branches• Nightly builds• Build after each commit on the trunk → early warning for errors• Code review before merge to the trunk• Definition of Done
![Page 29: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/29.jpg)
Agile/Scrum at ASTRON
So, do we do Scrum?Basically: no
![Page 30: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/30.jpg)
Agile/Scrum at ASTRON
So, do we do Scrum?Basically: no
Are we Agile?I think we are.
![Page 31: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/31.jpg)
What about DevOps?
![Page 32: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/32.jpg)
What about DevOps?
• DevOps is the combination of development (Dev) and operations (Ops)• Goal is to shorten the development life cycle
![Page 33: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/33.jpg)
What about DevOps?
• DevOps is the combination of development (Dev) and operations (Ops)• Goal is to shorten the development life cycle
• Is DevOps Agile?No.
![Page 34: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/34.jpg)
What about DevOps?
• DevOps is the combination of development (Dev) and operations (Ops)• Goal is to shorten the development life cycle
• Is DevOps Agile?No.
• But, Agile is an essential part of successful DevOps.
![Page 35: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/35.jpg)
DevOps Tools used at ASTRON
![Page 36: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/36.jpg)
DevOps Tools used at ASTRON
![Page 37: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/37.jpg)
DevOps at ASTRON● Daily builds, and commit-triggered builds● Frequent trunk releases● Automatic deployment● Continuous system monitoring● Collecting log-files for system debugging
![Page 38: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/38.jpg)
Agile/Scrum: Lessons Learned
![Page 39: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/39.jpg)
Agile/Scrum: Lessons Learned
What worked for us• Better planning• More grip on progress• Accurate planning for the next milestone• Good ball-park estimates for future milestones
• Improved software quality• Stable trunk, thanks to the use of feature branches• More focus of the team, thanks to short cycles
• More involvement of users and commissioners
![Page 40: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/40.jpg)
Agile/Scrum: Lessons Learned
What did not work for us• Really work as a Scrum team• Too much specialism makes it hard to take over someone else’s work
• Sometimes too many unknowns and unexpected setbacks
![Page 41: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/41.jpg)
Agile/Scrum: Lessons Learned
What did not work for us• Really work as a Scrum team• Too much specialism makes it hard to take over someone else’s work
• Sometimes too many unknowns and unexpected setbacks
What we found hard• Plan for the unknown• How to handle software architecture?
![Page 42: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/42.jpg)
Agile/Scrum: Lessons Learned
Improved understanding means improved planning
![Page 43: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/43.jpg)
Agile/Scrum: Lessons Learned
Improved understanding means improved planning● Do not start to work on stories that are unclear● Break-down a story into smaller tasks● If stories are too big, chop them up● Involve all stakeholders
– Operators and Science Support are often forgotten
![Page 44: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/44.jpg)
Agile/Scrum: Lessons Learned
Improved understanding means improved planning● Do not start to work on stories that are unclear● Break-down a story into smaller tasks● If stories are too big, chop them up● Involve all stakeholders
– Operators and Science Support are often forgotten
● … And make sure you have a Product Owner
![Page 45: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/45.jpg)
Conclusion
Agile/Scrum works!
But it requires:– organizational change– social change, and– a team that is willing to continously improve itself.
This is not a technical challenge, but a social challenge!
![Page 46: Agile and DevOps - Astronomical Data Analysis …adass2018.astro.umd.edu/abstracts/O6-1.pdfHow it all started • In 2011, LOFAR software development was in crisis • Focus had been](https://reader034.fdocuments.net/reader034/viewer/2022042309/5ed69c28843ed9152066b53f/html5/thumbnails/46.jpg)
Questions?