r2rtf: Easily Create Production-Ready ... - cran.r-project.org
Production-ready Software
-
Upload
uwe-friedrichsen -
Category
Technology
-
view
1.150 -
download
0
Transcript of Production-ready Software
![Page 1: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/1.jpg)
Production-ready software instead of just being “feature complete”
Uwe Friedrichsen – codecentric AG – 2014-2016
![Page 2: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/2.jpg)
@ufried Uwe Friedrichsen | [email protected] | http://slideshare.net/ufried | http://ufried.tumblr.com
![Page 3: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/3.jpg)
Why this talk?
![Page 4: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/4.jpg)
Ops Dev Business IT value chain Customer
feature complete
production ready
Work in progress Realized business
value
New business
idea
![Page 5: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/5.jpg)
It‘s all about production!
![Page 6: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/6.jpg)
But before we talk about production …
![Page 7: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/7.jpg)
… let’s talk about DevOps briefly
![Page 8: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/8.jpg)
What is that DevOps thing anyway?
![Page 9: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/9.jpg)
Let’s check the
“DevOps bible” http://itrevolution.com/books/phoenix-project-devops-book/
![Page 10: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/10.jpg)
DevOps in a nutshell
![Page 11: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/11.jpg)
The 3 ways of DevOps Systems thinking
Amplify feedback loops
Culture of continual experimentation & learning
http://itrevolution.com/the-three-ways-principles-underpinning-devops/
![Page 12: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/12.jpg)
• Maximize flow (minimize cycle times)
• Optimize for global goals (holistic view)
• Never pass defects downstream
• Limit work in progress
• Build systems and organizations that are safe to change
Ops Dev Business IT value chain Customer
Holistic optimization
Systems thinking
![Page 13: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/13.jpg)
• Facilitate constant flow of fast feedback from right-to-left
• Create quality at source (provide knowledge where needed)
• Create shared goals and shared pain between Dev and Ops
• Implement fast automated test suites
• Pervasively measure outcome (customer value), not output
Ops Dev Business IT value chain Customer
Amplify feedback loops
![Page 14: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/14.jpg)
• Create a culture that fosters two things • Continual experimentation, taking risks and learning from success and failure
• Understanding that repetition and practice is the prerequisite to mastery
• Allocate at least 20% of Dev and Ops cycles to NFRs
• Constantly reinforce that improvements are encouraged & celebrated
Ops Dev Business IT value chain Customer
Continual experimentation and learning
![Page 15: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/15.jpg)
If taken seriously DevOps will eventually rotate your IT organization by 90°
![Page 16: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/16.jpg)
Ops Dev IT value chain
Busin
ess
capa
bilit
ies
Spec
ialis
t tea
m (e
.g.,
prod
uct m
anag
er)
Spec
ialis
t tea
m (e
.g.,
UX e
xper
t)
Spec
ialis
t tea
m (e
.g.,
deve
lope
r)
Spec
ialis
t tea
m (e
.g.,
QA
expe
rt)
Spec
ialis
t tea
m (e
.g.,
sys
adm
in)
Spec
ialis
t tea
m (e
.g.,
DBA
)
…
Traditional IT organization
![Page 17: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/17.jpg)
Ops Dev IT value chain
Busin
ess
capa
bilit
ies
Cross-functional product team
…
DevOps IT organization
Cross-functional product team
Cross-functional product team
Cross-functional product team
![Page 18: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/18.jpg)
Ops Dev IT value chain
Busin
ess
capa
bilit
ies
Cross-functional product team
…
DevOps IT organization (optimized)
Cross-functional product team
Cross-functional product team
Cross-functional product team
Plat
form
team
API
![Page 19: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/19.jpg)
But that’s still a long way to gofor many organizations …
![Page 20: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/20.jpg)
Amplify feedback loops
• Facilitate constant flow of fast feedback from right-to-left
• Create quality at source (provide knowledge where needed)
• Create shared goals and shared pain between Dev and Ops
• Implement fast automated test suites
• Pervasively measure outcome (customer value), not output
Ops Dev Business IT value chain Customer
Thus, we’ll focus on this feedback loop
![Page 21: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/21.jpg)
Let’s talk about operations …
![Page 22: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/22.jpg)
Operations Developers Point of View
![Page 23: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/23.jpg)
Admin Developers Point of View
![Page 24: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/24.jpg)
Admin Closer to Reality Point of View
![Page 25: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/25.jpg)
Top 5 Needs of an Admin 1. Give me my peace and quiet!
2. Don‘t make me think!
3. Let me see that everything is fine!
4. Show me the problem – now!
5. Tell me what to do!
![Page 26: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/26.jpg)
Top 5 Needs of an Admin (translated) 1. Give me my peace and quiet!
(The application should just run smoothly)
2. Don‘t make me think! (Rollout, setup and operation of the application should be easy)
3. Let me see that everything is fine! (The application should show its state)
4. Show me the problem – now! (The application should provide concise error messages and enable easy root cause drilldown)
5. Tell me what to do! (The application should be documented properly – including error handling instructions)
![Page 27: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/27.jpg)
Top 3 Dev Challenges 1. Manageability
2. Resilience
3. Transparency
4. Documentation
![Page 28: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/28.jpg)
11 Design Principles
For production-ready Applications
• Manageability (4)
• Resilience (5)
• Transparency (2)
![Page 29: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/29.jpg)
Manageability
![Page 30: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/30.jpg)
Deployment
(Manageability)
![Page 31: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/31.jpg)
• One-click deployment
• Preserve settings
• Provide rollbacks or roll-forward
• Go for containers
![Page 32: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/32.jpg)
Configuration
(Manageability)
![Page 33: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/33.jpg)
• Avoid multiple configuration procedures
• Define default value handling
• Organize change traceability
• Notification about new parameters
![Page 34: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/34.jpg)
Configuration Parameter Types
(Manageability)
![Page 35: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/35.jpg)
• Context-related parameters Do not stage – managed by stage admin
• Application-related parameters Must be staged – managed by application admin
• Business-related parametersMust be staged – managed by business admin
![Page 36: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/36.jpg)
Backup
(Manageability)
![Page 37: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/37.jpg)
• Think about backup purpose
• Define backup strategy
• Provide tooling
• What about backup in cloud environments?
![Page 38: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/38.jpg)
Resilience
![Page 39: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/39.jpg)
Isolation
![Page 40: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/40.jpg)
• System must not fail as a whole
• Divide system in failure units (a.k.a. bulkheads)
• Avoid error propagation by isolating failure units
• Define fallback strategy
![Page 41: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/41.jpg)
Redundancy
![Page 42: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/42.jpg)
• Elaborate use caseMinimize MTTR / avoid latency / handle response errors / …
• Define routing & distribution strategy Round robin / master-slave / fan-out & quickest one wins / …
• Consider admin involvementAutomatic vs. manual / notification – monitoring / …
![Page 43: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/43.jpg)
Loose Coupling
![Page 44: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/44.jpg)
• Isolate failure units (complements bulkheads)
• Go asynchronous wherever possible
• Use timeouts & circuit breakers
• Make actions idempotent
![Page 45: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/45.jpg)
Implementation Example #1
Timeouts
![Page 46: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/46.jpg)
Timeouts (1) // Basics myObject.wait(); // Do not use this by default myObject.wait(TIMEOUT); // Better use this // Some more basics myThread.join(); // Do not use this by default myThread.join(TIMEOUT); // Better use this
![Page 47: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/47.jpg)
Timeouts (2) // Using the Java concurrent library Callable<MyActionResult> myAction = <My Blocking Action> ExecutorService executor = Executors.newSingleThreadExecutor(); Future<MyActionResult> future = executor.submit(myAction); MyActionResult result = null; try { result = future.get(); // Do not use this by default result = future.get(TIMEOUT, TIMEUNIT); // Better use this } catch (TimeoutException e) { // Only thrown if timeouts are used ... } catch (...) { ... }
![Page 48: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/48.jpg)
Timeouts (3) // Using Guava SimpleTimeLimiter Callable<MyActionResult> myAction = <My Blocking Action> SimpleTimeLimiter limiter = new SimpleTimeLimiter(); MyActionResult result = null; try { result = limiter.callWithTimeout(myAction, TIMEOUT, TIMEUNIT, false); } catch (UncheckedTimeoutException e) { ... } catch (...) { ... }
![Page 49: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/49.jpg)
Implementation Example #2
Circuit Breaker
![Page 50: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/50.jpg)
Circuit Breaker – concept
Client Resource Circuit Breaker
Request
Resource unavailable
Resource available
Closed Open
Half-Open
Lifecycle
![Page 51: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/51.jpg)
![Page 52: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/52.jpg)
Implemented patterns
• Timeout
• Circuit breaker
• Load shedder
• Fallback
![Page 53: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/53.jpg)
Supported patterns • Bulkheads
(a.k.a. Failure Units)
• Fail fast
• Fail silently
• Graceful degradation of service
• Failover
• Escalation
• Retry
• ...
![Page 54: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/54.jpg)
Hello, world!
![Page 55: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/55.jpg)
// Hystrix “Hello world” public class HelloCommand extends HystrixCommand<String> { private static final String COMMAND_GROUP = ”Hello”; // Not important here private final String name; // Request parameters are passed in as constructor parameters public HelloCommand(String name) { super(HystrixCommandGroupKey.Factory.asKey(COMMAND_GROUP)); this.name = name; } @Override protected String run() throws Exception { // Usually here would be the resource call that needs to be guarded return "Hello, " + name; } } // Usage of a Hystrix command – synchronous variant @Test public void shouldGreetWorld() { String result = new HelloCommand("World").execute(); assertEquals("Hello, World", result); }
![Page 56: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/56.jpg)
Source: https://github.com/Netflix/Hystrix/wiki/How-it-Works
![Page 57: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/57.jpg)
Fallbacks
![Page 58: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/58.jpg)
• What will you do if a request fails?
• Consider failure handling from the very beginning
• Supplement with general failure handling strategies
![Page 59: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/59.jpg)
Scalability
![Page 60: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/60.jpg)
• Define scaling strategy
• Think full stack
• Design for elasticity
• At least apply D-I-D rule
![Page 61: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/61.jpg)
Transparency
![Page 62: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/62.jpg)
Monitoring
(Transparency)
![Page 63: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/63.jpg)
• Think about required metrics
• Provide correlation Ids
• Design hook or event mechanism
• Plan for changing metrics
![Page 64: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/64.jpg)
Logging
(Transparency)
![Page 65: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/65.jpg)
• Consider log message structureAssume centralized logging: required information / machine readable / human readable
• Define logging policy Debug and less: developers perspective / Info and more: operations perspective
![Page 66: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/66.jpg)
11 Design Principles • Manageability
• Deployment • Configuration • Configuration Parameter Types • Backup
• Resilience • Isolation • Redundancy • Loose Coupling • Fallbacks • Scalability
• Transparency • Monitoring • Logging
![Page 67: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/67.jpg)
Don’t forget toread the “bible” of production-ready software … https://pragprog.com/book/mnee/release-it
![Page 68: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/68.jpg)
Wrap-up • The importance of “production readiness”
• The 3 ways of DevOps
• The needs of Ops
• The resulting challenges for Dev
• Design principles to support the needs • Manageability • Resilience • Transparency
![Page 69: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/69.jpg)
It’s all about production!
![Page 70: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/70.jpg)
@ufried Uwe Friedrichsen | [email protected] | http://slideshare.net/ufried | http://ufried.tumblr.com
![Page 71: Production-ready Software](https://reader031.fdocuments.net/reader031/viewer/2022030305/587156241a28ab8e5b8b5335/html5/thumbnails/71.jpg)