Fallacy of Fast
Transcript of Fallacy of Fast
![Page 1: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/1.jpg)
the FallacyOf Fast
![Page 2: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/2.jpg)
INES Sombra
@Randommood
![Page 3: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/3.jpg)
![Page 4: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/4.jpg)
Obligatory DisclaimerThings you will see in this talk
Fast Talking & Opinions TM
Un-tweetable moments
Rantifestos TM
Questionable language
A therapy llama
and ZERO Kevin Bacons
@Randommood
![Page 5: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/5.jpg)
Common Mistakes
What Matters
@Randommood
![Page 6: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/6.jpg)
SPOILER alert
@Randommood
![Page 7: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/7.jpg)
@Randommood
![Page 8: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/8.jpg)
Reason about design choices in terms of
trade-offs @Randommood
![Page 9: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/9.jpg)
chosen trade-offs make the foundation
of your system@Randommood
![Page 10: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/10.jpg)
@Randommood
![Page 11: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/11.jpg)
@Randommood
![Page 12: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/12.jpg)
@Randommood
![Page 13: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/13.jpg)
@Randommood
![Page 14: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/14.jpg)
Common Mistakes
![Page 15: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/15.jpg)
Accidentally de-emphasizing
long term quality & system Stability
@Randommood
![Page 16: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/16.jpg)
De-prioritizing TestingCutting corners on testing carries a hidden cost
Test the full system: client, code, & provisioning code
Code reviews != tests. Have both
Continuous Integration (CI) is critical to velocity, quality, & transparency
![Page 17: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/17.jpg)
De-prioritizing ReleasesRelease stability is tied to system stability
Iron out your deploy process!
Dependencies on other systems make this even more important
Canary testing, dark launches, feature flags, etc are good
@Randommood
![Page 18: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/18.jpg)
Automation shortcuts taken while in a rush will come back to haunt you
Playbooks are a must have
Localhost is the devil
Sloppy operational work is the mother of all evils
De-prioritizing Ops
!
@Randommood
![Page 19: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/19.jpg)
“Future you monitoring” is bad, make it part of MVP
Alert fatigue has a high cost, don’t let it get that far
Link alerts to playbook
Routinely test your escalation paths
De-prioritizing Insight
✨
@Randommood
![Page 20: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/20.jpg)
The inner workings of data components matter. Learn about them
System boundaries ought to be made explicit
Deprecate your go-to person
De-prioritizing Knowledge
@Randommood
![Page 21: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/21.jpg)
The internet is an awful place
Expect DoS/DDoS
Think about your system, its connections, and their dependencies
Having the ability to turn off features/clients helps
De-prioritizing Security
![Page 22: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/22.jpg)
Service ownership implies leveling-up operationally
Architectural choices made in a rush can have a long shelf life
Don’t sacrifice tests. Test the FULL system
What we learned
✨@Randommood
![Page 23: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/23.jpg)
RANTIFESTO!
@Randommood
![Page 24: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/24.jpg)
building shrines of AgileAssuming a given methodology will solve everything is naive at best
Magical thinking leads to misaligned expectations
All tools are terrible, avoid religious wars #
![Page 25: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/25.jpg)
Anarchy
Complex
Complicated
Complicated
Simple
Clos
e to a
gree
ment
far
from
agre
emen
t
Close to certainty far from certainty
* “When to Scrum?” stolen from Angela Druckman
Requirements
Technology
Scrum
![Page 26: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/26.jpg)
@Randommood
![Page 27: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/27.jpg)
@Randommood
![Page 28: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/28.jpg)
@Randommood
![Page 29: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/29.jpg)
@Randommood
$%&%
'''''''' '
![Page 30: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/30.jpg)
( (
@Randommood
![Page 31: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/31.jpg)
“In truth a range of approaches, a hybrid mix, of management methods is
required to succeed in today's enterprise IT environment. That customer
enterprise environment never was like the simplified product development environment where Agile software development was conceived…”
@Randommood
![Page 32: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/32.jpg)
“In truth a range of approaches, a hybrid mix, of management methods is
required to succeed in today's enterprise IT environment. That customer
enterprise environment never was like the simplified product development environment where Agile software development was conceived…”
@RandommoodDUH
![Page 33: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/33.jpg)
Agile GotchasUncertainty in problem domain (and company size) will challenge your ability to adhere to it
Has a cost but it’s different
Nihilism FTW?
![Page 34: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/34.jpg)
)?
@Randommood
![Page 35: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/35.jpg)
WhatMatters
@Randommood
![Page 36: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/36.jpg)
Mind system DesignSimple & utilitarian design takes you a long way
Use well understood components
NIH is a double edged sword
Use feature flags & on/off switches (test them!)
@Randommood
![Page 37: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/37.jpg)
Meet AliceI’m way too
cool for this outfit
![Page 38: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/38.jpg)
Alice’s Testing AreasCorrectness Error Performance Robustness
Good output from good
inputs
Reasonable reaction to incorrect
input
Time to Task (TTT) for Behavior after
Go
al Single node
Multi node
Clustered
Cache enabled
Given # of input/outputs
Given uptime
@Randommood
![Page 39: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/39.jpg)
a Testing HarnessIs a fantastic thing to have
Invest in QA automation engineers
Adding support for regressions & domain-specific testing pays off
@Randommood
![Page 40: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/40.jpg)
Mind system ConfigsSystem assumptions are dangerous, make them explicit
Standardize system configuration (data bags, config file, etc)
Hardcoding is the devil
![Page 41: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/41.jpg)
Mind system LimitsRate limit your API calls especially if they are public or expensive to run
Instrument / add metrics to track them
Rank your services & data (what can you drop?)
Capacity analysis is not dead
✨
![Page 42: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/42.jpg)
Mind system GrowthWatch out for initial over-architecting
“The application that takes you to 100k users is not the same one that takes you to 1M, and so on…” @netik
Expect changes & refactors
@Randommood
![Page 43: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/43.jpg)
Mind ProcessArchitectural reviews FTW
Request flow, API shape, Failure conditions, Reliability, Data Model, Threat modeling, Testing strategy, Operations, Monitor logging & Alerting, Pricing/Billing, Supported clients, etc
@Randommood
![Page 44: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/44.jpg)
Mind ResourcesRedundancies of resources, execution paths, checks, replication of data, replay of messages, anti-entropy build resilience
Mechanisms to guard system resources are good to have
Your system is also tied to the resources of its dependencies
![Page 45: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/45.jpg)
Distrust is healthyDistrust client behavior, even if they are internal
Decisions have an expiration date. Periodically re-evaluate them as past you was much dumber
A revisionist culture produces more resilient systems
✨
@Randommood
![Page 46: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/46.jpg)
about Resilience
Traditional engineering
Reactive ops unk-unk
* Stolen from Paul Borrill
Cascading or catastrophic failures & you don’t know where they will come from! Same area as other 2 combined
Probability of failure
Rank
![Page 47: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/47.jpg)
classical Engineering
reactive Operations unk-unk
Building Resilience
Code standards
Programming patterns
Testing (full system!)
Metrics & monitoring
Convergence to good state
Hazard inventories
Redundancies
Feature flags
Dark deploys
Runbooks & docs
Canaries
System verification
Formal methods
Fault injection
The goal is to build failure domain independence
![Page 48: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/48.jpg)
Keep track of your technical debt & repay it regularly
It’s about lowering the risk of change with tools & culture
Mind assumptions
What we learned
✨
@Randommood
![Page 49: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/49.jpg)
TL;DR
Easy to sacrifice things may be harder to
correct later
Think in terms of tradeoffs
TESTING MATTERS!
Not all process is evil
Keep in MindMake system boundaries &
dependencies explicit
Playbooks are your friends, have them
Use kill switches & limits
Prioritize your services
Distributed systems
![Page 50: Fallacy of Fast](https://reader030.fdocuments.net/reader030/viewer/2022011722/58eeed0c1a28abe23e8b45a9/html5/thumbnails/50.jpg)
github.com/Randommood/FallacyOfFast
@Randommood