Testing & Integration (The Remix)
-
Upload
ines-sombra -
Category
Engineering
-
view
272 -
download
2
Transcript of Testing & Integration (The Remix)
![Page 1: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/1.jpg)
Testing & Integration
THE REMIX!
![Page 3: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/3.jpg)
Globally distributed & highly available
![Page 4: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/4.jpg)
DS TestingContinuous IntegrationBring Home
"
!
$
Our AgendaDS =
Distributed Systems
![Page 5: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/5.jpg)
Visual Guide ^
Distributed Systems Testing
" $!
![Page 6: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/6.jpg)
Why do we test?We test to gain confidence that our system is doing the right thing (now & later)
" $!
![Page 7: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/7.jpg)
MEDIUM
Unit Integration System Acceptance
SMALL
Unit Integration (maybe)
COMPLEX SYSTEMFault injection Stress / performance Canary Regression
Unit Integration System Acceptance Compatibility
Many Types of Tests " $!
![Page 8: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/8.jpg)
Nondeterminism
TimingFailures
Unbounded inputs
Ordering
Many states
Concurrency
No centralized view
Challenges of DS Testing " $!
![Page 9: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/9.jpg)
Behavior is aggregate
Components tested in isolation also need to be tested together
?
!
A
!
B
" $!Challenges of DS Testing
![Page 10: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/10.jpg)
Reliability of a linear system is the product of the reliability of each of the
system’s components
" $!Challenges of DS Testing
![Page 11: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/11.jpg)
.95 * .95 * .95 = .85 Your system is only
85% reliable
Component A
95% reliable
!
Component B
95% reliable
!
Component C
95% reliable
!
" $!Challenges of DS Testing
![Page 12: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/12.jpg)
Hierarchy of Errors*BYZANTINE FAILURES
(fail by doing whatever I want)
OMISSION FAILURES
(fail by dropping messages)
CRASH FAILURES
(fail by stopping)
* Stolen from Henry Robinson’s PWLSF talk
Deadlocks
Livelock / starvation
Under specification
Over specification
" $!
![Page 13: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/13.jpg)
Distributed Systems TestingDifficult to approach & many factors in play
Behavior is aggregate
Goal is to gain confidence of proper system behavior now & later
" $!
![Page 14: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/14.jpg)
Continuous Integration
CI for short
" $!
![Page 15: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/15.jpg)
Waiting till the end of a project to integrate leads to 🔥🔥 hell 🔥🔥 (a special kind)
" $!
![Page 16: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/16.jpg)
What is CI?Development practice
Integrate code to master branch daily
Each integration is automatically verified
" $!
![Page 17: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/17.jpg)
Why CI Today?Often overlooked/ Taken for granted
“Eat your veggies”
Impacts your risk & quality
" $!
![Page 18: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/18.jpg)
Commit frequently
100% test must pass
Fix broken tests
Automation is 👑
CI Practices" $!
![Page 19: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/19.jpg)
CI gives usRapid feedback
Rapid deployment
Repeatable automated testing
Reduce risk
" $!
![Page 20: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/20.jpg)
Revisiting Tests
Differentiated by required setup
which affects running time
Unit
Integration
System
Functional (also Smoke /Acceptance)
" $!
![Page 21: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/21.jpg)
CI ChallengesComplex systems / dependencies
Multiple architectures
Type of applications & types of tests
" $!
![Page 22: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/22.jpg)
CI Challenges " $!
Overfitted/ brittle / slow tests
Writing good tests is hard
Maintenance
Consider Regulations (PCI)
![Page 23: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/23.jpg)
“Manage the CI
System or be
controlled by manual processes”
Is it worth it?
" $!
![Page 24: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/24.jpg)
You integrate to master very often (daily)
All of your tests pass & you fix broken builds!
Success indicators: # commits / build status
Am I doing CI right? " $!
![Page 25: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/25.jpg)
CI In the Wild
" $!
![Page 26: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/26.jpg)
Tools & Services
How to pick?
Security, Cost
Language support
Services & integrations
Scalability
" $!
![Page 27: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/27.jpg)
" $!
![Page 28: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/28.jpg)
CI In the WildBranches & pull requests. Merges & deploys
Consolidated view of build statuses across all codebases
" $!
![Page 29: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/29.jpg)
Master PRs
Kick off a suite
StacksOS + Our
Image
ScenarioLive setup + assertions
SuiteCollection of
scenarios
My Assertions: Installing, backing
up/restoring, setting up, replicating,
promoting, etc
Any prod deploy
Kicks off a suite
" $!Dredd TestsSystems,
boundaries, & integration
CI In the Wild
![Page 30: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/30.jpg)
CI In the Wild " $!
* Stolen from Google’s talk on their Continuous Build System
![Page 31: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/31.jpg)
" $!CI In the Wild
![Page 32: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/32.jpg)
* Slide stolen from Seth Falcon’s ChefConf 2015 talk
CI In the Wild: CD tools* " $!
![Page 33: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/33.jpg)
CI Matters Predictor of professional maturity at the organizational & individual level
Adds visibility & allows us to manage risk and quality
" $!
🏆
![Page 34: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/34.jpg)
How are we doing?
![Page 35: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/35.jpg)
Bringing it home
" $!
![Page 36: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/36.jpg)
" $!Getting Started
Test the full distributed system. This means testing the client, system, and provisioning code!
HAVE DECENT TESTS
ADD VISIBILITY
VERIFY & ENHANCE
![Page 37: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/37.jpg)
CI & Code QualityAutomated inspection as part of the build
Customizable by Org
Test coverage
Code duplication
" $!
![Page 38: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/38.jpg)
CI & Quality " $!
Languages help you: go cov, race detector,go lint
Infrastructure testing matters
But beware of misguided processes
😩
![Page 39: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/39.jpg)
CI & QualityCyclomatic Complexity Number (CNN)
Afferent/Efferent Coupling
Software instability as “quantifiable”
" $!
![Page 40: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/40.jpg)
And more! " $!
Use it for docs generation
Architectural/design diagrams too !
Accessibility testing
Used to test mobile architectures too
📄
![Page 41: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/41.jpg)
Pro-Tips " $!
Add CI early
Different machine for integrating & put everything you need to built in git
Parallelize builds to scale up or down
![Page 42: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/42.jpg)
Rapid feedback is critical. Slow is 😭💀
Staggered builds: run fast tests first, then second phase kicks in
The more likely a test is to fail the sooner you should execute it
" $!Pro-Tips
![Page 43: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/43.jpg)
Tl;DRDS TESTING
Different types of tests Can be tricky Behavior is aggregate
CI PRACTICES
Everyone commits to baseline daily Automate build & make it self-testing Keep builds fast Everyone can see builds & status
ON QUALITY
CI can be used to drive good coding standards Some metrics available to gauge stability / quality Keep track of them
![Page 44: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/44.jpg)
Rantifesto: Build a Culture of Quality
✨The technology is just a tool but your culture is really the key ✨
Lower the risk of change with tools & culture
" $!
![Page 45: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/45.jpg)
“Quality means doing it right when
no one is looking”
Henry Ford
![Page 46: Testing & Integration (The Remix)](https://reader031.fdocuments.net/reader031/viewer/2022012917/58aeb88e1a28ab00708b5ff7/html5/thumbnails/46.jpg)
Thank You!
Special thanks to: Caitie McCaffrey, Jordan West, Jon Hyman, Bruce Spang, Devon O’Dell, Kelsey Gilmore-Innis, Aysylu Greenberg, Alan Kasindorf, Paul Reed, André Arko, Mike O’Neill, Thomas Mahoney, Eric Kustarz, Ian Fung, Neha Narula, Karl Smith, and Greg Bako.
[email protected] | @Randommood | github.com/Randommood/CraftConf2015