Microservice Antipatterns QCon.key
Transcript of Microservice Antipatterns QCon.key
![Page 1: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/1.jpg)
Quick poll…
![Page 2: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/2.jpg)
Welcome to sunny London!
![Page 3: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/3.jpg)
Tammer Saleh
Geek: Unix, Ruby, Golang, etc
Cloud Foundry @ Pivotal
http://tammersaleh.com | [email protected]
![Page 4: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/4.jpg)
MicroserviceAnti-patterns
How not to go down in flames.
![Page 5: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/5.jpg)
Why microservices? What is a microservice, and why do I care?
![Page 6: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/6.jpg)
Monolithic
Entire application in a single codebase, deployed and scaled as a single unit.
![Page 7: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/7.jpg)
Monolithic
Hard to scale the application. Impossible to scale the team.
![Page 8: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/8.jpg)
It’s not about code… It’s about teams.
![Page 9: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/9.jpg)
But it can go wrong.Here are the most common problems we
see in the wild, and how to fix them.
![Page 10: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/10.jpg)
Overzealous ServicesThe most common
mistake is to start with
microservices
![Page 11: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/11.jpg)
Boring is Beautiful™
![Page 12: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/12.jpg)
Solution: Start monolithic and extract
Microservices are complex and add a constant tax to development.
Build a boring application and extract services as needed.
![Page 13: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/13.jpg)
/tweet
/dashboard
/…
/…
Heavy and Expensive
![Page 14: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/14.jpg)
/tweet
/dashboard
/…
/…
Lite and Cheap
![Page 15: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/15.jpg)
Congratulations You’re now a microservice architect.
![Page 16: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/16.jpg)
Schemas everywhere
/dashboard
/…
/…
/tweet v1
v1
V1 Schema
Deploy v2
![Page 17: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/17.jpg)
Schemas everywhere
/dashboard
/…
/…
/tweet v1
v2
V1 Schema
Deploy v2
Migrate v2
![Page 18: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/18.jpg)
/dashboard
/…
/…
/tweet v1
v2
V2 Schema
Deploy v2
Schemas everywhere
Migrate v2
![Page 19: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/19.jpg)
/dashboard
/…
/…
/tweet v2
v2
V2 Schema
Schemas everywhere
![Page 20: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/20.jpg)
Solution: Gatekeeper
/dashboard
/…
/…
/tweet v1
v2
Tweets
/bulk_add
Tweet Service
GET /tweets
GET /tweets/ID
Owns database and migrations
![Page 21: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/21.jpg)
Lock-step deployment
Service B
Service A
Service C — v1
v1
v1
![Page 22: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/22.jpg)
Lock-step deployment
Service B
Service A
Service C — v2
v1
v1
Deploy v2
![Page 23: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/23.jpg)
Lock-step deployment
Service B
Service A
Service C — v2
v1
v1
Deploy v2
![Page 24: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/24.jpg)
Lock-step deployment
Service B
Service A
Service C — v2
v2
v2
Deploy v2
![Page 25: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/25.jpg)
Solution: Semantic Versioning
Service B
Service A
Service C — v1
v1
v1
vMajor.Minor.Patch MYBAD.SHINY.OOPS
![Page 26: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/26.jpg)
Solution: Semantic Versioning
Service B
Service A
Service C — v1.2
v1
v1
Deploy v1.2(extra stuff)
:)
:)
![Page 27: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/27.jpg)
Solution: Semantic Versioning
Service B
Service A
Service C — v1.2
v1.2
v1.2
Deploy v1.2
![Page 28: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/28.jpg)
Solution: Semantic Versioning
Service B
Service A
Service C — v1.2
v1.2
v1.2
Deploy v2(breaking change)
Service C — v2
![Page 29: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/29.jpg)
Solution: Semantic Versioning
Service B
Service A
Service C — v1.2
v1.2
v2Service C — v2
Deploy v2
![Page 30: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/30.jpg)
Solution: Semantic Versioning
Service B
Service A
Service C — v1.2
v2
v2Service C — v2
Deploy v2
![Page 31: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/31.jpg)
OMG ALL THE STEPS!!!
See Rule #1
![Page 32: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/32.jpg)
Spiky load between services
Must maintain enough servers to handle peak load.
![Page 33: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/33.jpg)
/tweet
/dashboard
/…
/…
Spiky load between services
![Page 34: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/34.jpg)
/tweet
/dashboard
/…
/…
Spiky load between services
![Page 35: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/35.jpg)
Queues in between services provide buffers that smooth traffic.
Solution: Amortize via queues
Peak load is now much lower.
![Page 36: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/36.jpg)
/tweet
/dashboard
/…
/…
Solution: Amortize via queues
![Page 37: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/37.jpg)
Solution: Amortize via queues/tweet
/dashboard
/…
/…
Worker
Complexity: Now clients must deal with asynchronous responses.
![Page 38: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/38.jpg)
Hardcoded IPs and Ports
Simple to get started, but immediately leads to deployment issues.
![Page 39: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/39.jpg)
Solution 1: Discovery Service
etcdconsul
![Page 40: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/40.jpg)
Solution 1: Discovery Service
Service A
Service B (192.168.0.2:1234)
Where is B?
192.168.0.2:1234
![Page 41: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/41.jpg)
Solution 1: Discovery Service
Service A
Service B (192.168.0.2:1234)
Hey there!
![Page 42: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/42.jpg)
Solution 1: Discovery Service
Complexity: Your code must understand the service lookup system.
![Page 43: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/43.jpg)
Solution 2: Centralised router
Service AService B
Router
Service B Service A Service AService B
A A BB A A BBA A BB
![Page 44: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/44.jpg)
Solution 2: Centralised router
Simplicity: “It’s just DNS.”
![Page 45: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/45.jpg)
Router vs Discovery Service
Both require service registration.
Discovery Service is simpler to build and scale, since it doesn’t need to
route all data.
Discovery service does fewer network hops.
Router is transparent.
Router can be exposed externally.
Router can cache transparently.
Router can round-robin.
Both require HA and scalability.
![Page 46: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/46.jpg)
Dogpiles
Service BService A
You there?
No.
How about now?
I’m busy.
But I wanna chat!
Go away
Whachya doin?
Please leave me alone.
Wanna be friends?
I hate you.
![Page 47: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/47.jpg)
Dogpiles
Service B
Service A Service A
Service A Service A
x 100
![Page 48: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/48.jpg)
Solution: Circuit Breaker
Service B
Service A Service A
Where’s B?
B’s sick. Wait 10 min.
Hello!
Go to Hell!
Hey, B’s sick :(
![Page 49: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/49.jpg)
Debugging hellTurns out, distributed systems are hard.
![Page 50: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/50.jpg)
Debugging Hell
Service 3
Service 1 Service 2
time:1428374783 source:service3 msg:”Error with payment!”
Where?? Why?? Who??
![Page 51: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/51.jpg)
Solution: Correlation IDs
Service
Service ServiceCID: 1234 CID: 1234 C
ID: 1234
time:1427127483 source:service1 id:1234 msg:”Received request…”time:1427348748 source:service2 id:1234 msg:”Processing payment"time:1428374783 source:service3 id:1234 msg:”Error with payment!”
![Page 52: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/52.jpg)
Solution: Correlation IDs1. Tag all incoming requests with
unique ID
2. Service saves ID for all incoming requests
3. Include that ID in all log lines, etc.
4. Tag new requests with that ID
ServiceLogsID
ID
ID
ID
Complexity: Must be done manually.
![Page 53: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/53.jpg)
Missing Mock ServersEach consuming team has to create
their own mocks and stubs.
![Page 54: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/54.jpg)
Missing Mock Servers
Service A
Team A
![Page 55: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/55.jpg)
Missing Mock Servers
Service A Service BHTTP
Team A Team B
Product Surface Area
![Page 56: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/56.jpg)
Missing Mock Servers
Service A Service BHTTP
Mock of Service A
HTT
P
Team A Team B
For each service… For each team…
![Page 57: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/57.jpg)
Service A Service BHTTP
Mock of Service A
HTTP
Team A Team B
Better…Solution: Service Team Provides the Mock
Team B still needs to know how to run Mock Service A
![Page 58: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/58.jpg)
Solution: Service Team Owns the Client
Service A Service BHTTP
HTTP
Team A Team B
Client A
MOCK=“true”
Best…
Mock of Service A
Product Surface Area
Both modes are tested in CI
Team A can change the protocol as they see fit
![Page 59: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/59.jpg)
Flying Blind
![Page 60: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/60.jpg)
Solution: Graphs, alerts, pages.
![Page 61: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/61.jpg)
Solution: Graphs, alerts, pages.
Alerting Rules EnginePlatform Events
OMGWAT???AppsAppsAppsAppsAppsAppsAppsAppsAppsAppsApps
![Page 62: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/62.jpg)
Snowflakes
![Page 63: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/63.jpg)
Snowflakes
Ruby XSS exploit!
BASH remote exploit!
![Page 64: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/64.jpg)
Snowflakes
Containers
Virtual Machines
![Page 65: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/65.jpg)
Solution: Golden Image
![Page 66: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/66.jpg)
Solution: Golden Image
Golden OS Image
Common Runtimes and Frameworks
+Base Platform
=
![Page 67: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/67.jpg)
Doomsday Deployments
![Page 68: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/68.jpg)
Solution: Predictable Pipelines
![Page 70: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/70.jpg)
Solution: Predictable Pipelines
Need to trust your tests, your platform, and your automation.
![Page 71: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/71.jpg)
Operational Explosion!
![Page 72: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/72.jpg)
Operational Explosion!Operations block Development
![Page 73: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/73.jpg)
Solution: AUTOMATE ALL THE THINGS!!!!!
Form a team to build tools that enable developers to manage the system in an entirely automated way.
![Page 74: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/74.jpg)
OMG ALLTHETHINGS????App deployment, infrastructure provisioning, OS installation, configuration management, database provisioning, disaster recovery, application monitoring, HA, blue-green deployments, self-healing, scaling, runtime installation, log rotation, backups, security updates, database upgrades, application logs, system logs, database logs, continuous integration, continuous deployment, service discovery, monitor queue usage, security monitoring, hotspot detection, error monitoring, issue notification and escalation, virtual machine migration, shard rebalancing, circuit breaker monitoring, resiliency testing, database snapshots, flux capacitors, ion overdrive maintenance, change the oil, dog feeding, cat shooting, pig eating…
![Page 75: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/75.jpg)
Time and Money
Solution: AUTOMATE ALL THE THINGS!!!!!
![Page 76: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/76.jpg)
![Page 77: Microservice Antipatterns QCon.key](https://reader034.fdocuments.net/reader034/viewer/2022042619/58a2eb321a28ab343e8b933e/html5/thumbnails/77.jpg)
In summary…Start boring and extract to services.
Understand the hidden schemas.
Amortize traffic with queues.
Decouple through discovery tools.
Contain failures with circuit breakers.
Enable other teams through mockable clients.
Kill your snowflakes.
Automate your deployments.
Build in operations tools from the beginning.
Make use of a platform like Cloud Foundry.