Do's and don'ts when deploying akka in production
description
Transcript of Do's and don'ts when deploying akka in production
Deploying Akka in Production
Do’s and Don’ts when
Mike Nash
Introduction
We’ve deployed dozens of Akka-based services and apps to production in the past 5 years
We’ve scaled up to tens of millions of events per hour, day after day
Our team has learned where the potholes are
Architecting for Performance
Going reactive before the first line of code is written
Code Quality and ArchitectureQuality = lower cost of future changes
Correct architecture = longevity, e.g the system lives on even if parts are replaced in flight
Many small pieces usually work better than a few big ones
Find out about Ports and Adapters/Onion Architecture: Keep domain and infrastructure distinct
Delivery Guarantees
Understand what guarantees Akka provides - and what it doesn’t
The best design is often the one that doesn’t need many guarantees
Most “guarantees” are imaginary in any case
Single Points of ContentionDatabase?
Blocking OperationsLook for blocking operations
ImmutabilityEncapsulate state in Actors (or Agents)
Messages must be immutable
Keep it Simple
Just because you can doesn’t mean you should
Simple, independent pieces are best
Keep your code out of other people’s databases
Distinguish between Commands and Events
Think about back-pressureThink about back pressure: Look for any situations where mailboxes might be allowed to grow without limits
Slow producer is not a big problem
Slow consumer can be a big problem
Akka streams
Think about dispatchers and routers: The defaults are not always what you want.
SerializationThink about for remote actors
Default requires some class path magic
Choices include text, json, protobuf, many others, all with tradeoffs
Benchmark your serializer/deserializer!
Actor? Future? Agent?Actors encapsulate State or Behaviour, they have a mailbox and a name
Futures contain and retrieve the result of a concurrent operation
Agents encapsulate a value (at a single location) - immediate read, async writes
TestingCommon performance blockers under heavy load
Testing
Beyond the Obvious
Unit and Functional are the Foundation
Akka TestKit: Test actors as if they were synchronous
“Negative” tests are essential
JVM Tuning
MemoryYou need more than you think you need
Know your Garbage CollectorsMust run under load to see characteristics
Defaults are probably not what you want
Takes more memory to free up more memory…
-verbosegc is your friend
Production DeploymentsTips for a rock-solid deployment pipeline
Deployment Pipeline
Not just about getting to production
How will you deploy new versions without downtime?
How will you revert if you have to?
It’s 2 am - Do you know where your dependencies are?
Deployment PackagingHow will you get your app to production? Onejar?
In our experience, external containers are not helpful
How will you tell if its the right version and revision?
Rolling Restarts?
Dependencies
Your application or service sits atop a tall stack of dependencies
Is the stack in the same order on production?
Is it the same stack?
Configuration
Akka’s configuration works great - don’t fix it!
Discovery beats hard-wiring
Procrastinate
MonitoringBeyond Testing
MonitMore than monitoring
Statsd, GraphiteCustom timing and counts
KamonSpecifically for Reactive Applications
What happens when….?
Boundary Conditions
LoggingRelated to monitoring, but very different Surprisingly expensive - consider async, macros (like ScalaLogging) to omit when level disabled Line numbers are *very* expensive, check your formats Log rolling strategy is critical Consider Graylog or similar to view logs across multiple nodes, filter, route and alert on log contents Consider ssh server or remote message to change log level on the fly for a running system Logback’s Mapped Diagnostic Context can be handy
Security
Beyond the usual security concerns any application has
How will you prevent “fake” messages from being handled?
There is such a thing as too much of a good thing
Summary