Akka - Developing SEDA Based Applications
-
Upload
benjamin-darfler -
Category
Technology
-
view
6.096 -
download
0
description
Transcript of Akka - Developing SEDA Based Applications
![Page 1: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/1.jpg)
AkkaDeveloping SEDA Based
Applications
![Page 2: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/2.jpg)
Me Ben Darfler
@bdarfler http://bdarfler.com Senior Software Engineer at Localytics
![Page 3: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/3.jpg)
Localytics Real time mobile analytics platform
40M+ events per day and growing rapidly
3x growth over the past 3 months
Heavy users of Scala/Akka/NoSql We are hiring (seriously, come talk to me)
![Page 4: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/4.jpg)
Localytics
How to keep up with our growth?
![Page 5: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/5.jpg)
Actor Model Lock free approach to concurrency No shared state between actors Asynchronous message passing Mailboxes to buffer incoming messages
![Page 6: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/6.jpg)
AkkaConfigurable● Dispatchers● Mailboxes
Fault Tolerant● Supervisors
Great community● @jboner● @viktorklang
![Page 7: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/7.jpg)
AkkaPerformant
http://blog.jayway.com/2010/08/10/yet-another-akka-benchmark/
![Page 8: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/8.jpg)
SEDAStaged Event Driven Architecture
"Decomposes a complex, event-driven application into a set of stages connected by queues."1
"The most fundamental aspect of the SEDA architecture is the programming model that supports stage-level backpressure and load management."1
1. http://www.eecs.harvard.edu/~mdw/proj/seda/
![Page 9: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/9.jpg)
Backpressure
Whats the big deal?
![Page 10: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/10.jpg)
BackpressureManditory to prevent OutOfMemoryError● Messages backup in memory faster than they
can be processed
Cassandra was seriously bitten by this● Less crappy failure mode when swamped with
inserts than "run out of memory and gc-storm to death" (CASSANDRA-401)
● Add backpressure to StorageProxy (CASSANDRA-685)
![Page 11: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/11.jpg)
BackpressureMailboxes
case class UnboundedMailbox(val blocking: Boolean = false) extends MailboxType
case class BoundedMailbox( val blocking: Boolean = false, val capacity: Int = { if (Dispatchers.MAILBOX_CAPACITY < 0) Int.MaxValue else Dispatchers.MAILBOX_CAPACITY }, val pushTimeOut: Duration = Dispatchers.MAILBOX_PUSH_TIME_OUT) extends MailboxType
BoundedMailbox(false, QUEUE_SIZE, Duration(-1, "millis"))
Backpressure Mailbox
![Page 12: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/12.jpg)
Stages
How do we decompose the problem?
![Page 13: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/13.jpg)
StagesOne actor class per stage
Shared dispatcher
Individually tunable● I/O Bound● CPU Bound
Easier to reason about
Code reuse
![Page 14: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/14.jpg)
DispatchersThreadBasedDispatcher
● Binds one actor to its own thread
ExecutorBasedEventDrivenDispatcher● Must be shared between actors
ExecutorBasedEventDrivenWorkStealingDispatcher● Must be shared between actors of the same type
![Page 15: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/15.jpg)
Queues
SEDA has a queue per stage model
Akka actors have their own mailbox
How do we evenly distribute work?
![Page 16: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/16.jpg)
Work StealingExecutorBasedEventDrivenWorkStealingDispatcher
"Actors of the same type can be set up to share this dispatcher and during execution time the different actors will steal messages from other actors if they have less messages to process"1
1. http://doc.akka.io/dispatchers-scala
![Page 17: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/17.jpg)
Work StealingReally a work "donating" dispatcher
�"I have implemented a work stealing dispatcher for Akka actors. Although its called "work stealing" the implementation actually behaves more as "work donating" because the victim actor takes the initiative. I.e. it actually donates work to its thief, rather than having the thief steal work from the victim."1
1. http://janvanbesien.blogspot.com/2010/03/load-balancing-actors-with-work.html
![Page 18: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/18.jpg)
Work Stealing
Doesn't that conflict with blocking mailboxes?
![Page 19: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/19.jpg)
Work StealingSending actor will block on the receiving actors mailbox before it can "donate"
Might be fixed in Akka 1.1● I owe @viktorklang a test of his latest changes
![Page 20: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/20.jpg)
Load Balancing
Can we distribute work on the sender side?
![Page 21: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/21.jpg)
Load BalancingRouting.loadBalancerActor()● Creates a new actor that forwards
messages in a load balancing fashion
InfiniteIterator● CyclicIterator● SmallestMailboxFirstIterator
![Page 22: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/22.jpg)
Load Balancing
Doesn't the load balancer need a blocking mailbox?
![Page 23: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/23.jpg)
Load BalancingCan't easily change the load balancer's mailbox
Use SmallestMailboxFirstIterator directly
new SmallestMailboxFirstIterator(List(actor, actor, actor))
![Page 24: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/24.jpg)
Fault ToleranceSupervisors
● Restarts actors● Stops after x times within y milliseconds
Restart Strategies● OneForOne● AllForOne
![Page 25: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/25.jpg)
Fault ToleranceGreat for transient issues
● Network failures
Not great for permanent issues● OutOfMemoryError
![Page 26: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/26.jpg)
Final Product// Actor creationval supervisor = Supervisor(SupervisorConfig( OneForOneStrategy(List(classOf[Exception]), RETRIES, WITH_IN_TIME), Supervise(myActors))
def myActors: List[Supervise] = { val mailbox = BoundedMailbox(false, QUEUE_SIZE, Duration(-1, "millis")) val dispatcher = Dispatchers.newExecutorBasedEventDrivenDispatcher( "my-dispatcher", 1, mailbox).setCorePoolSize(POOL_SIZE).build (1 to POOL_SIZE toList).foldRight(List[Supervise]()) { (i, list) => Supervise(actorOf(new MyActor("my-actor-" + i, dispatcher)), Permanent) :: list }}
// Sending a messageval actors = new SmallestMailboxFirstIterator(actorsFor(classOf[MyActor]).toList)def actor = actors.nextactor ! Message()
![Page 27: Akka - Developing SEDA Based Applications](https://reader033.fdocuments.net/reader033/viewer/2022051609/546d0a05af795976298b521e/html5/thumbnails/27.jpg)
Thanks
@bdarflerhttp://bdarfler.com