Building large scale, job processing systems with Scala Akka Actor framework
-
Upload
vignesh-sukumar -
Category
Technology
-
view
10.160 -
download
1
description
Transcript of Building large scale, job processing systems with Scala Akka Actor framework
Building massive scale, fault tolerant,
job processing systems with Scala Akka
frameworkVignesh Sukumar
SVCC 2012
About me
• Storage group, Backend Engineering at Box• Love enterprise software! • Interested in Big Data and building distributed
systems in the cloud
About Box
• Leader in enterprise cloud collaboration and storage
• Cutting-edge work in backend, frontend, platform and engineering services
• A really fun place to work – we have a long slide!
Talk outline• Job processing requirements• Traditional & new models for job processing
• Akka actors framework• Achieving and controlling high IO throughput• Fine-grained fault tolerance
Typical architecture in a cloud storage environment
Practical realities
•Storage nodes are usually of varying configurations (OS, processing power, storage capacity, etc) mainly because of rapid evolution in provisioning operations•Some nodes are more over-worked than the others (for ex, accepting live uploads)•Billions of files; petabytes
Job processing requirements
• Iterate over all files (billions, petabyte scale): for ex, check consistency of all files
• High throughput
• Fault tolerant
• Secure
Traditional job processing model
Why traditional models fail in cloud storage environments
• Not scalable: petabyte scale, billions of files• Insecure: cannot move files out of storage
nodes• No performance control: easy to overwhelm
any storage node• No fine grained fault tolerance
Compute on Storage
• Move job computation directly to storage nodes
• Utilize abundant CPU on storage nodes• Metadata store still stays in a highly available
system like a RDBMS • Results from operations on a file are
completely independent
Master – slave architecture
Benefits
• High IO throughput: Direct access; no transfer of files over a network
• Secure: files do not leave storage nodes• Better performance control: compute can
easily monitor system load and back off• Better fault tolerance handling: finer grained
handling of errors
Master node
• Responsible for accepting job submissions and splitting them to tasks for slave nodes
• Stateful: keeps durable copy of jobs and tasks in Zookeeper
• Horizontally scalable: service can be run on multiple nodes
Agent
• Runs directly on the storage nodes on a machine-independent JVM container
• Stateless: no task state is maintained• Monitors system load with back-off• Reports results directly to master without
synchronizing with other agents
Implementation with thethe Scala Akka Actor
framework
Actors
• Concurrent threads abstraction with no shared state
• Exchange messages• Asynchronous, non-blocking• Multiple actors can map to a single OS thread• Parent-children hierarchical relationship
Actors and messages• Class MyActor extends Actor { def receive = { case MsgType1 => // do something }}
// instantiation and sending messages val actorRef = system.actorOf(Props(new MyActor))actorRef ! MsgType1
Agent Actor System
Achieving high IO throughput• Parallel, asynchronous IO through “Futures” val fileIOResult = Future { // issue high latency tasks like file IO } val networkIOResult = Future { // read from network }
Futures.awaitAll(<wait time>, fileIOResult, networkIOResult)fileIOResult onSuccess { // do something } networkIOResult onFailure { // retry }
Controlling system throughput
• The problem: agents need to throttle themselves as storage nodes serve live traffic
• Adjust number of parallel workers dynamically through a monitoring service
Controlling throughput: Examples
•Parallelism parameters can be gotten from a separate configuration service on a per node basis•Some machines can be speeded up and others slowed down this way•The configuration can be updated on a cron schedule to speed up during weekends
Fine grained fault tolerance with Supervisors
• Parents of child actors can define specific fault-handling strategies for each failure scenario in their children
• Components can fail gracefully without affecting the entire system
Supervision strategy: Examples
Class TaskActor extends Actor { // create child workers override val supervisorStrategy = OneForOneStrategy(maxNrOrRetries = 3) { case SqlException => Resume // retry the same file case FileCorruptionException => Stop // don’t clobber it! case IOException => Restart // report and move on}
Unit testing
• Scalatra test framework: very easy to read! TaskActorTest.receive(BadFileMsg) must throw
FileNotFoundException• Mocks for network and database calls val mockHttp = mock[HttpExecutor] TaskActorTest ! doHttpPost there was atLeastOne(mockHttp).POST
• Extensive testing of failure injection scenarios
Takeaways• Keep your architecture simple by modeling
actor message flow along the same paths as parent-child actor hierarchy (i.e., no message exchange between peer child actors)
• Design and implement for component failures• Write unit tests extensively: we did not have
any fundamental level functionality breakage• Box Engineering is awesome!