Camunda and Apache Cassandra
Embed Size (px)
Transcript of Camunda and Apache Cassandra
Stan Levine firstname.lastname@example.org
CONTEXTSPACE, CAMUNDA AND CASSANDRA
ContextSpace is a platform for executing secure digital customer
engagement on a very large scale.
Digital engagement requires consistency, a.k.a. Business Process
Digital engagement requires Big Identity, with many contextual
profiles per person and massive quantities of behavioral data
from customer interactions and mobile devices, hence Cassandra.
WHO IS CONTEXTSPACE
Founded July 2011
Tel Aviv, Israel
WHO IS CONTEXTSPACE DESIGNED FOR?
Very large communities of consumers, with many contextual relationships, typically found in: Health Care Telecom Smart Cities Digital Media Retail Energy Service Providers
TYPICAL CONTEXTSPACE USE OF CAMUNDA
ContextSpace employs Camunda as a toolset to allow its customers to develop consistent digital engagement business processes, with a common underlying API access to all services and data. We also create Camunda-based service processes to support highly specialized industrial patterns that require: Specific conditional processing Industry data formats (such as HL7 FHIR) Multiple service escalation paths Long running processes
SIMPLE HEALTHCARE PROCESS PATTERN: DIABETES MILLITUS TYPE 2 MONITORING
HIGH VELOCITY HEALTH OBSERVATION INGESTION
mHealth Camunda Processes Cassandra Persistence Analytics
WHY DOES CONTEXTSPACE USE CASSANDRA?
Exceptional performance, linearly scaled No DBAs, no tuning, fault tolerant, lights out operation Inherently multi-data centre aware for active-active
distributed operations Simple - replaces more than a dozen heterogeneous
technologies that our solution would otherwise require All ContextSpace data services are based on Cassandra, enabling reliable operations with low operational costs.
OPEN SOURCE CAMUNDA ON CASSANDRA
Begun in Berlin as hackathon session in July 2015 between Camunda and ContextSpace
Amazing Camunda Team! Maintained by ContextSpace Purpose: extend scalability, availability and
distributed processing to Camunda operations Initial goal: support core BPM Engine functions,
including Job Executor Target production launch: Q4 2015
Great for stable or immutable data Writes are faster than reads (both are very fast) Highly granular, tunable consistency Inherent data retention management Very reliable All nodes identical, no SPOF, no fail-over required No effective limit to quantity of managed data
No ad-hoc searches need to model your queries, not your data
Deleting data can be problematic. Deleting entire rows works well, deleting columns creates tombstones that can decrease performance and compromise stability.
Rapidly changing columnar data is an anti-pattern No native join operations Rudimentary locking support Rudimentary native indexing support
Out of the Box secondary indices are limited to low-cardinality data. These are useless or even dangerous to use.
Most developers employ custom indexing schemes very successfully. For Camunda, a simple, unsorted reverse lookup table will meet most requirements.
Camunda job scheduler requires sorted indices Therefore, we have developed a Cassandra indexing
framework for Camunda that supports both sorted and unsorted indexing.
Native Cassandra locking is rudimentary Atomic batches can be used for transactions (multiple
insert, update, delete in a single logical operation) However, if batch performs a lock, all statements can
only apply to one partition (row). With separate tables used for custom indices, scheduled
jobs, etc., we need to use multiple batches within one transaction, however, this breaks the atomicity.
LOCKING APPROACH AND LIMITATIONS
We lock an entire process, including executions, event subscriptions, and variables by placing all this data in one Cassandra partition (row).
Indices and jobs are still not updated atomically, leading to potential data integrity issues.
When locking the entire process, parallel (non-exclusive) tasks will execute sequentially due to optimistic locking. This is tolerable for many use cases, but undesirable for others (such as events that need to occur on time). For example, our urgent customer messaging events need to conform to service levels.
OVERCOMING LOCKING LIMITATIONS - ZOOKEEPER
Industry best practice for Cassandra locking is to use an external lock manager. ContextSpace uses Zookeeper, which shares Cassandra distributed and lights-out management strengths.
With Zookeeper, we can maintain any degree of locking granularity, which permits parallel execution to be performed.
By removing locking from atomic batches, we can maintain full atomicity for transactions.
We are currently implementing Zookeeper support and recommend this configuration for production applications.
JOB SCHEDULER (ASYNCH EXECUTION)
Support for the Job Scheduler has recently been added to this project.
Camunda job scheduling presents us with another challenge.
We only need to pick jobs that are due. This requires using global ordering of jobs by time.
This is essentially a queue.
JOB SCHEDULER (ASYNCH EXECUTION) Cassandra and queuing patterns dont like each other. Cassandra can only maintain order within a single partition. Remember the tombstones? Queuing operations generate a
heavy columnar delete workload. This creates the perfect storm for Cassandra.
Accessing a common partition (which is always stored on a single Cassandra node), will also create a database hotspot.
JOB SCHEDULER (ASYNCH EXECUTION) In this situation, best practice is to partition scheduling data to
contain one time slice per partition (such as a day or an hour). This addresses the problem with excessive tombstones, but
does not fully alleviate the hotspot. However, given Cassandra performance, this will only become an issue for very large workloads.
For high workload deployments, we will implement support for an external queue manager. Kafka is a distributed queue manager that shares lights-out characteristics with Cassandra and Zookeeper and is already part of the ContextSpace architecture.
CASSANDRA PERSISTENCE FOR CAMUNDA
Is Cassandra a natural match for Camunda? In a word, not really.
Camunda creates and then deletes a lot. Cassandra generally hates deletions.
Cassandra locking is enforced at partition (row) level only. Using Zookeeper, we can update process, job execution and indices atomically.
Locking one process within a row precludes parallel executions. Again, Zookeeper locking addresses this.
Job Executor always searching for next jobs at current time, will consistently create hotspots in Cassandra. This is fine for most workloads, but for very high loads we will support Kafka.
UNDERSTANDING THE STACK OPTIONS CAMUNDA + CASSANDRA implements optimistic locking
and will not support parallel executions, but will be highly available across multiple distributed data centres.
CAMUNDA + CASSANDRA + ZOOKEEPER implements granular locking and can support the full capabilities of the CAMUNDA BPM engine.
CAMUNDA + CASSANDRA + ZOOKEEPER + KAFKA eliminates all anti-patterns and will provides support for virtually unlimited workloads in a distributed environment
SUMMARY Cassandra is not a native fit for Camunda Therefore, much work has been done to counter anti-
patterns Nevertheless, backing Camunda with Cassandra promises
incredible performance, scalability, availability and operational gains
ContextSpace focus is on completing operational support for the core BPM engine
Additional Camunda user application queries can then be incrementally supported as they are required.
THANK YOU AND BRIEF Q&A
Stan Levine email@example.com