Cassandra at Pollfish

download Cassandra at Pollfish

of 23

  • date post

    15-Feb-2017
  • Category

    Software

  • view

    95
  • download

    0

Embed Size (px)

Transcript of Cassandra at Pollfish

  • Stavros KontopoulosSoftware Engineer, MSc

    Cassandra at Pollfish.com

  • What we do at Pollfish?

    Target mobile users with surveys through our android/ ios sdk which is installed via thousands of mobile apps. Developers benefit from completed surveys, companies also may run a survey campaign in real time. Now analytics/ML pipeline

    Why Apache Cassandra?

    Store time series about system events, user activities, survey results and much more.. Amazing write throughput, take advantage of idempotent writes with proper resolution. Decent read speed throughput and low latency. Integrates with spark to implement our analytics, insights pipeline. A new business model.

    A flashback why I am here

  • Our Tech Team- Disciplinary oriented

    Front End Development - UI Design Back-end, data engineer(s) Data Scientist DevOps

  • Pollfish High Level ArchitectureMobile Users (~600K active per day)

    APP SERVER 1

    APP SERVER N

    Other systems (PostGres, Geo, Redis)

    DSECassandra/Spark Cluster

    Survey Customers

  • Why Datastax

    Start up program and support when you go to production

    Many tools for development and maintenance: Pig, hive, shark, CFS, OpsCenter.

    Clear setup and support for real time data storage and analytics in the same cluster. Can be extended for other workloads like Search (Solr).

    Supports multiple DCs easily for other purposes staging, backup etc.

    Product version we used: DSE 4.5.1 (spark 0.9.1), 4.5.3, 4.6.0 (Spark 1.1.0)

  • Our Cassandra Cluster- Setup

    2 DataCenters (Cassandra & Analytics):

    1 for real-time data storage, read/write path. 1 for analytics nodes (Spark and Hadoop enabled), read/write path ETL, machine learning. Use the DSE setup with DSESimpleSnitch for mixed workloads multi-DC clusters.

    3 Nodes per DC (details next), planning for more.

    Data is written at Cassandra DC and replicated to Analytics DC for all keyspaces needed.

    2 seed nodes, one per DC.

  • Our Cassandra Cluster- Setup

  • Our Cassandra Cluster- Setup

  • Our Cassandra Cluster- Infrastructure Details

    Cassandra DC: Node 0,1,2: Standard_A4 (8 cores, 14 GB memory), 15DISKs in RAID 0

    Analytics DC: Node 0,1,2: Standard_D13 (8 cores, 56 GB memory), 15DISKs in RAID 0

    A bit about disks (ideally JBOD s but we can only have network disks) FS type: Ext4 Analytics disks per node:

    Azure Os disk (/dev/sda1 , mp: /): 29GBAzure VM tmp disk (/dev/sdb1, mp: /mnt/resource): 400GB -> for spark temp filesdata_file_directories: /mnt/dsedata/lib/cassandra/data (2,9T)commitlog_directory: /var/lib/cassandra/commitlog (197GB disk)

  • A few tips towards production

    Follow as a start: http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.htmlhttp://www.datastax.com/wp-content/uploads/2014/04/WP-DataStax-Enterprise-Best-Practices.pdf

    NTP is a must for all servers, Cassandra needs that for node data synchronization and when you do analytics

    Proper limits: cassandra - memlock unlimited cassandra - nofile 100000 cassandra - nproc 32768 cassandra - as unlimited

    Optimum blockdev --setra settings for RAID (OK): All value must be set to 128KB for RA.

    No swap

    http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html%0Dhttp://www.datastax.com/wp-content/uploads/2014/04/WP-DataStax-Enterprise-Best-Practices.pdf%0D

  • A few Cassandra optimization tips

    For Cassandra DC: concurrent_reads: 64 concurrent_writes: 32 Adjust according to the load and node IOPS/throughput..

    Heap size adjustments: Datastax does a good job to automatically handle this In general for production 8GB for heap is ok to avoid gc pauses found in bigger jvms. We use 4GB in smaller Cassandra nodes and 8GB in Analytics. In case you have a load with short lived objects adjust accordingly the parNew size. Your tools: jconsole,jvisualvm,hprof, jstack, enable GC reports at logging level

    JVM_OPTS=$JVM_OPTS -XX:+PrintGCDetailsJVM_OPTS=$JVM_OPTS -XX:+PrintGCDateStampsJVM_OPTS=$JVM_OPTS -XX:+PrintHeapAtGC

  • Managing nodes

    When you have a running car how you change the tire?

    Adding a new clean node to the cluster fast : Add the following line to /etc/dse/cassandra/cassandra-env.sh : JVM_OPTS=$JVM_OPTS -Dcassandra.join_ring=false Start the service Revert change and execute nodetool join After joining finishes execute a node rebalance from OpsCenter. Fallback: remove data and commit log dirs. And start again.

    Stopping a node:

    nodetool drain -h sudo service dse stop

  • Integration with Spark- Use cases

    Commmon ETL case:

    Mobile User profile (ML):

    HDFS Compatible (CFS)

    Cassandra Raw Data

    Spark/Cassandra

    Spark/Cassandra

    HDFS Compatible (CFS)

    Cassandra User profile data

    Customers Amazon S3

  • Integration with Spark- Our job framework

    We have built an analytics framework to run spark jobs on top of Cassandra. Executes jobs remotely through maven.

    SparkContext is created on a non-dse node. Needed some tricks to make it work, now we move to a more dse official approach with spark submit. Note: We started with 0.9.1 DSE version where no spark submit was available.

    The framework provides an API to run jobs in production and use common functionality at the Data science side for code re-use.

  • Datastax Cassandra Opscenter

    Facilitates: monitoring, health check and daily operations at the cluster level.

    We use it for: node rebalance, repairs,snapshotting, latency throughput checks along with low level tools like iotop, ioping etc.

  • Datastax Cassandra Opscenter (Pending cluster operations)

  • Datastax Cassandra Opscenter (Keyspace Details)

  • Datastax Cassandra Opscenter (Last 10 days)

  • The Good and the Bad with Cassandra and DSE

    good

    Easy to start with, enough documentation.

    Reliable performance so far.

    Easily scalable, easy to add/remove nodes.

    Support.

    bad

    Bugs at the Cassandra level and the Spark level. Need active follow up of the lists for Spark, Cassandra and DSE tools. Upgrade maybe the only solution and not a piece of case.

    Gets tricky to optimize depending on your load specifically when you dont have time to measure everything upfront and you are in production already.

  • The Good and the Bad with Cassandra and DSE (Some issue examples)

    Apply backpressure gently when overloaded with writes, https://issues.apache.org/jira/browse/CASSANDRA-7937

    Reproduce by writing large volume of data with large column values eg. 300K, crawling scenario...

    Spark Cassandra Connector issues (eg. java driver not closing connection with cluster)...

  • As a developer some cool stuff I mess with

    Currently I am extending the framework I have built to deliver spark jobs on top of Cassandra. IT with Cassandra embedded connector. Cassandra Schema Design. ETL use cases.

    Organizing dev env: Optimizing development environment with vagraant, automatic installation - avoid upgrade headaches, setup and IT level testing. Introduced CI with Jenkins, Artifactory etc

  • thank you