Cassandra At Wize Commerce

download Cassandra At Wize Commerce

of 59

  • date post

    06-May-2015
  • Category

    Business

  • view

    1.943
  • download

    1

Embed Size (px)

description

Did a presentation at Cassandra meetup explaining how we used Cassandra internally in Wize Commerce to improve our object cache. Also, I talked about a performance evaluation we carried out before we moved into Cassandra.

Transcript of Cassandra At Wize Commerce

  • 1.CASSANDRA AT WIZE COMMERCE Eran Chinthaka Withana Eran.Withana@wizecommerce.comCASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012)

2. About me Engineer in Platform and Infrastructure team at WizeCommerce (formerly Nextag) Member, PMC Member and a committer of ApacheSoftware Foundation Contributed to Web services project since 2004 (in a different life) PhD in Computer Science fromIndiana University, Bloomington, Indiana Today 2CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 3. In the next hour Wize Commerce Impact of Cassandra on Wize Commerce Object Cache Personalized Search Performance evaluation of Cassandra in a multi-datacenter and a read/write heavy environment 3CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 4. WIZE COMMERCECASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 5. About Wize Commerce Helping companies maximize their eCommerceinvestments across every channel, device and digital ecosystem an expertise weve honed for years with our eCommercecustomers providing them with unmatched traffic and monetizationservices at incredible scale 5CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 6. About Wize Commerce Scale of Wize Commerce We drive over $1.1 Billion in annual worldwide sales Shopping Network includes Nextag, guenstiger.de,FanSnap, and Calibex Each week, we manage 21 Million Keyword Searches 105 Million Retargeted Ads 140 Million Bot Crawls 300 Million Facebook Ads 700 Million Keywords 560 Million Product SKUs 1000s of Simultaneous A/B Test 6CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 7. CASSANDRA AT WIZE COMMERCE - CACHECASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 8. Cache ArchitectureCASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 9. Cache Architecture Multi-tiered read-through cache, optimized for performance TTLs at upper levels to keep the data fresh JMS based infrastructure to refresh objects on-demand 9CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 10. Cache - Expectations For each object Less than 30ms 95th percentile read latency Less than 1-hour of update latency with 30M updates(phase 1, with existing components) 10 minutes with eventing system integrated Fault tolerance Low maintenance overheads Ability to scale 10CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 11. Cache Cassandra Integration 11CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 12. Cache Cassandra Integration DC1 DC2DC3DC4 Replication factors to facilitate required number of copies perregion Consistency level to suit business requirements 6 multi-data center clusters with total nodes per clusterranging from 24 to 32 In house monitoring system for continuous monitoring andescalations 12CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 13. Cache Cassandra Integration Clients Hector with DynamicLoadBalancing policy Started experimenting with Astyanax Maintenance Weekly repair and compaction tasks Monitoring System health monitoring End-to-end latency Update latency 13CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 14. Cache Cassandra Integration Ring output of a cluster Address DC Rack Status State Load Owns Token148873535527910577765226390751398592512 xx.xx.xx.79DC1 RAC1 Up Normal 90.19 GB 12.50% 0 xx.xx.xx.75DC2 RAC1 Up Normal 51.15 GB 0.00% 1 xx.xx.xx.75DC3 RAC1 Up Normal 126.62 GB 0.00% 2 xx.xx.xx.80DC1 RAC1 Up Normal 88.57 GB 12.50% 21267647932558653966460912964485513216 xx.xx.xx.81DC1 RAC1 Up Normal 89.82 GB 12.50% 42535295865117307932921825928971026432 xx.xx.xx.76DC2 RAC1 Up Normal 51.1 GB 0.00% 42535295865117307932921825928971026433 xx.xx.xx.76DC3 RAC1 Up Normal 124.49 GB 0.00% 42535295865117307932921825928971026434 xx.xx.xx.82DC1 RAC1 Up Normal 85.78 GB 12.50% 63802943797675961899382738893456539648 xx.xx.xx.83DC1 RAC1 Up Normal 84.34 GB 12.50% 85070591730234615865843651857942052864 xx.xx.xx.77DC2 RAC1 Up Normal 49.34 GB 0.00% 85070591730234615865843651857942052865 xx.xx.xx.77DC3 RAC1 Up Normal 123.54 GB 0.00% 85070591730234615865843651857942052866 xx.xx.xx.84DC1 RAC1 Up Normal 82.94 GB 12.50% 106338239662793269832304564822427566080 xx.xx.xx.85DC1 RAC1 Up Normal 83.1 GB 12.50% 127605887595351923798765477786913079296 xx.xx.xx.78DC2 RAC1 Up Normal 47.98 GB 0.00% 127605887595351923798765477786913079297 xx.xx.xx.78DC3 RAC1 Up Normal 121.25 GB 0.00% 127605887595351923798765477786913079298 xx.xx.xx.86DC1 RAC1 Up Normal 83.41 GB 12.50% 148873535527910577765226390751398592512CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 15. Cache Cassandra Integration Column family stats of a cluster Keyspace: XXXX Read Count: 37060467 Read Latency: 3.0589244618800944 ms. Write Count: 37013052 Write Latency: 0.05114632081677566 ms. Pending Tasks: 0 Column Family: YYY SSTable count: 11 Space used (live): 71463479840 Space used (total): 71463479840 Number of Keys (estimate): 66231424 Memtable Columns Count: 314964 Memtable Data Size: 68140546 Memtable Switch Count: 628 Read Count: 37060467 Read Latency: 3.138 ms. Write Count: 37013052 Write Latency: 0.058 ms. Pending Tasks: 0 Bloom Filter False Postives: 10653 Bloom Filter False Ratio: 0.01611 Bloom Filter Space Used: 173770024 Key cache capacity: 60000000 Key cache size: 13309399 Key cache hit rate: 0.9210111414757199 Row cache: disabled Compacted row minimum size: 925 Compacted row maximum size: 8239 Compacted row mean size: 2488CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 16. Cache Cassandra Datastore Performance 3-6ms average read latency across all objects in alldata centers 15-20ms 95th percentile read latency 30mins average update latency at 25M updates Zero downtime even with multiple node failures 16CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 17. Cache Snapshot of Live SystemMedian Read Latency Objects Scrubbed in Last 24hrsScrubber Latency17CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 18. Cassandra Integration Lessons Learned Try to understand the internals, read code and find solutionson your own before getting into support requests Assumption: you have adventurous engineers :D Use IRC channels, user lists Never use RoundRobinLoadBalancingPolicy if you care aboutperformance DynamicLoadBalancingPolicy: based on the probability of failure ofnode Divide keyspace within the datacenter and use token + 1method in other data centers Experiment different configurations but make sure to have aquick fallback plan 18CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 19. Cassandra Integration Lessons Learned Compaction are crucial for read/write heavyenvironment 24 x 7 automated monitoring and alerts Read/write latencies , read misses and node status at least Consistency levels are important, if you expect nodefailures in a multi-data center environment Concentrate on key cache and forget about rowcache if you have limited resources. Rely on OS file cache 19CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 20. Cache: Future 20CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 21. Cache: Future Exposing Cache system using SOA basedinfrastructure Thrift services enabling all cache accesses Event based updates Event based pipeline for changes for system-of-record Based on Storm (Twitter) Getting rid of Memcached 21CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 22. CASSANDRA AT WIZE COMMERCE PERSONALIZED SEARCHCASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 23. Personalized Search Aggregates user data from multiple data sources, e.g.site search, banner clicks. Uses statistical model to re-rank search resultstailored to the user. Decomposes user information into model variables:brand preference, merchant preference, productcategory preference, etc. 23CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 24. Personalized Search Cassandra Integration Serves 30-40MM banner ad impressions daily Before: rely on user cookie (stores up to 4 weeksdata) After: use user cookie for todays data, combinedwith Cassandra Data Store to keep up to 3 monthsdata 24CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 25. PERFORMANCE EVALUATION OF APACHE CASSANDRA IN A MULTI-DATA CENTER, READ/WRITE HEAVY ENVIRONMENTCASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 26. Objectives Understand the limitations of Cassandra whendeployed in a multi-data center environment Find out the best set of parameters that canbe used and tuned to improve theperformance Find out the limits of Cassandra cluster and foreach version. 26CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 27. Objectives Understand its scalability characteristics withvarying amount of operations per second This will help us to understand how much of loadwe can serve without causing any significantperformance degradations. Understand the implications of node failureson its capability to efficiently serve data toclient requests 27CASSANDRA AT Wize Commerce Eran Chinthaka Withana. Cassandra Meetup (07/25/2012) 28. Environment Setup Test Metrics Operation Latency for a given throughput (set in the client) Average, Minimum, Maximum, 95th percentile Test Setup Versions: Apache Cassandra 0.8.6 and 1.0.1 Node Distribution: 12-nodes distributed over threegeographically distributed data centers in US Key Distribution: Keyspace is divided into four in each datacenter