Cassandra Summit 2014: Internet of Complex Things Analytics with Apache Cassandra
C* Summit EU 2013: The Cassandra Experience at Orange
-
Upload
planet-cassandra -
Category
Technology
-
view
2.229 -
download
0
description
Transcript of C* Summit EU 2013: The Cassandra Experience at Orange
Jean Armel Luce
Orange France/DSIF/DF/SDF
V1.0
The Cassandra Experience at Orange Project PnS 3.0
2 Cassandra Summit Europe – October 17 2013
#CassandraEU
Summary
§ Short descripGon of PnS. Why did we choose C* ?
§ Our migraGon strategy
§ AMer the migraGon …
§ AnalyGcs with Hadoop/Pig/Hive over Cassandra
§ ContribuGons and open sourced modules from Orange & conclusions
Jean Armel Luce - Orange-DOP-PnS 3.0
Short description of PnS3
4 Cassandra Summit Europe – October 17 2013
#CassandraEU
PnS – Short description
§ PnS means Profiles and Syndication : PnS is a highly available service for collecting and serving live data about Orange customers
§ End users of PnS are :
– Orange customers (logged to Portal www.orange.fr) – Sellers in Orange shops – Some services in Orange (advertisements, …)
Jean Armel Luce - Orange-DOP-PnS 3.0
5 Cassandra Summit Europe – October 17 2013
#CassandraEU
PnS – The Big Picture
Jean Armel Luce - Orange-DOP-PnS 3.0
End users
Millions of HTTP requests (Rest or Soap) Fast and highly available
Database
WebService to get or set data stored by pns : - postProcessing(data1) - postProcessing(data2) - postProcessing(data3) - postProcessing(datax) - …
PNS Data providers
Thousands of files (Csv or Xml) Scheduled data injection
DB Queries R/W operations
6 Cassandra Summit Europe – October 17 2013
#CassandraEU
§ Until 2012, data were stored in 2 differents backends :
ü MySQL cluster (for volatile data)
ü PostGres « cluster » (sharding and replication)
§ and web services
(read and writes)
§ for batch updates
PnS2 – Architecture
Jean Armel Luce - Orange-DOP-PnS 3.0
Bagnolet
Sophia Antipolis
2 DCs architecture for high availability
7 Cassandra Summit Europe – October 17 2013
#CassandraEU
Timeline – Key dates of PnS 3.0
Jean Armel Luce - Orange-DOP-PnS 3.0
PNS 2
2010 to 2012
• Study phase We did a large study about a few NoSQL databases (Cassandra, MongoDB, Riak, Hbase, Hypertable, …) è We chose Cassandra as the single backend for PnS
06/2012 • Design phase
We started the design phase of PnS3.0
09/2012
• Proof Of Concept We started a 1st (small) Cassandra cluster in producGon for a non criGcal applicaGon : 1 table, key value access
04/2013 • Produc7on phase
MigraGon of the 1st subset of data of PnS from mysql cluster to Cassandra in produc7on
05/2013
to 12/2013
• Complete migra7on MigraGon of all other subsets of data from Mysql cluster and Postgres to Cassandra Add new nodes in the cluster (From 8 nodes in each DC to 16 nodes in each DC) Add a 3rd datacenter for AnalyGcs
8 Cassandra Summit Europe – October 17 2013
#CassandraEU
PnS – Why did we choose Cassandra ?
§ Cassandra fits our requirements :
– Very high availability
– Low latency
– Scalability
§ And also :
– Ease of use : Cassandra is easy to administrate and operate – Some features that I like (rack aware, CL per request, …) – Cassandra is very efficient for simple requests :
« SELECT mycol1, mycol2, …, mycolx FROM mytable WHERE myprimarykey = ‘mycustomerid’ »
Jean Armel Luce - Orange-DOP-PnS 3.0
PnS2 = 99,95% availability we want to improve it !!!
20 ms < RT PnS2 web service < 150 ms we want to improve it !!!
Higher load, higher volume next years ?
unpredictable; better scalability brings new businesses
Our migration strategy
10 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
The migraGon -‐ Input
§ During the migration, we need to :
§ maintain a very high availability
§ maintain (or lower) the latency during the migration
§ guarantee no functional regression
§ Question :
§ How can we migrate the data to Cassandra without any interruption of service ?
11 Cassandra Summit Europe – October 17 2013
#CassandraEU
S ubdivision intosubsets
Migration data of the subset
Check/validation of the migration
S witch queriesto Cassandra for the subset
goto 1st subset
goto next subset
§ Subdivision of data into many subsets according to many criteria : § Same source of data
§ Relationships between data
§ And then, migrate each subset 1 by 1
§ Definition of a generic process for all the subsets
Jean Armel Luce - Orange-DOP-PnS 3.0
The migraGon : Step by step processing
12 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
The migraGon : Tools and Gps
§ The strategy of migration is based on 2 main facilities :
the timestamp management by Cassandra Each data stored in C* is timestamped.
It is possible to set this timestamp when inserting/updating/deleting a data in Cassandra. When Cassandra retrieves a data item, it returns the value having the most recent timestamp. We use this feature to distinguish the values stored before the migration started and the values inserted during or inserted after the migration
mod_dup An Apache module developped (and open sourced) by Orange teams. Mod_dup can duplicate web requests, filter them on some criteria, substitute characters (regexp), and send the duplicated requests to another pool of web servers. Used in order to fill legacy (relational) database and Cassandra database simultaneously during the migration of the subset
HTTP Req
PNS 2
PNS 3
mod_dup
13 Cassandra Summit Europe – October 17 2013
#CassandraEU
The migration : initial state
Jean Armel Luce - Orange-DOP-PnS 3.0
Data providers
End users
HTTP Rest/Soap Read
Files transfer via FTP or CFT PNS 2
DB
HTTP Rest/Soap Write
SQL Read/Write
WebServer
BatchInjector
Step 0
14 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
The migration : double feed
PNS 3 Cassandra
DB
CQL update
Duplicate streams (files) from data providers BatchInjector
WebServer
Duplicate HTTP update streams from end users
Data providers
mod_dup
Step 1
15 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
The migration : copy data form PnS2 to PnS3
PNS 3 Cassandra
DB BatchInjector
WebServer
Batch Injection TimeStamp = start date of extraction
mod_dup
Data providers
HTTP Write
Step 2
16 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
The migration : control
PNS 3 Cassandra
DB BatchInjector
WebServer
Synchro Control
SQL
CQL
mod_dup
Data providers
HTTP Write
Step 3
17 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
The migration : switch reads
PNS 3 Cassandra
DB BatchInjector
WebServer End users
Files transfer via FTP or CFT PNS 2 DB
HTTP Rest/Soap Write WebServer
BatchInjector
100 % read now on Cassandra HTTP Read requests
HTTP Rest/Soap Read
mod_dup
Data providers
HTTP Write
Step 4
18 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
The migration : stop double feed
PNS 3 Cassandra
DB BatchInjector
WebServer End users
PNS 2 DB
WebServer
BatchInjector
100 % read on Cassandra 100% write on Cassandra
for HTTP request
100% write on Cassandra for Data injection
Files transfer
HTTP Rest/Soap Write
HTTP Read/Write requests
Data providers
Files transfer via FTP or CFT
mod_dup
HTTP Write
HTTP Read requests
Step 5
19 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
The migraGon
§ Using this procedure :
§ We can migrate to Cassandra without any interruption of service
§ It is possible to switch progressively to Cassandra rather than doing a one shot switch
§ During the control phase, we can take time (a few days, a few weeks) to check that everything is OK before switching to Cassandra
§ It is possible to easily rollback the migration of a subset if errors are found during the control phase, without losing any update
§ Doesn’t work if the queries are not idempotent.
§ After the migration, we can easily duplicate production requests (entirely or partially) and send them to a bench platform thanks to mod_dup
After the migration …
21 Cassandra Summit Europe – October 17 2013
#CassandraEU
§ Comparison before/after migration to Cassandra
§ Some graphs about the latency of the web services are very explicit :
Service push mail Service push webxms
Jean Armel Luce - Orange-DOP-PnS 3.0
The latency
dates of migration to C*
22 Cassandra Summit Europe – October 17 2013
#CassandraEU
§ Read and write latencies are now in microseconds in the datanodes :
Thanks to and
This latency will be improved by (tests in progress) : ALTER TABLE syndic WITH compacGon = { 'class' : 'LeveledCompacGonStrategy', 'sstable_size_in_mb' : ?? };
Jean Armel Luce - Orange-DOP-PnS 3.0
The latency
23 Cassandra Summit Europe – October 17 2013
#CassandraEU
• We got a few hardware failures and network outages
• No impact on QoS :
• no error returned by the application
• no real impact on latency
Jean Armel Luce - Orange-DOP-PnS 3.0
The availability
24 Cassandra Summit Europe – October 17 2013
#CassandraEU
• PnS activity is always increasing (volume of data and requests/sec)
• How to measure the capacity of a cluster ?
Capacity of a C* cluster = capacity of a node * number of nodes (true if all nodes are identical)
• there are 2 ways to deal with the expansion of activity :
Ø scale up (add more resources such as CPU, disks, RAM to each node)
Ø scale out (add new nodes in the cluster)
Jean Armel Luce - Orange-DOP-PnS 3.0
The scalability
25 Cassandra Summit Europe – October 17 2013
#CassandraEU
• Thanks to vnodes (available since Cassandra 1.2), it is easy to scale out
With NetworkTopologyStrategy, make sure to distribute evenly the nodes in the racks
Jean Armel Luce - Orange-DOP-PnS 3.0
The scalability
Analytics with Hadoop/ Pig/Hive over Cassandra
27 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
Basic architecture of the Cassandra cluster
§ Cluster without Hadoop : 2 datacenters, 16 nodes in each DC
§ RF (DC1, DC2) = (3, 3)
§ Requests from web servers in DC1 are sent to C* nodes in DC1
§ Requests from web servers in DC2 are sent to C* nodes in DC2
Poolofweb
serversDC1
Poolofweb
serversDC2
DC1 DC2
28 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
Architecture of the Cassandra cluster with the datacenter for analyGcs
§ Cluster with Hadoop : 3 datacenters, 16 nodes in DC1, 16 nodes in DC2, 4 nodes in DC3
§ RF (DC1, DC2, DC3) = (3, 3, 1)
§ Because RF = 1 in DC3, we shall need less storage space in this datacenter
§ We favor cheaper disks (SATA) in DC3 rather than SSDs or FusionIo cards
§ Works better with HSHA Thrift server (tests in progress)
29 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
Architecture of the Cassandra cluster with the datacenter for analyGcs
DC1 DC2
DC3
Poolofweb
serversDC1
Poolofweb
serversDC2
Contributions and open sourced modules from Orange & conclusions
31 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
ContribuGons and open sourced modules
§ Open sources by Orange
§ PHP driver for Cassandra :
https://github.com/Orange-OpenSource/YACassandraPDO
Thanks to Sandro Lex & Mathieu Lornac
§ Mod_dup (Migration to Cassandra)
§ https://github.com/Orange-OpenSource/mod_dup
Thanks to Jonas Wustrack & Emmanuel Courreges
§ Other contributions
§ C driver (libdbi driver) :
http://libdbi-drivers.cvs.sourceforge.net/viewvc/libdbi-drivers/libdbi-drivers/?pathrev=Branch-2012-07-02-cassandra
Thanks to Emmanuel Courreges
32 Cassandra Summit Europe – October 17 2013
#CassandraEU
§ With Cassandra, we have improved our QoS
§ We are able to open our service to new opportunities
§ There is an ecosystem around C* (Hadoop, Hive, Pig, Storm, Shark, …), which offers more capabilities. However, we would love to have some of the components (Hive) integrated in C* core (as Pig)
§ PnS3 works better and hopefully cheaper than PnS2
Jean Armel Luce - Orange-DOP-PnS 3.0
Conclusions
33 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
Thank you
34 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
Questions
35 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
A few answers about hardware/OS version /Java version/Cassandra version
§ Hardware :
§ 16 nodes in each DC at the end of 2013 :
§ 6 CPU Intel® Xeon® 2.00 GHz
§ 24 GB RAM
§ FusionIO 320 GB MLC
§ OS :
§ Ubuntu Precise (12.04 LTS)
§ Cassandra version :
§ 1.2.2 (with a few patches backported from 1.2.3)
§ Java version :
§ Java7u7 : not recommended, upgrade scheduled soon
36 Cassandra Summit Europe – October 17 2013
#CassandraEU
Jean Armel Luce - Orange-DOP-PnS 3.0
A few answers about data and requests
§ Data types : § Volume : 6 TB at the end of 2013
§ elementary types : boolean, integer, string, date
§ collection types
§ complex types : json, xml (between 1 and 20 KB)
§ Requests : § 10.000 requests/sec at the end of 2013
§ 80% get
§ 20% set
§ Consistency level used by PnS : § ONE (95% of the queries)
§ LOCAL_QUORUM (5% of the queries)