Cassandra Summit 2014: Launching PlayStation 4 with Apache Cassandra
-
Upload
planet-cassandra -
Category
Technology
-
view
546 -
download
5
description
Transcript of Cassandra Summit 2014: Launching PlayStation 4 with Apache Cassandra
Launching PS4 with Cassandra
Introduction • Alexander Filipchik – Staff Software Engineer at SNEI
• Dustin Pham – Staff Software Engineer at SNEI
Agenda • Journey towards Cassandra • Cassandra-backed PS4 Features • Ops-y Stuff • Lessons learned
Journey towards Cassandra
Challenges • Small Team • Legacy Support • Hardware Deadline • Scaling @ Peak Time
Why Cassandra • Strong community • Horizontally scalable architecture • Good performance • Cost effective • New adventure J
6
PS4 Features backed by Cassandra
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more
Cassandra-backed PS4 features
• What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • + more
Ops-y Stuff
Infrastructure • Hosted in cloud and physical DCs • Several hundred nodes and growing • Cluster by feature • Vnodes and Assigned token clusters • Astyanax Client
Stats for PS4 cloud nodes • Data throughput: Gigabytes / sec • Cassandra read/writes: > 200,000 / sec • Data size: tens of terabytes • 10M PS4 and 80M PS3 sold
24
Clusters • Cluster per Read/Write pattern initially • Now use cluster per feature • Seeds referenced by DNS names • Size Tiered compaction • Manual compactions for some CFs
25
A typical node • m2.4xl + i2.2xl • 2 ephemeral disks (~ 2 x 800 GB) • Commit log on root partition • Topology managed in the topology file
managed by chef
26
AWS • Nodes are
interleaved between AZs – Replication factor
spreads data across AZ’s
– Minimizes downtime due to AZ outage
Availability Zone A Availability Zone C
Eph1
Disk Layout
Pre-Launch Launch Current
ü 2 Ephemerals in a RAID 0 ü Higher throughput (io
spreads into 2 devices for reading & writing)
ü If you lose 1 device, you loose the array !
ü 2 Ephemerals in a RAID 1 ü Higher throughput for
reading (io spreads into 2 devices), but not for writing
ü If you lose 1 device, the array continues up in degraded mode.
ü ½ the available space
ü 2 individual Ephemerals ü Higher throughput (io
spreads into 2 devices for reading & writing)
ü You lose 1 device, Cassandra stops (configurable)
ü No RAID overhead
Eph0
AWS m2.4xl
RAID 0
Eph1
Eph0
AWS m2.4xl
RAID 1
Eph1
Eph0
AWS m2.4xl
Cluster Resizing
Thrift Payload Size
thri%_framed_transport_size_in_mb thri%_max_message_length_in_mb
Bouncing Nodes phi_convict_threshold
Inter-DC Latency
Monitor system health • Nagios • Kibana/Elasticsearch • Graphite • AWS Cloudwatch • App level monitoring • Opscenter
App level metrics
Lessons Learned
Fun with Astyanax Client • Cross DC Latencies – Several second latencies in JP and EE data
centers – Astyanax configs to ensure local datacenters
used • Imbalanced node traffic – Hashing algorithm (MD5 vs Murmur3)
• DNS Caching in the JVM – Stale seed nodes
A tale of 2 Nodes
Cluster lessons • A single bad node can raise app
latencies significantly • Taking out an entire cassandra cluster is
easy (not so fun) – Compressing data before sending to
cassandra helps a lot. • Corrupted SStable resulted in
cascading failure
• Monitoring – Memtable flush frequency – Hinted handoffs – Garbage collection – Compactions – Histograms
• VPNs are a dangerous bottle neck
• Easier to rebuild a node than to fix
• Backup data – Replication factor helps
but does not account for data corruption
• Denormalization costs • Disk is cheap but EC2s are
not • TTL on almost everything • Adjust gc_grace_period
based off TTL times • Transactions ? Be creative • Load test with real data
• Replication strategy: – Read / Write pattern – Data is source of truth or not – Data locality – User Level data vs App level
data • Cluster wide commands
should be staggered – Global repair L
Tokens • Vnodes vs Assigned Tokens – Increased chattiness on gossip protocol
with vnodes – Perceived slowness on repair and cleanup
operations on vnodes enabled cluster – Astyanax client does not like vnodes…
Compactions • Compactions are your worst enemy – larger disk usage = high cpu & longer
compactions • Leveled compaction vs sized compaction – Start up time – Cpu tradeoff – IO tradeoff
• Updates + Removals eat up disks
We are hiring… sonyentertainmentnetwork.com/careers