A State of Xen Chaos Monkey & Cassandra

Who we are

Jean-Sebastien Jeannotte – JS

Senior Software Engineer Platform Automation Engineering

jjeannotte@netflix.com

@jsjeannotte

http://www.linkedin.com/in/jsjeannotte

Nir Alfasi

Senior Software Engineer Platform Automation Engineering

alfasi@netflix.com

@niralfasi

http://www.linkedin.com/in/alfasin

Christos Kalantzis

Director of Engineering Cloud Database Engineering

Cassandra MVP

ckalantzis@netflix.com

@chriskalan

http://www.linkedin.com/in/christoskalantzis

AWS Boot re: September 2014, Every AZ

Our stack during Re:boot 2014

C* Priam

C* Priam

C* Priam

REST + SSH

Our stack during Re:boot 2014

Our stack during Re:boot 2014

Our stack during Re:boot 2014

C* Priam

C* Priam

C* Priam

REST + SSH

Atlas Atlas App 1

App 2

Our stack during Re:boot 2014

Our stack during Re:boot 2014

Disappearing  instance?  

Launch  new  instance  

All  good  

Is  the  C*  ring  healthy?  

Yes  

Are  all  instances  healthy?  

Yes  

All  good  

Can  we  fix  automa>cally?  

Replace  bad  instance  

All  good  

Is  there  an  offline  maintenance?  

First  failure?  Sleep  for  X  minutes  and  

retry  

PagerDuty  No   Is  there  an  offline  maintenance?  

First  failure?  

All  good  

Every 30 min

Our stack during Re:boot 2014

AWS Boot re: September 2014, Every AZ

Gaps we identified

Gaps we identified

Gaps we identified

Gaps we identified

New direction

New direction – What others are doing

New direction – What we decided to do

New direction – What we decided to do

New direction – What we decided to do

C* Priam

C* Priam

C* Priam

Atlas Atlas App 1

App 2

New direction – What we learned (principles)

New direction – What we learned (principles)

New direction – What we learned (principles)

Synchronous   Asynchronous  

SSH   HTTP  /  REST  

New direction – What we learned (principles)

New direction – What we learned (principles)

What does the future look like?

What does the future look like?

What does the future look like?

Check out our https://jobs.netflix.com page for current openings

Who we are

Jean-Sebastien Jeannotte – JS

Senior Software Engineer Platform Automation Engineering

jjeannotte@netflix.com

@jsjeannotte

http://www.linkedin.com/in/jsjeannotte

Nir Alfasi

Senior Software Engineer Platform Automation Engineering

alfasi@netflix.com

@niralfasi

http://www.linkedin.com/in/alfasin

Christos Kalantzis

Director of Engineering Cloud Database Engineering

Cassandra MVP

ckalantzis@netflix.com

@chriskalan

http://www.linkedin.com/in/christoskalantzis