Beginning&the&DevOps&Journey&in&Real&Money&

Gaming&

Kelly&Looney&27.11.2014&

bwin

Partygaming

DevOps in Real Money Gaming Context: 1B € company – Online Sports betting, Online Casino, Poker, other games Two primary technologies combined via a business merger

(turn of the century architecture) •  Sports - .Net/SQL Server •  Poker, Casino, and “Platform” – Java/Oracle

•  Datacenters in Gibraltar, Vienna, and other points in Europe, now in US •  200 people in Ops and Infrastructure

•  Development Centers in Vienna, Ukraine, and Hyderabad •  Over 700 development team members

3

4

Different&faces/rules&For&different&markets&

Monolithic&App&with&many&single&points&of&failure&

In 2013…the Challenge

Dev&Teams&focused&on&Horizontal&Components&

Totally&separate&Ops,&Maintenance,&and&Dev&teams&

Clashing&cultures&from&Merger/LocaOons/&Code&bases&

Up&24/7&with&Millions&of&€&a&day&wagered&

InThouse&Build&Deploy&Monitoring&…&

What we have done and are doing…new ideas •  Global Agile Transformation – classes, coaches, 90 Scrum teams

•  Craig Larman, Luke Hohmann (Innovation Games) •  DTO (Damon Edwards, Alex Honor) for DevOps principles

•  Several Organization Changes •  Components -> Features -> Services •  Ops -> LeanOps -> Delivery Units

5

What we have done and are doing…tech •  Tool changes

•  SVN->Git, In house depoy -> Jenkins/Team City, Chef, Rundeck •  Bare Metal -> VMWare -> Now headed to Docker/containers •  AppDynamics for application monitoring – (amazing!)

•  Architectural Principles •  Less centralized, fewer failure points •  Throw cheap machines, not faster CPUs or bigger DBs at scaling

problems •  Use RDBs when needed otherwise avoid

6

Cultural changes we are encouraging •  Old style Developers

•  Responsibilities: Write code •  Focus: Know ONE THING really really well. •  Deep expertise = respect

•  What we want now is Developers that: •  Understand our company goals •  Understand requirements and tests •  Write, build, integrate, and test code incrementally •  Can demonstrate and explain working systems •  Maintains his/her code in production •  Understands operations

Deep expertise is great, but varied knowledge is just as important 7

Wow, you want developers to do everything…

•  First the right attitude…then •  Todays Tools and Processes:

1.  Scrum provides continuous “customer” access

2.  Distributed versioning (typically Git) puts full source control into individual developers hands

3.  Continuous Integration isolates mistakes

4.  Jenkins-Vagrant-Puppet-Chef-Saltstack pipelines make infrastructure and deployment mostly automatic regardless of complexity

How is that possible?

8

Automated Tests

•  It’s not “How many automated tests do I have?”

•  We could have easily run days worth or tests whenever we wanted

•  It’s “I have the right tests to quickly decide if I can move forward”

•  Also BTW “We run Jenkins to do a build every night”

•  Does != Continuous Integration…

9

Containers are changing hosting •  Virtualization efficiency and cost savings are obvious

•  The most interesting issue is the separation of concerns presented

•  “developer-land” vs infrastructure 10

Roundabout: The Bwin.Party Data

Center of the Future

Goals:1. Proven Failover and DR

Recent events certainly put this in focus...

2. Frequent Releases with Rollback

Our Agile/DevOps goal for modern development and delivery

Blue/Green/Yellow

— Three full production replicas

Blue = Live

Green = Staging/Rollback

Yellow = Failover

— DB replication across sites

Roundabout Process

1. Live data center hosts our applicationsSwitch to Failover site can be initiated at any time

2. Staging is used to build and do final test on a new release

3. Staging is switched to Live4. Live becomes Rollback.

Roundabout Process cont.

1. The new Live is cloned to Failover - Rollback available if needed

2. When we feel good about Live we can start changing Rollback to a new Staging

Assumption: DB replication supports all this switching without loss

In Conclusion: DevOps for large enterprises is possible!

Just really f-ing hard…