Continuous Delivery of Infrastructure

The case of the growing startup

Nicolas Blanc - BlaBlArchitect

SinfomicSinfomic (1999)

@thewhitegeek

(2001)

(2005)

(2008)

(2012)

3/49

Summary

● Introduction● Why Continuous Delivery of Infrastructure● Automate Configuration● Hardware Industrialization● Automate OS Install● Hardware availability● Continuous delivery

Introduction

80€ 20€20€

20€

@BlaBlaCar_FR

20€

Trusted Ridesharing

People Powered Travel

A travel search engine

A trusted community

@BlaBlaCar_FR

Fast growth

@BlaBlaCar_FR

8/49

Why Infrastructure Automation ?

9/49

Few numbers of BlaBlaCar today

2 Data Centers in Paris

● 242 servers

● 109 virtual servers (VMWare)

● 110 EC2 instances during working day

– 20 instances the rest of the time

Our Continuous Delivery at the top.

Jealous from the development team !

11/49

We have a strong agility culture

● Push our main app on production 10x a day● Developers make the push themselves

● App can be pushed only after a Bamboo's merge● Bamboo merge only after thousands of tests

12/49

tools to manage them all

Old Infrastructure Overview

14/49

Physical Server

● 3 racks on a single DC

● 12 servers per rack

● More than 20 differents

hardware configurations

15/49

Cloud (Amazon EC2)

● Only 10 to 20 instances always running

● Used for elasticity

● 10 to 20 instances running for test (Bamboo)

16/49

Network

● Strong Team who designed a very strong network– Rémi (@shtouff)

● Not a problem for us to scale the network

Automate the Configuration

18/49

Choosing the right tool

19/49

Improve our Service Deployment

● Started to work on a front web server– Ease configuration deployment

– Ease a new installation

● Then worked on all servers– First just for configuration

– Then all service deployment

Hardware Industrialization

21/49

Haute Couture

● Before our automation process:– One server configuration by service

– Multiple services

22/49

Haute Couture - Caveats

● Tailored server are complicated to design● Hard to upgrade

23/49

Next Step - Virtualization

● Create a virtualization platform– 4 big servers

– A big disk array

24/49

Virtualization - Caveats

● I/Os Performances compare to physical server● Network Performances

25/49

Cloud: Why on premise in 2015 ?

● Data privacy concerns● Limited elasticity required● Cost efficiency● History & Culture

26/49

Now – Standard Servers

● Define a single 1U server– Dual-CPU (12 cores)

– 128 Go of RAM

– Disk configuration vary on 3 flavors:● Only 2 SSD mirrored (RAID) ● Fast disk (SSD) needed● Very big capacity (lot of disks)

Automate OS Installation

28/49

Automate debian installation

● All our infra is debian based● PXE● ipmi

● preseed– (thx to @jbfavre)

29/49

Virtualization => Template

● Work well on EC2 (only option for instance install)

● Never used in VMWare– Same install as physical server debian based

(thanks to Chef)

30/49

Define our needs to choose a tool

● IP allocation● Bootstrap chef-client● PXE preseed debian● Can install either a physical, virtual or cloud server● Configure the BMC interface

– HP (iLo)

– Dell (DRAC)

– SuperMicro

31/49

● Because Simon (@s_ourea) chose it !– More likely to be maintained

– Big community

● Fits our needs

Hardware Availability

33/49

We need more

● Chef / Foreman, great tools to:– Install OS

– Install all services and configure them

● No easy solution when you need a specific hardware on never used server

34/49

Collins !

● Nice tool to manage our hardware inventory.● A lot of good reasons to choose it:

– Simon (@s_ourea) chose it

– Fit our needs

● Linked with

Foreman

Hey ? What about the initial setup of a physical server ?

36/49

Install servers into DC● Limited added value in the team racking servers● Time consuming● DCs can be far away

● Simon(@s_ourea) & Rémi(@shtouff) pushed a new idea– Buy servers only when we can use a full rack

– Write an installation documentation

– Choose a company to rack

37/49

Our procedure to have a new rack

● Buy a new rack into the DC

● Our network team installs 2 switchs in the rack.

● They add links to our core network equipments

● Hey! Let's ask to our provider to rack and setup all servers, following Simon's documentation

38/49

The typical life of a server

● Discovery, and go into Collins

● Another team takes it and installs it via Foreman

● Chef is called via Foreman

● Chef is always running

Tests, and Continuous delivery really ?

40/49

Chef and tests

● Yes, we test all of our

infrastructure's services !

● Thanks to Chef and all of it's community:

KitchenCI + vagrant + serverspec, chefspec

41/49

Chef - Cookbook life

● Each cookbook in it's own git repository

● First test and merge with Bamboo

● Push in production with a new version number

● All servers converge to the desired state

42/49

What do you test ?

● Everything! (one day perhaps)– Server install

– Bootstrapping

– Convergence

– Clustering /!\ (not automatic)

– Version upgrade ( one day !)

– Coffee quality

43/49

Continuous delivery, really ?

● New cookbook versions pushed 10x times / day

● Provision several new servers / VM / instances by day

● Buy a rack full of servers, and install them in less than a day

So cool ! But...What will you do in the future ?

45/49

Container

CoreOS

46/49

What we will continue to do

● Test our infra (with or without container)● Industrialize our processes● Spread knowledge to all other teams● Dream solutions, and finally build them

● Listen to Simon (@s_ourea)

Conclusion

48/49

What to keep in mind

● Involve overall company

● Support company growth

● Don't get attached to tools

● Keep a “small team” spirit

That's all folks!

Thanks you!

Nicolas Blanc(@thewhitegeek)