Implementing Rolling Upgrades using Mesos, Marathon, Docker, HAProxy

22
Implementing Rolling Upgrades Containerized Environment powered by Mesos, Marathon, Docker and HAProxy

Transcript of Implementing Rolling Upgrades using Mesos, Marathon, Docker, HAProxy

Implementing Rolling Upgrades

Containerized Environment powered by Mesos, Marathon, Docker and HAProxy

1. Deploy NEW application (with a new App ID) with the minimum set of instances (eg: 1 in the beginning)

as needed to manage the traffic

2. Wait for the NEW instance(s) to go healthy

3. Add the NEW Instance(s) to the HAProxy Pool

4. Load gets balanced between OLD and NEW Pool

5. DISABLE one of the OLD instances in the HAProxy Pool

6. Wait for the HAProxy Connection to drain on the Disabled OLD Instance

7. After waiting period is over, Remove the Disabled OLD Instance by issuing Docker STOP on the

Container

8. Scale the NEW Instances and let them add to the HAProxy Pool

9. Repeat the steps 4 - 8 till all OLD instances are removed, and all the NEW instances are added to

HAProxy

10. Delete the OLD Application

ALGORITHM: Rolling Upgrades

OLD OLD OLD OLD OLD OLD

HAProxy

CURRENT STATUSOld Instances are running and load balanced by HAProxy

OLD OLD OLD OLD OLD OLD

HAProxy

L4

DNS

Customer

Service Port 1

CURRENT STATUSOld Instances are running and load balanced by HAProxy

NEWOLD OLD OLD OLD OLD OLD

HAProxy

Minimum required to serve just

enough traffic

NEW INSTANCE DEPLOYED1 Instance of the New Version of Application is deployed

NEWOLD OLD OLD OLD OLD OLD

HAProxy

NEW INSTANCE DEPLOYEDHealth Check Passed

NEWOLD OLD OLD OLD OLD OLD

HAProxy

NEW INSTANCE DEPLOYEDHealth Check Passed. Added to Load Balancer.One Old Instance is DISABLED from Load Balancer

OLD OLD OLD OLD OLD OLD

HAProxy

L4

DNS

Customer

Service Port 1

NEW

NEW INSTANCE DEPLOYEDAdded to Load Balancer. 16% new requests goes to NEW Instance. No New Request will go to Old Instance #1

disabled

NEWOLD OLD OLD OLD OLD OLD

HAProxy

ZDD

Control

Marathon

Mesos

Control

HOW IS ROLLING UPGRADE IMPLEMENTEDZDD requests HAProxy to DISABLE one of the OLD Instance

ZDD is a Python Script part of the Marathon-lb Project from Mesosphere

disabled

NEWOLD OLD OLD OLD OLD OLD

HAProxy

ZDD

Control

Marathon

Mesos

Control

HOW IS ROLLING UPGRADE IMPLEMENTEDZDD now waits for N seconds to drain any existing connections

N seconds

NEWOLD OLD OLD OLD OLD OLD

HAProxyMesos

STOP

ZDD

Control

Marathon

Control

HOW IS ROLLING UPGRADE IMPLEMENTEDAfter N seconds, ZDD requests Marathon to STOP one of the Old Instances

NEWOLD OLD OLD OLD OLD OLD

HAProxyMesos

STOP

M seconds

ZDD

Control

Marathon

Control

HOW IS ROLLING UPGRADE IMPLEMENTEDMesos invokes Docker STOP on the Old Container, and waits for M seconds

NEWOLD OLD OLD OLD OLD

HAProxyMesos

Mseconds

STOP

N seconds

OLD

NEW NEWOLD OLD OLD OLD OLD

HAProxyMesos

Mseconds

STOP

N seconds

NEW NEWOLD OLD OLD OLD

HAProxy

M seconds

Mesos

STOP

NEW

N seconds

NEW NEWOLD OLD OLD

HAProxy

M seconds

Mesos

STOP

NEWNEW

N seconds

NEW NEWOLD OLD

HAProxy

M seconds

Mesos

STOP

NEWNEWNEW

N seconds

NEW NEWOLD

HAProxy

M seconds

Mesos

STOP

NEWNEWNEWNEW

N seconds

NEW NEW

HAProxyMesos

NEWNEWNEWNEWNEW

NEW NEW

HAProxyMesos

NEWNEWNEWNEWNEW

ROLLING UPGRADE COMPLETE !

NEW NEW

HAProxy

NEWNEWNEWNEW

L4

DNS

Customer

HOW IS ROLLING UPGRADE IMPLEMENTEDNow all NEW Instances have been added to HAProxy Pool

APP APP APP APP APP APP

HAProxy

ZDD

Control

Marathon

Mesos

Control

DEV TEST STAGING PROD JENKINSPIPELINE

ROLLING UPGRADESHAPPEN ACROSSALL ENVIRONMENTZDD orchestratesthe deployment