Misadventures in Continuous Delivery Dan DeMeyere, Chief Product Officer @ thredUP

2

BACKGROUND

● thredUP: the world’s largest online consignment store

● Joined thredUP in 2010 as a full-stack engineer

● Responsible for engineering + product (~60)

● Cross-functional pod system

● Isomorphic React/Apollo SPA, GraphQL orchestration, Ruby/Java/Node on the back-end

● Migrated to k8s three years ago

Confidential 3

4

CONTINUOUS DELIVERY

“The ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.” - Jez Humble

5

ENGINEERING MISADVENTURES

6

WHY FOCUS ON MISADVENTURES?

● To avoid making the same mistakes we did

● Everyone can relate

● Learnings in how to scale are hard to get

Confidential

‹#›

You know there’s a problem with continuous delivery when…

RAISE YOUR HAND IF:

Have you ever worked in E-Commerce?

RAISE YOUR HAND IF:

Have you ever worked in E-Commerce? Have you ever heard of “Cyber Week”?

RAISE YOUR HAND IF:

Have you ever worked in E-Commerce? Have you ever heard of “Cyber Week”? Have you had to do any dev work for anything related to “Cyber Week”?

11

CYBER WEEK IN E-COMMERCE

12

CYBER WEEK IN E-COMMERCE

13

CYBER WEEK IN E-COMMERCE

Confidential

‹#›

You know there’s a problem with continuous delivery when…

1. Your team issues code freezes

15

HOW TO PICK THE ENGINEER

16

OUR DEPLOY PIPELINE

17

CI PIPELINE CHECKLIST

● Is your end-to-end deploy pipeline quick?

● Is it safe? Does it gracefully degrade?

● Can you roll back with ease?

● Is the deployment process sustainable?

● Do engineers have confidence when deploying?

18

CI PIPELINE CHECKLIST

● Is your end-to-end deploy pipeline quick?

● Is it safe? Does it gracefully degrade?

● Can you roll back with ease?

● Is the deployment process sustainable?

● Do engineers have confidence when deploying?

19

LAZY ENGINEERS

“I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it.” - Bill Gates

20

CLEVER ENGINEERING

Confidential

‹#›

You know there’s a problem with continuous delivery when…

1. Your team issues code freezes

2. You see your team implementing delivery workarounds

22

CONTINUOUS DELIVERY

“The ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.” - Jez Humble

RAISE YOUR HAND IF:

Have you ever tried to decompose a monolith?

24

MONOLITH DECOMPOSITION Has anyone ever done something small and easy like decompose a monolith?

25

MONOLITH DECOMPOSITION ● Migrate RethinkDB to AuroraDB

● Pull out user personalization into separate service

● Pull out shopping cart responsibilities into separate service

● Introduce RabbitMQ so the back-ends can talk to each other

● Consolidate all client API requests to GraphQL orchestration layer

● Pull out auction responsibilities into separate service

26

MIGRATION DEPLOYS (IN THEORY - BEFORE)

27

MIGRATION DEPLOYS (IN THEORY - DURING)

28

MIGRATION DEPLOYS (IN THEORY - AFTER)

29

MIGRATION DEPLOYS (IN PRACTICE - BEFORE)

30

MIGRATION DEPLOYS (IN PRACTICE - DURING)

31

MIGRATION DEPLOYS (IN PRACTICE - AFTER)

32

WHY HOLE-SHOT DEPLOYS ARE PROBLEMATIC

● Staging != Production ● Extended QA in staging -> elongates the development process

● Risk for the business -> creates unnecessary stress for the team

● Repeated stress -> recipe for burn out

Confidential

‹#›

You know there’s a problem with continuous delivery when…

1. Your team issues code freezes

2. You see your team implementing delivery workarounds

3. Large system migrations are stressful

34

FEATURE BRANCH WORKFLOW (IN THEORY)

35

FEATURE BRANCH WORKFLOW (IN PRACTICE)

36

WHEN YOU HAVE TO ROLL BACK THE FEATURE YOU DEPLOYED

Confidential

‹#›

You know there’s a problem with continuous delivery when…

1. Your team issues code freezes

2. You see your team implementing delivery workarounds

3. Large system migrations are stressful

4. Exposing a feature to customers is harder than flipping a switch

38

CONTINUOUS MERGE WORKFLOW

39

CODE REVIEW FOR LONG PULL REQUESTS

40

CONTINUOUS WORKFLOW BENEFITS

● Bite-sized pull requests -> easier code review ● Continuous deploys -> earlier production feedback

● More granular time estimations -> consistency -> predictability -> sustainable

● Last 10% of the project is actually the last 10%

Confidential

‹#›

You know there’s a problem with continuous delivery when…

1. Your team issues code freezes

2. You see your team implementing delivery workarounds

3. Large system migrations are stressful

4. Exposing a feature to customers is harder than flipping a switch

5. Engineers think velocity and stability are mutually exclusive

42

CONTINUOUS DELIVERY

“The ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.” - Jez Humble

43

“SHOP HER CLOSET” LAUNCH

44

LAST MINUTE CHANGES

45

WHEN YOU REALIZE YOU HAVEN’T ROLLED BACK ON K8S BEFORE

46

WHEN YOU’RE TRYING TO THINK THROUGH A CI BYPASS

47

LAST MINUTE CHANGES

Confidential

‹#›

You know there’s a problem with continuous delivery when…

1. Your team issues code freezes

2. You see your team implementing delivery workarounds

3. Large system migrations are stressful

4. Exposing a feature to customers is harder than flipping a switch

5. Engineers think velocity and stability are mutually exclusive

6. You have Devs who don’t Ops

Confidential

‹#›

You know there’s a problem with continuous delivery when…

1. Your team issues code freezes

2. You see your team implementing delivery workarounds

3. Large system migrations are stressful

4. Exposing a feature to customers is harder than flipping a switch

5. Engineers think velocity and stability are mutually exclusive

6. You have Devs who don’t Ops

7. You’re satisfied with continuous delivery

50

CLOSING THOUGHTS

“The most important characteristic of high performing teams is that they are never satisfied: they always strive to get better.” - Jez Humble

THANK YOU

Misadventures in Continuous Delivery Dan DeMeyere, Chief Product Officer @ thredUP Twitter: @dandemeyere Medium: medium.com/@dandemeyere Email: dan@thredup.com