EVOLUTION OFTICKETMASTER’S JOURNEY TO DEVOPS

WINOPS LDN SEPTEMBER 2017

#WinOps

@connonm

@tmTech

WHO AM I & WHY AM I HERE?

Connon MacRaeVP, Technical Operations(International)

#WinOps

@connonm

@tmTech

HISTORY IS USEFUL

CONTEXT IS EVERYTHING

DON’T BE AFRAID

MICROSOFT VS LINUX

MICROSOFT LINUX

NO MORE CYGWIN ON WINDOWS#WinOps

@connonm

@tmTech

#WinOps

@connonm

@tmTech

WHO IS TICKETMASTER?

LIVE NATION ENTERTAINMENT – 42 MARKETS

Live Nation & Ticketmaster

Live Nation ONLY

Ticketmaster ONLY

#WinOps

@connonm

@tmTech

#WinOps

@connonm

@tmTech

THE HOST

8 HOURS

5-9 HOURS

1975 – TM started with Albert Leffler, Peter Gadwaw, Gordon Gunn . . . . . . Also Bill Gates and Paul Allen develop a BASIC program for the Altair 8800, Microsoft becomes a registered trademark. BetaMax is introduced. Motorola obtains patent for mobile phone!

1976 – TM Starts . . . . IBM Launch the first laser printer.1977 – ELO is first ticketed event at Johnson Gymnasium, University of New Mexico1978 – Norway is the first international client with Oslo Nye Theatre1981 – Ticketmaster UK Opens above the Albery Theatre1983 – Ticketmaster UK starts to work with Arsenal1988 – Ticketmaster Australia operations begin

THE THING YOU REALLY NEED TO KNOW#WinOps

@connonm

@tmTech

I WANT THE BEST SEAT(S)

ONSALE . . . 9AM . . .

#WinOps

@connonm

@tmTech

GROWING FAST

8-9 HOURS

5-9 HOURS

5-6 HOURS

10 HOURS

1-4 HOURS

5-9 HOURS

1991 – Ticket Tron in North America acquired 1996 – UEFA Euros with Ticketmaster United Kingdom1996 – First Ticketmaster internet sale!1997 – Archtics (Sports & Arts) integrated with and later ‘DSA’ acquired. 1997 – Synchro Systems (later Ticketmaster Sport)1997 – Ticketmaster Ireland as a joint venture1998 – Mexcio starts operations1998 – ticketmaster.co.uk goes online, Windows NT 4 1999 – Joint venture in Australia1999 - Rugby World Cup in Wales2000 – TicketWeb, Vista and Admission Canada 2001 – First deal with Clear Channel – later to become Live Nation2001 – Norway Billetservice acquired2002 – Manchester Commonwealth Games 2002 – Netherlands & Denmark acquired2004 – Athens Olympics2004 – Sweden and Finland acquired2005 – New Zealand established and Kartenhaus in Germany acquired2006 – Acquires TickTackTickets in Spain and Biletix in Turkey2007 – TM China joint venture2008 – Beijing Olympics2008 – TM acquires TicketsNow & GetMeIn, 2008 - Paperless tickets debut in North America 2008 – Paperless debuts in UK with 18k people attending at the O2 Arena2008 – Ticketmaster Spun off from IAC2008 - FrontLine Entertainment management acquired2009 – TM UK breaks records with Michael Jackson, 50 dates at the O2.2010 – LN and TM Merge to become Live Nation Entertainment2010 - Ticnet in France acquired2011 – Expansion in Spain with ServiCaixa2012 – Ticketing for London2012, over 10.9 million tickets sold and processed. Unprecedented at the time.2013 - Launches in the middle east with UAE 2014 – Glasgow Commonwealth Games. 2014 - TM acquires Seatwave across Europe. 2014 - Middle east expands to Qatar2014 – Relaunch DE, ES, acquires PL and Sherpa in Belgium. 2015 – Universe and Frontgate join the company2015 – England Rugby World Cup2016 – TM acquires Tickethour2017 – TM acquires TicketPro in Czech Republic and launches Ticketmaster Israel

14 CORE TICKETING PRODUCTS

#WinOps

@connonm

@tmTech

THE INTERNET - 1998

WWW

NT4SQL

THE ‘HOSTS’ - VAXEN 1999

8%?

20%

GULP

#WinOps

@connonm

@tmTech

REBUILD v1 – 2000

WWW

LINUX

ORACLE

THE ‘HOSTS’ - VAXEN

MySQL

#WinOps

@connonm

@tmTech

THEN THESE . . . . .

SOX -2002

PCI v1 -2004

#WinOps

@connonm

@tmTech

REMEMBER THESE?

#WinOps

@connonm

@tmTech

WE BUILT THIS . . .

DEV QA

CAP

STAGE

5-8 HOURS

PRODUCTION

. . . WE WERE PROUD OF IT!

LESS THAN . . . OPTIMAL?

#WinOps

@connonm

@tmTech

Group A

WINDOWOPS

ServiceDelivery

WINDOWSSYSTEMS

DBA TEAM

LINUX OPS

LINUXSYSTEMS

Group B Group CPLATFORM

Product Support

CLOUDSYSTEMS

TOOLING

NETWORKINGDATACENTRE

INTERNATIONAL TECHNOLOGY TEAMS IN 2016

#WinOps

@connonm

@tmTech

SECURITY

CORP IT

ARCHITECTURE

NA TEAMS

ProductGroup 1

ProductGroup 2

ProductGroup 1

ProductGroup 2

ProductGroup 1

ProductGroup 2

ProductGroup 3

ProductGroup 3

THE MORE YOU TIGHTEN YOUR GRIP, THE MORE STAR SYSTEMS WILL SLIP THROUGH YOUR FINGERS

#WinOps

@connonm

@tmTech

WHERE HAVE WE BEEN?

Engineering Ops

Syseng

Database

Engineering Ops

SysengDatabase

Engineering Ops

SysengDatabase

#WinOps

@connonm

@tmTech

WHERE HAVE WE BEEN?

#WinOps

@connonm

@tmTech

Engineering

Ops

Syseng Database

Platform

PUSH ME, PULL YOU

#WinOps

@connonm

@tmTech

PUSH ME, PULL YOU

#WinOps

@connonm

@tmTech

Ops & Reliablity

ServiceManagement

Ops & Reliablity Ops & Reliablity

PLATFORMENGINEERING

Product Support

Runtime

Services& Data

Tooling

Infrastructure(including DC)

NetworkEngineering

Technical Service Delivery

SCALING THE SILOS, LOWERING THE WALL

PLATFORMINFRASTRUCTURE

#WinOps

@connonm

@tmTech

Group A Group B Group C

ProductGroup 1

ProductGroup 2

ProductGroup 1

ProductGroup 2

ProductGroup 1

ProductGroup 2

ProductGroup 3

ProductGroup 3

Group B

ProductGroup 1

ProductGroup 2

Ops & Reliablity

HOW CAN WE EVOLVE?

#WinOps

@connonm

@tmTech

CAT = LOVE = PRODUCT!#WinOps

@connonm

@tmTech

TOO LONG, EXPENSIVE

#WinOps

@connonm

@tmTech

DOES NOT FIT ANYMORE#WinOps

@connonm

@tmTech

PLAY (RE)NICE? KILL -9

#WinOps

@connonm

@tmTech

UNACCEPTABLE!!!!!!

TICKETMASTER CLOUD#WinOps

@connonm

@tmTech

SOMEBODY ELSES BOX

TRANSPARENT COSTS

#WinOps

@connonm

@tmTech

CONTAINERS – PART I

#WinOps

@connonm

@tmTech

CONTAINERS – PART II#WinOps

@connonm

@tmTech

31

(MORE) MODERN TECH

CHANCE TO CLEAN UP

WHAT ARE WE USING?

#WinOps

@connonm

@tmTech

TECH MATURITY

Open Source @https://github.com/Ticketmaster/techmaturity

1. CONTAINERISE your application with Docker or AUTOMATE your Operating Systems with CHEF

2. IMPLEMENTATE BETTER INSTRUMENTATION - no SSH or RDP in production!Log aggregation and rotation, metrics

3. Use SYNTHETIC MONITORING to understand the health of your productDefine SLAs, track & alert on availability and performance

4. AUTOMATE YOUR BUILD PIPELINE by reusing common toolsTerraform & GitLab CI are the products of choice

5. GET SECUREPass pen/static testingNo passwords/keys in configs

6. DESIGN A SHARED NOTHING ARCHITECTURENo shared filesystems (e.g. NFS, CIFS)

7. BUILD FOR AVAILABLITY- NO SINGLE POINTS OF FAILUREUse auto-scaling groups and without hard-coded IPs

OUR PRINCIPLES FOR MODERNISATION

#WinOps

@connonm

@tmTech

SUMMARY#WinOps

@connonm

@tmTech

HISTORY IS USEFUL

CONTEXT IS EVERYTHING

DON’T BE AFRAID

Q & A

#WinOps

@connonm

@tmTech