SCORE!SUPERIOR DIGITAL EXPERIENCES WITH MLS
MAY 10, 2016
Brian Aznar(Director of Engineering, MLS Digital)
Steven Merrill (Director of DevOps, Phase2)
BACKGROUND MLS & DIGITAL PRODUCTS
MAJOR LEAGUE SOCCER STATE OF THE LEAGUE
‣ 20 current teams with plans to expand to 24 by 2020 ‣ Adding state of the art soccer-specific stadiums around the US -
average in stadium audience exceeds NHL and NBA ‣ Expanding our reach in the US and around the world with great new
TV distribution deals ‣ Young and multicultural fan base that is hungry for digital
MLS FANBASE
‣ Avids ‣ Regularly watches and
attends MLS matches ‣ Consume soccer related
content ‣ Regularly visits
MLSsoccer.com and other soccer specific sites
‣ Casuals ‣ Occasionally
watches and attends MLS Matches
MLS DIGITAL PRODUCT ECOSYSTEM
‣ Web: 20 club websites + 1 league web site using Drupal 7 ‣ Mobile Apps: iOS, Android ‣ Connected TV: Roku, Chomecast, Apple TV ‣ Services: Content API, Video & Stats ingest pipelines,
Content delivery, etc … ‣ Infrastructure: AWS
2015 CASE STUDY
GOAL
‣ Modernize and stabilize MLS’ web platform (backend and frontend) and successfully migrate 21 sites by the end of 2015 to AWS and D7.
THE TEAM
PHASE2
‣ Founded in 2001 ‣ Offices in DC, NYC, San Francisco, and Portland ‣ Committed to open source technology ‣ Focus on how content is delivered, how
teams collaborate, and how audiencesexperience information
PHASE2 & MLS
‣ Long-term partnership over 3 years ‣ Helped with D7 platform build ‣ March 2015 - we kicked off their migration to
MP7 Drupal platform and into AWS
PHASE2 & MLS TECHNOLOGY TEAM
‣ Drupal Developers (Phase 2) ‣ Fullstack Javascript Developers: React, Backbone, Node.js & Go (MLS) ‣ DevOps (Phase 2) ‣ Process = Scrumban
‣ Maximize development velocity, agility and responsiveness to business needs.
‣ Centralized Development Team ‣ Moving from segmented teams towards a single cross disciplinary team.
CHALLENGES & SOLUTIONS
CHALLENGES WE NEEDED TO ADDRESS
1. Outdated UX originally built for desktop viewing only
2. Inflexible Authoring - Needed an efficient and scalable way to give clubs more control to build out their own content
3. Wasted Server Capacity - High variability in traffic patterns means planning for the worst
CHALLENGES WE NEEDED TO ADDRESS
4. High Fixed infrastructure costs
5. Poor Performance
6. Technology Soup - Go, node, CoreOS/Fleet, Docker,PHP/Drupal (Multiple Versions) and Salt all managed with a lean team
7. Aggressive Migration - 21 websites and No downtime
a. Avoiding major tent pole events that varied across clubs
1. MODERN UI EXPERIENCE
‣ Building blocks that editors can put together ‣ All features were designed to be responsive ‣ Video highlights are pushed from MLS’s video providers ‣ Import of MLS entities (teams/players/seasons) from MLS’s pipeline
2. EDITORIAL FLEXIBILITY
3. SPIKEY TRAFFIC
‣ Traffic Spikes (avg 4x) centered around games and events ‣ League represented about half the traffic on average ‣ Great candidate for autoscaling
4.a HIGH INFRASTRUCTURE COST
‣ 2 Infrastructures (One foot in the cloud) ‣ Drupal running in Rackspace Servers ‣ Services Running in AWS
‣ Drupal running on fixed set of large servers configured to handle the average amount of traffic plus physical devices (firewalls, NetApp storage…)
‣ Paying for unnecessary resources needs during off peak and off season traffic
4.b MOVING INTO AWS
‣ Rackspace => AWS - everything in one cloud infrastructure ‣ Autoscaled Instances ‣ Additional operational benefits by using:
‣ Amazon RDS, ElastiCache, S3
AUTOSCALING‣ Reduces stress ‣ Automatic healing ‣ Performance benefits
‣ Manual adjustment up and down ‣ Automatic scaling based on metrics
‣ Cost benefits
‣ For autoscaling, you should build cattle, not pets ‣ Previous MLS autoscaling provisioned from scratch
‣ Phase2 built “fat” AMIs with all software + code ‣ Far faster deployment time ‣ An AMI can come up if everything else is down
‣ See Steven’s DevOps track session tomorrow for more info!
AUTOSCALING BACKGROUND
A USUAL MONTH
THE OVERALL RESULT
5. POOR PERFORMANCE
‣ Unsuccessful attempt at cloud multi region architecture ‣ Caching anomalies between Drupal in 2
datacenters ‣ Slow cache clearing bottlenecks
‣ Slow server response times > 1000 ms
‣ Right-sized AWS instances to their workloads ‣ Upgraded to PHP 5.5 for a performance boost ‣ Optimal instance size + PHP 5.5 = 40% increased throughput ‣ Added stability fixes
PERFORMANCE TUNING: QUICK WINS
‣ Load Testing & Application Monitoring to find and remove bottlenecks ‣ Disabled DB read/write split for a massive speed increase ‣ Doubled the size of memcache instance ‣ Killed legacy code that cleared all of Drupal’s caches ‣ Results:
‣ Instance sizes reduced by half ‣ Massive decrease in average page render time
PERFORMANCE TUNING
6. TECH COMPLEXITY
‣ Clubs and League Content Sites ‣ Drupal (6/7) ‣ PHP (5.1/5.3) ‣ Jenkins / Salt for Deployment
& Configuration Management
‣ Web Services / Stats & Video Ingest Pipeline / Misc Web Apps ‣ Node.js & Go ‣ CoreOS/Fleet/Docker & Teamcity for
Deployment & Configuration Management
‣ Server Configuration Drift
7. MIGRATION
‣ Seattle, Orlando, NYCFC were in place ‣ Large project management lift
‣ Coordination with stakeholders ‣ Training schedule ‣ Working with club schedules /
game schedules
‣ Central Digital Club Services team played a huge role by managing coordination w/ clubs and drupal editors
‣ Drupal multisite ‣ Saved server resources ‣ Made deploys smaller ‣ Worked for MLS Digital specifically because of
the integrated nature of their Drupal platform ‣ No bespoke development per-club ‣ Centralized feature development ‣ Far lower support burden (6 people)
MULTI-TENANT SETUP
PROJECT MILESTONES
‣ April 2015: Migrated first team site (NY Red Bulls)
‣ July: Turned on autoscaling ‣ April - November: Migrated MP7 ‣ October: League site launched! ‣ October: Decision Day
‣ December: Turned off Rackspace hosting ‣ January 2016: Helped hire in-house
DevOps Engineer ‣ March: Successful 2016 Season Kickoff
TOOLS
TOOLS
‣ New Relic for Application Performance Management / Instrumentation
‣ Blaze meter for Load testing ‣ Datadog for Infrastructure Metrics
NEW RELIC
‣ Table Stakes Tooling for the Modern Technology Team
‣ Real Time End to End Client & Server metrics and performance measurement ‣ Introspection of everything from end
page rendering on a user’s browser to PHP & DB bottlenecks on the server
‣ Tiers: ‣ Free - 1 day retention -
good enough for load tests ‣ Pro - 90 day retention
REALTIME FULL STACK METRICS: CLIENT TO SERVER TO REPO
CLUB ROLLUPS
REALTIME FULL STACK METRICS: CLIENT TO SERVER TO REPO
CLUB ROLLUPS
MODULE, HOOK & VIEW PERFORMANCE
GLOBAL ROLLUPS
MULTI SITE TOP LEVEL METRICS: AGGREGATE
CLUB ROLLUPS
MULTI SITE TOP LEVEL METRICS: INDIVIDUAL SITE
SUMMARY
OUTCOMES
‣ Migrated fully onto AWS ‣ Lowered hosting bill significantly by leveraging AWS features ‣ Helped reduce support load through policy changes
and new MLS hires ‣ Go see Steven’s session for more info
on immutable infra!
PROBLEMS ADDRESSED IN 2015
1. Outdated front-end UI experience originally built for desktop viewing only
2. Club editorial teams needed more control and flexibility to build content that met their needs but we needed it all to fit within the same brand ecosystem
3. High variability in traffic patterns resulting wasted capacity
PROBLEMS ADDRESSED IN 2015
4. High fixed infrastructure costs
5. Managing multiple technologies (CoreOS/Fleet,Salt, PHP, Go, node) and versions (PHP, Drupal) with a small team
6. Migrating 21 websites without any down time and avoiding major tent pole events that varied across clubs
THE FUTURE
THE FUTURE
‣ Improving performance ‣ Improving editorial usability ‣ Further consolidation & simplification
of tech stack ‣ #docker!
‣ Preparing for omnichannel
CHECK US OUT!
labs.mlssoccer.com
mlssoccer.com
phase2technology.com
SOCCER STADIUMSNEAR YOU!
QUESTIONS
So How Was It? - Tell Us What You ThinkEvaluate th is session - ht tps: / /events.drupal .org/neworleans2016/schedule
Thanks!
Top Related