DockerCon SF 2015: Maintaining the Official Node.js Docker Image
DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker
-
Upload
docker-inc -
Category
Technology
-
view
6.275 -
download
0
Transcript of DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker
![Page 1: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/1.jpg)
It's in the game: The path to micro-services at Electronic Arts with Docker
Andrew HatelyIBM CTO Cloud Architecture
Scott PorterEA Cloudcell Server LeadMelbourne Firemonkeys
![Page 2: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/2.jpg)
GAMING MARKET GROWING ACROSS ALL PLATFORMS AND REGIONS
Mobile PC/F2P Console
NORTH AMERICA LATAM EUROPE ASIA/ROW
GROWTH YoY 12%
REVENUES 2015est.
21BGROWTH YoY 19%
REVENUES 2015est.
4BGROWTH YoY 7%
REVENUES 2015est.
17BGROWTH YoY 9%
REVENUES 2015est.
40B
17% 46% 37%41% 16% 43%
195M 182M 553M 817M
33% 24% 43% 51% 44% 5%
2015 est. Revenues Consist of Software Gaming Market
![Page 3: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/3.jpg)
MOBILE FUELING INDUSTRY GROWTH
2009 2010 2011 2012 2013 2014 2015$3.3 $4.3 $6.1
$9.9
$17.0
$27.8
$34.0
Mobile games industry has grown 10x in less than a decade
Mobile Gaming Revenue, WW $B
![Page 4: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/4.jpg)
Mobile Game Server architecture/requirements• Mobile Gaming Back-end is a very typical Web application• Frequent updates to game content - infrequent updates to game
engine• Daily peak users 150,000 to 250,000 simultaneous users and 1-2
million daily • Need to scale-up/down around release and content pack releases• Stateless front-ends with memory caching and both SQL and no-
SQL DB• 50TB per day of data - Increasing use of analytics on user
experience, recorded game telemetry, replays, etc.
Firew
all
Load
Ba
lanc
er
NGNI
X
Game Server REDIS
Memcache
MySQL cluster
PHPPHPPHPPHPPHP
Object Storage
![Page 5: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/5.jpg)
Demo – Docker for Gaming
![Page 6: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/6.jpg)
![Page 7: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/7.jpg)
• Our current Ops complexity
• Partnership with IBM
• Why we believe Docker can help
• How we integrated Docker
![Page 9: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/9.jpg)
EA MELBOURNE FIREMONKEYS
• Moble studio based in Melbourne, Australia
• Formed in 2012 from two EA Mobile studios, Iron Monkey and Firemint
• Focused on developing the highest quality mobile titles
![Page 10: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/10.jpg)
• Firemonkeys in house Client/Server Tech Stack
• Mobile focused connected features
• Originally developed by Firemint for Flight Control
![Page 11: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/11.jpg)
Integrated into all Firemonkeys live titles
![Page 12: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/12.jpg)
• Live Service Games
• Bi monthly client releases
• No downtime
What do we do?
![Page 13: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/13.jpg)
• 13.5k Peak Requests/s
• ~900M Requests/day
• ~430k Peak Concurrent Players
• 5M – 8M Daily Active Players
• > 100 TB Binary Data
• 170 Hosts (60 Live Hosts)
• Team of 7
Current Scale
![Page 14: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/14.jpg)
• Baremetal Hosts
• Nginx, PHP, Memcache, MySQL, Redis
• Batched client requests, short-lived, stateless
• Requests can touch any datastore
Architecture
![Page 15: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/15.jpg)
Game Logic Data StorageLoad Balancing
Architecture
External Services
StatsD / Diamond / Graphite / Graphana
Object Storage
Datadog
![Page 16: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/16.jpg)
• Automated Development / Manual Live process
• In house database migration and code deployment system
• Bash & rsync
• Rollbacks are handled via code duplication
Current Deployment Methodology
![Page 17: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/17.jpg)
• Asynchronous processes (workers) manually configured ( services, cron jobs )
• Brittle.
• SaltStack is awesome!
• But still slow, complex, no HA
Current Orchestration Methodology
![Page 18: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/18.jpg)
• Moved binary data to SoftLayer Object Storage (Open Stack Swift)
• Currently provision for Peak
• Shard live environments per game title
• Interested in transitioning to microservices
Current Scalability
![Page 19: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/19.jpg)
• Improved deployments
• Growing ecosystem
• Scalable
• Easy transition, run side by side with existing stack
Why Docker?
![Page 20: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/20.jpg)
• Focused on the Game Logic Layer
• Decomposition• Nginx• Admin• Application• Workers
Packaging Cloudcell
![Page 21: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/21.jpg)
• Generic Containers
• Run multiple containers, not processes
• Tradeoff considerations
Packaging Cloudcell
![Page 22: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/22.jpg)
Code
Building Images
Jenkins
Docker Registry
Base Images
nginx
php-fpm
php-worker
Built Images
php-fpm-nfs-0001
php-fpm-nfs-0002
php-worker-a-1
nfs-0002.tar
worker-a-1.tar
nfs-0001.tar
Dev Env
Jenkins
Application Code (SVN)
![Page 23: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/23.jpg)
• Fleet
• Unit files per service• Nginx• Admin• Application• Each worker type
Orchestration
![Page 24: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/24.jpg)
Load Testing
Host
Baremetal
nginx
Application
Worker A Load Generation
LTG LTG
LTG LTG
LTG
LTG
Host
Docker
Docker
nginx
php-fpm-nfs-0001
php-worker-a-1
etcd
Fleet
![Page 25: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/25.jpg)
• Overall 5 – 10% overhead running Load Test with Docker
Load Testing Results
PLACEHOLDER NEED A REAL GRAPH HERE!
![Page 26: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/26.jpg)
• Docker Private Registry
• Jenkins• Studio Jenkins ships code to build machine• Build machine builds and commits images to
Registry• Generates Fleet configuration from templates• Executes Fleet deployments
Deployments
![Page 27: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/27.jpg)
• Config Editor• Bridges Cloudcell Application deployments to
environment state• Provides a simple UI for building, deployments,
and state via Jenkins and Fleet
Deployments
![Page 28: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/28.jpg)
Deployments
Docker Build Machine Host
Docker
etcd
Jenkins
Docker Registry
Config Editor nginx
php-fpm-nfs-0001
php-worker-a-1
Fleet
![Page 29: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/29.jpg)
Docker enhanced architecture on SoftLayerCentOS 7 host
Discovery/Config
Fleet managed Docker services
nginx-confd.service
NGINX-confd container
Orchestration Docker
eea/php-fpm:latest
1
eea/php-fpm:latest
2php-fpm-discovery
@2.service
ea/php-worker:latest
2
ea/php-worker:latest
1
ea/php-worker:latest
1php-worker-A-discovery
@1.service
Redi
sEx
tern
al d
ata
serv
ices
MyS
QLM
emca
ched
Load
Bal
ance
r
![Page 30: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/30.jpg)
Configuration, orchestration and deployment
host
service
service
service
service
host
service
service
service
service
host
service
service
service
service
host
service
service
service
service
Configure topology, maintain state.
Build images with code, deploy to or scale target environments.
Run containers in staging, load test, and production. Capture
metrics.
![Page 31: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/31.jpg)
• Centralised Logging
• Monitoring - processes inside containers
Future Work
![Page 32: DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker](https://reader035.fdocuments.net/reader035/viewer/2022081604/58a6afaa1a28ab0a7a8b6c73/html5/thumbnails/32.jpg)
— Scott Porter
“Using IBM and Docker container solutions, we have been able to build a gaming platform that is massively scalable, highly performant and meets our deployment velocity needs.”
33