© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Cloud Based Game Development

David Pippenger, Director DevOps, GREE

NET308

How GREE Launched New Games Faster and

More Securely with AWS Marketplace and

Amazon VPC

Company Overview

GREE International

Entertainment, Inc.

San Francisco, CA

~200 People, 2011

GREE Inc.

Tokyo, Japan

~2000 People, 2004

Started using AWS 2012

Evolved toward

immutable infrastructure

to run games

Now continuing to

evolve toward

decentralized model

Business Model: Publishing Engine

Buy a studio,

operate it as a

programming

division

First Party

Commission a

game to GREE’s

specifications, pay

development costs

Second Party

Buy an existing

game

Third Party

Game Publishing Engine

Marketing Analytics LiveOps

in-game

sales events

TechOps

AWS Infrastructure

Games have a lifecycle. Nothing lasts forever.

New

Game

Creation

Long tail games with loyal users

BUY GAME SELL GAME

TOP 10

REVENUE

GAMES

Gree started in

colos until the

Funzio acquistion

in 2012

Funzio was a

small mobile

gaming company

with 3 successful

games

Dedicated jobs and cron hosts

Deploying tarballs

Hadoop

EC2 Based Relays

EC2 Based Vertica

Ma

nagem

ent

Amazon EC2 memcache clusters

EC2 MySQL

EC2 Zeus load balancers

Provisioning hosts using Runbooks

Amazon ElastiCache

Amazon RDS

Elastic Load Balancing

Puppet

Jenkins for code deployments

Deploying packages (RPM, Docker)

Amazon Elastic Map Reduce (EMR)

Amazon Kinesis

Amazon Redshift

AW

S

SSH Jump hosts and OpenSwan

Netw

ork

ing

ELK logging on EC2

Nagios on EC2

SumoLogic

DataDogSaaS

Technical

Stack

EC2 AutoscalingCloudformation S3 ELB

Kinesis RDS Elasticache DynamoDB

1 Leverage SaaS

2 Avoid building things you can buy

3 Infrastructure as code

1 Leverage AWS services

2 Automate everything

3 Immutable infrastucture

4 Servers are cattle not pets

DevOps

Principles

GAME STACK

DevOps at GREE – Architecture High Level Design

Amazon

S3

Amazon

Route 53

AWS

Directory

Service

AWS

Code

Commit

Active

Directory

GitHub

Enterprise

AWS

VPC

DevOps at GREE – Architecture Low Level Design

Service 1 Service 2Amazon

DynamoDB

ILB

ELB

GAME

Puppet

Jenkins

Aviatrix

GatewayAuto Scaling

Auto Scaling Auto Scaling

VPC

Secure Remote User Access

GREE AWS Account

Region 1 Region 2

IPSEC

Aviatrix

GatewayAviatrix

Gateway

AWSAWS

Dragon Soul AWS Account

VPC VPC

Remote Users

(dev, devops, admins)

SSL VPN

Patterns of Enterprise Application Architecture

inspired by Martin Fowler’s Books

Codebase

• One codebase tracked in

revision control, many deploys

• Use Git

I

Dependencies

• Explicitly declare and isolate

dependencies

• Use Puppet

II

Config

• Store config in the environment

• Use Puppet with HieradataIII

Backing Services

• Treat backing services as

attached resources

• Use Puppet with Hieradata to

capture endpoints

IV

Build,

release, run

• Strictly separate build and run

stages

• Use Jenkins to produce

package artifacts

• Use Jenkins pipeline to

orchestrate deployment

V

Processes

• Execute the app as one or

more stateless processes

• Use ELB

• STONITH

• https://en.wikipedia.org/wiki/ST

ONITH

VI

Port Binding

• Export services via port binding

• Use ILBVII

Concurrency

• Scale out via the process

model

• Use ELB

VIII

Disposability

• Maximize robustness with fast

startup and graceful shutdown

• Use Auto Scaling

• Use custom AMIs

• Use ElastiCache or DynamoDB

• STONITH

IX

Dev/Prod

Parity

• Keep development, staging

and production as

similar as possible

• Use AWS CloudFormation

• Use Puppet

• Use Docker

X

Logs

• Treat logs as event streams

• Use Sumo logicXI

Admin Processes

• Run admin/management tasks

as one-off processes

• Use Jenkins

XII

14 DevOps engineers

Takeaways

Hundreds of monitoring pages

6

10

VPC History at GREE

VPC I

• AWS Classic

• EU-west-1 VPC only

• 6 week deadline,

things were rushed

• Poor choices and

bad shortcuts

VPC II

• Second attempt at

VPC: hire an expert

• Company mandate to

move from AWS Classic

to VPC

• 6 months of design, lots

of over engineering

VPC III

• Initial setup is less

than an hour

• Clean and simple

model

Demo video: Bringing a cloud controller online

Aviatrix - New VPC with AGW

GREE purchased the

DragonSoul game from PerBlue

During the due diligence phase,

Aviatrix was used to link

GREE to the VPC

Use Case

W1 W2 W3 W4 W10

RDS

Aviatrix

Gateway

World

Game

Backend

Systems

Game

Admin

VPC

Payment ADS

Original Deployment

Dragon Soul Deployment 1

Region 1 Region 2

IPSEC

Cross Account / Region Peering

Aviatrix Gateway Aviatrix Gateway

AWSAWSDragon Soul Deployment 2

VPC VPC

W11Wn

RDS

11

ELB11Aviatrix

Gateway VPC ELB12 ELBn

W12

Auto Scaling

Group

Auto Scaling

Group

Auto Scaling

Group

RDS

12

RDS

n

Second Deployment

Challenges

Single points of failure

RDS db.r3.8xlarge at capacity

World servers are stateful and unwinding the Java ORM

will take time.

Goals

Grow the game fast: double the

number of players

Don’t rock the boat: maintain continuity in existing

game service

In Summary:

DevOps is not just technology, it’s people

using technology.

Docker is great

but it has some

sharp edges

Games have a

lifecycle,

nothing lasts

forever

CloudFormation

is great but it

is hard to get

started

Use dedicated

AWS sub-

accounts

Use Aviatrix

cloud

networking

People leave,

institutional

knowledge is lost,

so capture

everything in

automation

People take

shortcuts, magic

happens, so

enforce

automation

Thank you!