Deploying a Low-Latency Multiplayer Game Globally: Loadout

1

GDC 2015 – Amazon Developer Day

Deploying a Low-Latency Multiplayer Game Globally

Nate WigerPrincipal, Gaming Solutions

@nateware

[email protected]

James GwertzmanCEO, PlayFab

@playfabnetwork

http://playfab.com

http://playfab.com

AWS Gaming Solutions

[Game] Online Problems Continue; Features

Disabled as [Game Studio] Works on Fixes

“We are working hard to get everyone online

as soon as possible,” [Game Studio] says.

Days later, [Game Studio] leaving

disconnected players stranded

[Game] has a great online mode – if you

can get online, that is

“This is total bull**** I paid for a preorder and

I can’t even connect WTF [Game Studio]??”

AWS Gaming Solutions http://bit.ly/verizon-latencyhttp://bit.ly/superdata-latency

http://bit.ly/verizon-latency

http://bit.ly/superdata-latency


100+ms 100+ms

100+ms100+ms


How To Be Awesome

1. Game servers near players

2. Everything else as HTTP APIs

3. Data replication = bad

4. Local caches = good

5. Nobody really “plays everywhere”


Loosely Coupled Pods

Tokyo

Oregon

Frankfurt

Virginia


VPC Subnet

VPC Subnet

Game API Pods

Availability Zone A Availability Zone B

VPC Subnet

VPC Subnet

Auto Scaling group

WEB

VPC Subnet

WEB

JOBS

Cognito

SNS Mobile

Push

SES


Region

① Login via HTTP API

② Download Game Assets

③ Matchmaking to Game Server

EC2

Game Flow

EC2

EC2


Region




④ Connect to Server

⑤ Hack Apart Your Friends

⑥ Game Over

Game Flow

EC2

EC2


Region




④ Connect to Server

⑤ Hack Apart Your Friends

⑥ Game Over

⑦ Write via HTTP API

Game Flow

EC2

EC2


VPC Private Subnet

VPC Public Subnet

Game Server Pods


VPC Public Subnet

VPC Private Subnet

GAME GAME GAME GAME GAME GAME


Make. Network. Faster.

• C3/C4 instance family (or R3 for more memory)

• In a VPC

• HVM AMI – not PV

• Enable Enhanced Networking (SR-

IOV)http://bit.ly/ec2-enhanced-net

• Linux: Tweak Networking Stack http://bit.ly/linux-

tuning

http://bit.ly/ec2-enhanced-net

http://bit.ly/linux-tuning


Enhanced Networking (SR-IOV)

Before:

Hypervisor

After:

Hardware


VPC Private Subnet VPC Private Subnet

RabbitMQ + Elastic Load Balancing


10.1.0.13 10.2.0.16

TCP 5672

rabbitmq-node1 rabbitmq-node2

TCP 4369

& 25672


RabbitMQ + Elastic Load Balancing

• Clustering Tutorial https://www.rabbitmq.com/clustering.html

• Set Queue HA policy to "all" https://www.rabbitmq.com/ha.html

• Create Internal load balancer

– Listen Port: TCP 5672

• VPC Security Group

– For load balancer: TCP 5672

– Between EC2 nodes: TCP 4369 & 25672

• Set up Client to Heartbeat

https://www.rabbitmq.com/clustering.html

https://www.rabbitmq.com/ha.html


/etc/rabbitmq/rabbitmq.config

5672

rabbitmq-node1

rabbitmq-node2


VPC Private Subnet

VPC Public Subnet

Redis Pub / Sub


VPC Public Subnet

VPC Private Subnet

GAME GAME GAME GAME GAME GAME

Auto Scaling group


VPC Private Subnet

VPC Public Subnet

CloudFormation + Chef

Availability Zone A

GAME GAME GAME

Auto Scaling group


Unreal Engine with CloudFormation

UnrealServer

unreal_linux.template.json

true

my-bucket-name

Unreal-123.zip

sshkeyname

1.2.3.4/24

MyPass123


Where My Servers At?

Tokyo

Oregon

Frankfurt

Virginia

?

?


VPC Subnet

Server Registration


VPC Subnet

Auto Scaling group

WEB WEB

Oregon

Tokyo

VPC SubnetJOBS


Server Registration & Scaling

• HTTPS POST /api/servers/register

• Include an HMAC (RFC 2104)

• Send Server Status– Public IP

– # Players

– Game Modes

• Matchmaking Service– Maintains server list

– Removes servers


VPC Private Subnet

VPC Public Subnet

Game Server Logs

Availability Zone A

S3 Bucket

Amazon CloudWatch Logs

GAME GAME GAME


VPC Private Subnet

VPC Public Subnet

Game Server Logs

Availability Zone A

Amazon

Kinesis

S3 Bucket

Amazon Redshift

GAME GAME GAME


Mini Pods Everywhere

China

Oregon

Frankfurt

VirginiaChina

Leaderboard


PlayFab: Loadout Architecture


What is PlayFab?

PlayFab is a complete live game operations platform.

Focus on the fun – we’ll do the boring bits.

+

Tools for operating &

monetizing live games

Complete set of cloud-based

backend services

Eco-system of pre-integrated

third-party services

.. and

more

+


Some Loadout Stats

• Launched 1/31/14

• Over 1 billion API Calls

• At Peak (Feb 2014):

– 300k DAU

– 120 fixed game servers

– 110 EC2 game servers


PlayFab Core Architecture Tenants

• No scheduled downtime.

• Automatic scaling for load.

• Backend changes should be

invisible to developers.

• Use buffering to smooth over

spikes or failures.


AWS cloud: PlayFab Web Services

US EU

Hardware

Game Servers

Hardware

Game Servers

Game

Client

Amazon Route 53

(3.playfabapi.com)

Amazon EC2

(API handling)Matchmaker

Instance Instance

Game Server

Monitor

DynamoDB Amazon RDS Amazon S3

Elastic Load Balancing

Amazon RedshiftReports

service

Instance

LogsA

rch

itec

ture

Overv

iew

Matchmaker

(Secondary)

InstanceUS EU

SA AUS

Amazon EC2

Game Servers

Amazon EC2

Game Servers

Amazon EC2

Game Servers

Amazon EC2

Game Servers


List of PlayFab Web APIsClient Server

Authentication Title-Wide Data Management Shared Group Data Authentication

AddUsernamePassword GetCatalogItems AddSharedGroupMembers AuthenticateSessionTicket

LoginWithAndroidDeviceID GetStoreItems CreateSharedGroup Account Management

LoginWithFacebook GetTitleData GetSharedGroupData GetUserAccountInfo

LoginWithGameCenter GetTitleNews RemoveSharedGroupMembers SendPushNotification

LoginWithGoogleAccount AddUserVirtualCurrency UpdateSharedGroupData Player Data Management

LoginWithIOSDeviceID Player Item Management Server-Side Game Logic GetLeaderboard

LoginWithPlayFab ConsumeItem GetLogicServerUrl GetLeaderboardAroundUser

LoginWithSteam GetUserInventory GetUserData

RegisterPlayFabUser RedeemCoupon GetUserInternalData

SendAccountRecoveryEmail SubtractUserVirtualCurrency GetUserReadOnlyData

Account Management UnlockContainerItem GetUserStatistics

GetAccountInfo StartPurchase UpdateUserData

GetPlayFabIDsFromFacebookIDs PayForPurchase UpdateUserInternalData

GetUserCombinedInfo ConfirmPurchase UpdateUserReadOnlyData

LinkFacebookAccount PurchaseItem UpdateUserStatistics

LinkGameCenterAccount Friend List Management Title-Wide Data Management

LinkSteamAccount AddFriend GetCatalogItems

UnlinkFacebookAccount GetFriendsList GetTitleData

UnlinkGameCenterAccount RemoveFriend SetTitleData

UnlinkSteamAccount SetFriendTags Player Item Management

UpdateEmailAddress IOS-Specific APIs AddUserVirtualCurrency

UpdatePassword RegisterForIOSPushNotification GetUserInventory

UpdateUserTitleDisplayName ValidateIOSReceipt GrantItemsToUsers

Player Data Management Matchmaking APIs SubtractUserVirtualCurrency

GetFriendLeaderboard GetCurrentGames Matchmaking APIs

GetLeaderboard GetGameServerRegions NotifyMatchmakerPlayerLeft

GetLeaderboardAroundCurrentUser Matchmake RedeemMatchmakerTicket

GetUserData StartGame Steam-Specific APIs

GetUserReadOnlyData Android-Specific APIs AwardSteamAchievement

GetUserStatistics AndroidDevicePushNotificationRegistration

UpdateUserData ValidateGooglePlayPurchase

UpdateUserStatistics Analytics

LogEvent


Example of Web API Call

API Request: Response from Web Services:


ELB Logging for Title Tracking

Game ClientAmazon Route 53

(3.playfabapi.com)

API Request:

Elastic Load

Balancing

Logs

Amazon RedshiftAmazon S3

bucket

Reports

service

LogsInstance

Logs include title-specific

endpoint for analytics

AWS wildcard DNS

(*.playfabapi.com)

Log event:


Loadout API Usage Report (Oct 2014)

API NAMETOTAL

CALLS

AVERAGE

BACKEND

TIME (S)

AVERAGE

RESPONSE

TIME (MS)

AVERAGE

REQUEST

TIME (MS)

TOTAL

REQUEST

TIME (M)

AVERAGE

RECIEVED

(BYTES)

AVERAGE

SENT

(BYTES)

TOTAL

ERRORS

adminapi_userinfo 93,081,685 0.03 0.04 0.04 59.83 0 452 1979

gameserver_usercustomdatareadonly 1,582,348 0.06 0.04 0.04 1.06 42 902 290

matchmaker_userinfo 1,138,842 0.20 0.04 0.04 0.81 0 10046 319

matchmaker_playerjoined 1,138,247 0.01 0.04 0.04 0.79 64 2 93

matchmaker_playerleft 1,136,797 0.01 0.04 0.04 0.80 64 2 400

matchmaker_authuser 736,607 0.01 0.03 0.04 0.52 0 48 107

gameserver_usercustomdata 297,735 0.19 0.04 0.04 0.20 0 4080 0

gameserver_usercustomdatainternal 297,469 0.08 0.04 0.04 0.20 0 237 2


Instance

Making a Singleton Highly Available

Health

Check

Auto Scaling group

Auto Scaling

service

Instance

Matchmaker

(Primary)

Instance

Amazon Route 53)

Elastic Load

Balancing

Matchmaker

(Secondary)

1.Auto Scaling is set to keep

two instances up.

2.Elastic Beanstalk Health

Check terminates bad

instance.

3.Amazon Route 53 fails

over all traffic to secondary

server.

4.Auto Scaling detects

failure and starts new

instance.


Monitoring Service Health

Amazon EC2

(API handling)

Elastic Load

Balancing CloudWatch

Health

Check

Email AlertsOn-call Engineer

Mobile Alerts


Handling Spikes Without Losing Data

Amazon EC2 API Server

Server SSD

Amazon Redshift

Local Reports

service Local event

logsInstance

Events:

logins and

purchasesReports

service

Logs

Customer’s game

management dashboard

Analytics

Server

Logs

RabbitMQ

message

queue

LogsInstance

Logstash

service

Logstash

Instance

ElasticSearch

Instance

Instance


Storage in DynamoDB, Amazon RDS, Amazon S3,

Amazon Redshift


Player data:

• Accounts

• Profile

• Inventory

• Save data

• Player stats

• Player currency

values

• Auditing

• Log files

• Game replays

Amazon Redshift

• Analytics data


Managing Tables in DynamoDB

Amazon

DynamoDB

• Player data: accounts, profile, inventory, save data

User:

UserID: Hash

TitleID: range key

- DisplayName

- Linked accounts

- …

Custom user data:

UserID: Hash

TitleID: range key

- Key/value 1

- Key/value 2

- …

Inventory:

UserID: Hash

ItemID: range key

- titleID

- Catalog Item

- …

…


Physical Deployment

AZ 1

AZ 2

PlayFab Web

Services

PlayFab Web

Services

N. California

AZ 1

AZ 2

PlayFab Web

Services

PlayFab Web

Services

Oregon

AZ 1

PlayFab Web

Services

Beijing

• Original deployment: us-west-1

• Expanded to us-west-2 in July

• Coming soon: cn-north-1

Amazon EC2

Game Server

Singapore

Game Client

Instance


Managing Cross-Region Data

Player accounts are shared

across all regions:• Cross-regional wrapper for

user.get sets order of

databases and regions to

check

• user.get then checks for

player account data

• High availability, transparent

to customers and players

// Returns the first matching player record, searching DynamoDB and SimpleDB

// in both regions in the specified order.

// Updates the player record in the store and region from which it was loaded.

// Creates a player record in the default store and region.


AWS cloud: PlayFab Web Services

US EU

Hardware

Game Servers

Hardware

Game ServersGame

Client

Amazon Route 53

(3.playfabapi.com)

Amazon EC2

(API handling)Matchmaker

Instance Instance

Game Server

Monitor


Elastic Load Balancing

Amazon RedshiftReports

service

Instance

LogsA

rch

itec

ture

Overv

iew

Matchmaker

(Secondary)

InstanceUS EU

SA AUS

Amazon EC2

Game Servers

Amazon EC2

Game Servers

Amazon EC2

Game Servers

Amazon EC2

Game Servers

52

Nate WigerPrincipal, Gaming Solutions

@nateware

[email protected]

Deploying a Low-Latency Multiplayer Game Globally: Loadout

Technology

Transcript of Deploying a Low-Latency Multiplayer Game Globally: Loadout