DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K....

48
DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K.

Transcript of DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K....

Page 1: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

DevOps in action

Yuta Imai

Solutions Architect, Amazon Data Services Japan, K.K.

Page 2: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

今日のテーマ

• いままで見てきたお客様や海外事例のなかで「これはおもしろいなー、すごいなー」と感じたDevOpsっぽい話を紹介していきます。

Page 3: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Agenda: Case Studies!

• NetFlix 1/2 : Service Oriented Architecture and

Origanization

• A Global Gaming : Application integrated deploy

• MMO RPG : Application integrated deploy

• NetFlix 2/2 : Monitor everything

Page 4: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Case study: Netflix 1/2Service Oriented Architecture and

Origanization

Page 5: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

NetflixMeetup in Kyoto

http://connpass.com/event/9837/

Page 6: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Netflix’s way

• Service Oriented Architecture

• Appropriate delegation

• No dependency among teams

http://techblog.netflix.com/2015/02/a-microscope-on-microservices.html

Page 7: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Components in each Services

Service

Manager

DAO

Data

Page 8: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Build

Source .war .deb .ami

SBT Gradle Aminator

Page 9: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Deploy

1)

2)

3)

dev

mastertest

deploybuild

Page 10: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Deploy with ASGARD

Page 11: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Deploy with ASGARD

Page 12: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

More Netflix OSS..

http://netflix.github.io/

Page 13: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

You can try with ready made docker containers.

http://techblog.netflix.com/2014/11/zerotodocker-easy-way-to-evaluate.html

Page 14: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Case study: A Global GamingApplication integrated deploy

Page 15: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

http://bit.ly/verizon-latencyhttp://bit.ly/superdata-latency

Page 16: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

VPC Subnet

VPC Subnet

Game API Stack

Availability Zone A Availability Zone B

VPC Subnet

VPC Subnet

Auto Scaling group

Amazon

CloudFront

WEB

VPC Subnet

WEB

JOBS

Queue

Cognito

SNS Mobile

Push

SES

S3: DLC,

Assets

Page 17: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ
Page 18: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

100+ms 100+ms

100+ms100+ms

Page 19: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Loosely Coupled Pods

Tokyo

Oregon

Frankfurt

Virginia

Page 20: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Region

① Login via HTTP API

② Download Game Assets

③ Matchmaking to Game Server

EC2

Game Flow

EC2

EC2

Page 21: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Region

① Login via HTTP API

② Download Game Assets

③ Matchmaking to Game Server

④ Connect to Server

⑤ Hack Apart Your Friends

⑥ Game Over

Game Flow

EC2

EC2

Page 22: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Region

① Login via HTTP API

② Download Game Assets

③ Matchmaking to Game Server

④ Connect to Server

⑤ Hack Apart Your Friends

⑥ Game Over

⑦ Write via HTTP API

Game Flow

EC2

EC2

Page 23: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

VPC Private Subnet

VPC Public Subnet

Game Server Pods

Availability Zone A Availability Zone B

VPC Public Subnet

VPC Private Subnet

GAME GAME GAME GAME GAME GAME

Page 24: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

VPC Private Subnet VPC Private Subnet

RabbitMQ + Elastic Load Balancing

Availability Zone A Availability Zone B

10.1.0.13 10.2.0.16

TCP 5672

rabbitmq-node1 rabbitmq-node2

TCP 4369

& 25672

Page 25: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

VPC Private Subnet

VPC Public Subnet

CloudFormation + Chef

Availability Zone A

GAME GAME GAME

Auto Scaling group

AWS CloudFormation:

Launch, Bootstrap

Chef: Configure, Install

Page 26: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Where Are Servers?

Tokyo

Oregon

Frankfurt

Virginia

?

?

Page 27: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

VPC Subnet

Server Registration

Availability Zone A Availability Zone B

VPC Subnet

Auto Scaling group

WEB WEB

Oregon

Tokyo

VPC Subnet

Cleanup

EC2 API

Start / Stop

Instances

JOBS

Page 28: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Global Services

China

Oregon

Frankfurt

LoginChina

Leaderboard

Page 29: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Case study: MMO RPGApplication integrated deploy

Page 30: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

World

Island A Island B

Usual way of MMO implementation: Single big server

Server A

InteractionInteraction

Page 31: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

World

Island A Island B

Distribute users by geo location in game map

Server A Server B

no interaction

Page 32: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

World

Island A Island B

Distribute hot area with more game servers

no sync

Low density

AreaHigh density

Area

Page 33: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

User density differentiated by map’s popularity

Area Aless monsters

low experience points

poor item drop rate

Area Bmedium experience points

enough gold drops

avg. item drop rate

Area Cboss monsters

high chance of get items

Page 34: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

When an area’s population get higher

Area Cboss monsters

high chance of get items

Server C-1

- newly created

Server C-2

- newly created

Server C

- already existed

Geo size

3

Page 35: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Case study: Netflix 2/2Monitor everything

Page 36: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Monitor everything

• Let’s monitor everything– User behavior

– Application log

– Access log

• Then make it available!

Page 37: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

SPS : the Pulse of Netflix Streaming

• Starts Per Second

• They use SPS to

detect issues

• Deviation modeling

is important.

Static

Exponential Smoothing

Double Exponential Smoothing

http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-streaming.html

Page 38: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

A/B Testing

A

B

Page 39: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

A/B Testing

A

B

A

B

Page 40: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

A/B Testing

A

B

A

B

A

B

Page 41: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

A/B Testing

A

B

C

D

E

F

G

Test1

A

B

C

D

E

F

G

Test2

A

B

C

D

E

F

G

Test3

A

B

C

D

E

F

G

Test4

A

B

C

D

E

F

G

Test5

A

B

C

D

E

F

G

Test6

A

B

C

D

E

F

G

Test7

Page 42: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Huge numbers in A/B testing

• 2,097,152 unique experience across 7 tests

• Hundreds of new A/B tests per year

• 2,105 Templates, 566 CSS, 685 JS

• 2.5 million unique css/js packages per push cycle

This cannot be handled manually!

Page 43: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

A/B tests should be automatically convergent

A

B

C

D

E

F

G

Test1

A

B

C

D

E

F

G

Test2

A

B

C

D

E

F

G

Test3

A

B

C

D

E

F

G

Test4

A

B

C

D

E

F

G

Test5

A

B

C

D

E

F

G

Test6

A

B

C

D

E

F

G

Test7

Page 44: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

• Monitor metrics

• Then– Deploy multiple versions of application

– Configuration

A/B tests should be automatically convergent: How?

Page 45: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Atlas: Telemetry Platform

http://techblog.netflix.com/2014/12/introducing-atlas-netflixs-primary.html

Page 46: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Let’s build data stream/pipeline

App1

App2

App3

App5

App4

Expose API to users for:

• Adding new data source

• Adding new application

Page 47: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Conclusion

Page 48: DevOps in action...DevOps in action Yuta Imai Solutions Architect, Amazon Data Services Japan, K.K. 今日のテーマ •いままで見てきたお客様や海外事例のなかで「これ

Conclusion

• Infra team should become ‘Tool team’

• Many things to go other than Test, Build, Deploy

• DevOps affect software architecture