AWS SDKDevOps AutomationSachin Dole

About me

▪ AWS Certified Solutions Architect

▪ Independent AWS Consultant – Genvega Inc.

▪ Trelair.com – now defunct startup

▪ www.cricclubs.com – ½ million+ page views per month

▪ Backup & Archival on S3 – project for law firm

▪ EC2 automation for image processing

▪ Available for AWS consulting – find me: www.genvega.com

About CricClubs (My customer)

30th Jan

2016@2015 CRICCLUBS.COM | www.cricclubs,.com | CONFIDENTIAL 3

Started in 2008

Geographic Usage20+ Countries

600+ Leagues 100,000+ Players

HeadquartersMilwaukee, WI,

USA

Features50+ features on

Website and iOS, Android, Windows

Mobile Apps

Premium Support24/7 via E-mail, Phone andTicketing Systems

Data CentersHigh Availability with servers across 12 Data Centers

4 million pageviews/month

About you?

▪ Have day to day AWS experience?

▪ Have personal projects on AWS?

▪ Want a good introduction to DevOps & AWS?

Console access

▪ https://tvarit.signin.aws.amazon.com/console

▪ ijugdemo

▪ devops123

▪ Read only access to all services

Topics

▪ DevOps for Trelair

▪ Lessons learnt

▪ Tvarit.io & AWS Java SDK

▪ Questions?

Trelair.com

System Architecture

Clients

ELB

Wildfly on EC2

S3 & Postgres

External

Services

Building Trelair

Commit

Build

FTP

Deploy

Test

Backup & Release

1 to 8 person-hrs

DML Disk Compute DB App server Backups

Commit

Build

FTP

Deploy

Test

Release

Enterprises?

1 to 8 person years

DML Disk Compute DB App server Backups

Dev

EA

Test Ops

EA

Trelair Automation

GitCommit• Teamcity

• JBossDMR

Auto-Deploy

SeleniumTest &

Release

DevOps + 0 person hoursCustom Maven Plugin

Used AWS Java SDK

AMI DML/DDL DNS Snapshots ScalingAuto

Automated Flow

A. Commit Code (the only manual step)

B. CI build

C. DB DML patches are applied to replica

D. Deploy code to candidate server

E. UI tests run

F. DML patches applied to production

G. Candidate server is added to cluster

H. Old servers are terminated

AWS API

us - eastAWS Java

SDK

JBoss DMR Wildfly AS

Provision

Deploy

Lessons

▪ Automation is easy – mindset is difficult

▪ Better to automate in small increments – big bang is a waste

▪ Automate everything – partial automation is overhead

▪ Release frequently – rely on automation

▪ Use selenium automation (e.g. saucelabs.com)

▪ Rely on AWS products

Tvarit

▪ http://www.tvarit.io, https://github.com/sdole/tvarit-maven

▪ Trelair automation on steroids, opensource

▪ Automated provisioning, deployment and scaling

▪ Alpha release available in Maven Central

▪ Free support through GitHub issues

▪ Working α version & a deep roadmap

Console access

▪ https://tvarit.signin.aws.amazon.com/console

▪ ijugdemo

▪ devops123

▪ Read only access to all services

Tvarit – demo

▪ CloudFormation templates

▪ Maven goals

▪ VPC, IAM console

▪ EC2: LC, ASG, ELB

▪ S3 bucket

▪ Cloudwatch monitoring, flow logs

▪ Code walkthrough

API structure

▪ Many flavors – Java, C++, JS, Ruby, Python, CLI etc.

▪ APIs are RESTful

▪ Deep documentation

API Structure

AmazonEc2Client ec2 =

new AmazonEc2Client(new BasicAwsCredentials());

SomeEc2Request req = new SomeEc2Request();

req.withSomething(thing1).withSomething(attr1);

SomeEc2Result result = ec2.submitRequest(req);

inspectWithYourLogic(result);

Automation Security - Individuals

IAMus - east

us - west

Access

Keys

groups

users

roles

Authorized

Automation Security - Services

IAMOther

services

Other

services

ec2

lambda

S3

Assumed

RoleAuthorized

AWS Automation Features

▪ Autoscaling

▪ Monitoring

▪ Alarms

▪ Cloudwatch Events

▪ Lambda Functions

▪ S3 Events, SQS and SNS events

▪ SWF

▪ CloudFormation

Tvarit Target Architecture

Clients

WAF

ELB

EC2

ELB

EC2

Lambda

Java SDKAutoscaling

Cloudwatch

Zookeeper

S3, RDS

S3, RDS

Cloudformation

ELB

Router

App 1

Version 1

App 2

Version 1

Other concepts:Blue-Green Deployment

http://martinfowler.com/bliki/BlueGreenDeployment.html

Other concepts:Canary Release

http://martinfowler.com/bliki/CanaryRelease.html

Revisit Tvarit Target Architecture

Clients

WAF

ELB

EC2

ELB

EC2

Lambda

Java SDKAutoscaling

Cloudwatch

Zookeeper

S3, RDS

S3, RDS

Cloudformation

ELB

Router

App 1

Version 1

App 2

Version 1

Tvarit Release concept (future)

ELB

ELB

Router

App Versions

Router: routes customers based on Http Headers to correct app version

Step Back – what is (not) DevOps?

Is…

Infrastructure as code

Not…

Auto-scale, failover, monitoring

Continuous delivery

Dedicated teams, Dev, Test, Ops

Forms, Emails, Release plans

Infrastructure Architects/EA

Soft side of DevOps

▪ Cloud-native vs Lift ‘n shift

▪ CALMS

▪ C – Culture

▪ A – Automation

▪ L – Lean

▪ M – Measurement

▪ S – Sharing

▪ Automation as Kaizen – not a single project

Costs of AWS Automation

▪ In most cases – FREE

▪ Priced as usual: Compute, Storage, Monitoring, Network

▪ Monitoring is a must

▪ CPU, Memory, Network, Disk…

▪ Health checks, failed hardware, traffic spikes

▪ Audit trail – API actions, modifications

▪ Configuration changes

How to improve Tvarit

▪ Open an Issue on GitHub:

“We use a customized version of tomcat – can you make the app server container configurable so we can use our own?”

Fork it on GitHub and DIY

Email me

Other things

▪ Chef, Puppet, Ansible, SaltStack

▪ AWS OpsWorks, Beanstalk

▪ AWS CodeCommit, CodeDeploy

How to find me

@sdoledotcom

https://linkedin.com/in/sdole

sdole@genvega.com

http://genvega.com/blog/

https://github.com/sdole/

ΩThank you