Rugged DevOps at Scale with Rich Mogull
30
Rich Mogull @rmogull Rugged DevOps at Scale
-
Upload
seniorstoryteller -
Category
Technology
-
view
823 -
download
2
Transcript of Rugged DevOps at Scale with Rich Mogull
Rich Mogull@rmogull
Rugged DevOps at Scale
From this…
Account
Virtual Network
Subnet
Security Group
Virtual Network
Subnet
Security Group
To this…
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
GroupAccount
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
Account
Virtual Network
Subnet
Security
Group
Virtual Network
Subnet
Security
Group
And this…
To this…
Scale• An average Rugged DevOps
project uses at least 3-5 cloud accounts and a dedicated deployment pipeline.
• The average enterprise has one custom application per 100 employees.
• Enterprises quickly scale from 1-2 test projects to dozens, hundreds, and even thousands of accounts.
http://thebrickboyz.blogspot.com/2012_05_01_archive.html
Ruggedize At Scale
• Centralize policies, patterns, and templates.
• Harden distributed pipelines.
• Automate the crap out of everything.
https://www.pinterest.com/pin/530510031078540624/
Source Code
GitCloudformation Templates
Jenkins
Functional Tests
Chef Recipes
Chef Server
NonFunctional Tests
Security Tests
Test Prod
Source Code
GitCloudformation Templates
Jenkins
Functional Tests
Chef Recipes
Chef Server
NonFunctional Tests
Security Tests
Test Prod
Build a security repo(including architectures)
Source Code
GitCloudformation Templates
Jenkins
Functional Tests
Chef Recipes
Chef Server
NonFunctional Tests
Security Tests
Test Prod
Provide scalable security testing
Source Code
GitCloudformation Templates
Jenkins
Functional Tests
Chef Recipes
Chef Server
NonFunctional Tests
Security Tests
Test Prod
Automate security operations
The Security Repo• Policies– Detailed, written in Markdown.
Revisable.• Design patterns– Specific, technical, flavored for
platforms.• Architectures– Diagrams.– Code templates for different
platforms.• Security tests
Quick Tips: Discovering Accounts
• Require consolidated/centralized billing for all accounts.
• Have accounting review credit card statements to find direct spends.
• Automate collecting, scanning, and reviewing centralized invoices.– Categorize by account identifier, and correlate to
your registry of known accounts.
Pattern 1: IAM• Separate
entitlement matrix per-project and account.
• Map roles to rights based on matrix, don’t just default.
• Brokers can be very useful.
• Keep code identities in the cloud.
Pattern 2: Rugged Hybrid
Pattern 3: Data Transfer
More Data Transfer
Pattern 4 (or 5, whatever): Logging
Automate
• Build a platform.• Create new account “deployment packages”• Automate controls, don’t just monitor and
respond.
Platform Architecture
Demo
New Account Deployment Packages
Demo• Remotely access a “new” pristine account.
– Does require a 1-time manual “hook” to allow the platform to connect.
• Fully configure monitoring (CloudTrail) with automated alerting for all IAM changes.
• Involves configuring:– CloudTrail– CloudWatch Logs– CloudWatch Alert– Simple Notification Service– IAM– S3
``
Account 123
SecServer
Role: Sec
• Configure IAM Roles• Create S3 bucket• Set bucket permissions• Create CloudWatch Log Group• Enable CloudTrail• Configure
CloudTrail/CloudWatch Log connection
• Create CloudWatch Alarm• Create SNS notification topic
Account 456
26
Demo: Self-Healing Infrastructure
Change a security group
Event Recorded to CloudTrail Passed to CloudWatch Log Stream
Triggers an CloudWatch Event
Lambda Function analyzes and reverses
Things I didn’t cover
• Architectures for a *lot* of security controls.• Cloud provider differences.– Because they are hella-big.
• Integrating existing security tools.– At least the ones that are still useful.
• Setting up different accounts/virtual networks for different security tool stacks.
Rich Mogull@rmogull
Rugged DevOps at Scale
Code at https://github.com/securosis
#RuggedDevOps
If you see something cool…
Thank You to Our Sponsors
