London Devops #9 - Security at a startup

0

Security at a startup

@neiljsaunders [email protected]

2

•  At a huge competitive advantage compared to enterprises •  No RFPs, Long winded procurement contracts •  Outsource commodity services to others:

–  Most obviously – Hosting – AWS –  Monitoring – Dataloop.io –  Alerting –  Email –  Source control and versioning –  Messaging

Startups

-  A security oversight in one system can often be used to compromise another

-  Are you aware when a new email address gets added? -  What’s this IP address in our AWS security group? When was it

added? Who added it? If I remove it what will stop working? -  Whose Github account is this? -  What’s this hook on my github repo? -  The list is endless, and is easily work for a dedicated team -  That team will never exist in a startup!-  I just want to focus on cutting our deploy time in half and

implementing Docker!

Security Jenga

14

One potential solution… -  A light weight process that actively audits security and alerts your

“Security team” when something doesn’t “smell” right -  Needs to be extensible to keep up with all those new third party

services being added -  Adding services needs to take up as little time as possible

15

Single (secure) point of truth -  Can be as simple as a text file on github -  We chose LDAP -  Deserving in its reputation as a PITA to setup: 6 months to learn, build, test, secure,

and populate (More on that later) -  It integrates with everything (More on that later too) -  Created a custom schema extension for a Beamly employees

-  Github ID -  Facebook User UID -  Public SSH Key

-  At first internal only -  Now externally accessible

16

Security “unit” tests •  Now you’ve got a version controlled state •  If you can programmatically extract a list of users from a

service, you can compare against that •  Applies to more than just user accounts:

–  AWS Security groups –  Github hooks

•  Store exceptions with explanatory comments •  Track changes with git history

17

Beamly Security Scripts •  Implemented as Python “unit” tests (py.unit) •  Short & quick to write •  Executed every 10 minutes via a scheduled CD pipeline •  Failures trigger an email to the platform team •  Email can trigger whatever else you want (Pagerduty) •  Platform team can login and take appropriate action

22

collected 1280 items test_all_github_hooks_are_known.py ........................................................................................................................................................................................................................................................................................................................................................................................... test_aws_iam_users_with_passwords_have_2fa.py ..................................................................................... test_aws_security_groups.py ................................................................................................................................................................................................... test_github_users_and_in_ldap_and_2fa.py ............................................................. test_jira_users_not_in_ldap.py ....................................................................................................................................................................................... test_o365_users.py ........................................................................................................... test_onelogin_users_are_in_ldap.py ........................................................................................... test_pagerduty_users_not_in_ldap.py ........................ test_slack_users_not_in_ldap_and_2fa.py ............................................................F.............................................................................................. =================================== FAILURES =================================== _________________________ test_unknown_slack_users[60] _________________________ realname = ’New Joiner', two_factor_enabled = False def two_factor_enabled(realname, two_factor_enabled): > assert two_factor_enabled E assert False test_slack_users_not_in_ldap_and_2fa.py:42: AssertionError =================== 1 failed, 1279 passed in 145.26 seconds ====================

27

Systems Integrated with LDAP -  VPN access to VPC -  Host access (Using SSH Public keys) -  Sudo access (Via group) -  Office appliances (NAS) -  ….any application that supports it – GO, Grafana (Soon)

Couldn’t have done this if we’d have used a text file

29

•  Herd all these 3rd party applications in to one place •  Password manager on steroids •  Centrally managed •  Every application used by Beamly added by Platform team •  Authenticate with LDAP account and 2FA •  Single sign on just to services that users have rights to (LDAP groups) •  Users never know the password for shared systems •  Can bulk rotate shared passwords for all systems

OneLogin

32

Now what?!

33

“Preppers”!

37

BE A PREPPER!

Part I: Stock your stores - Backups •  Taking backups isn’t good enough •  3 -2 -1 •  3 Copies of your data •  2 in different formats •  1 Offsite •  S3 mutli-region ISN’T OFF SITE •  S3 > (Another cloud provider || On site) •  March 31st: World backup day

Part II: Run through restoring from nothing •  In the event of the Armageddon, you need to fight to get back up quickly •  How long would if take to restore your AWS “Scaffolding” alone

–  VPC Configuration –  Security groups –  Routing tables –  Subnet configurations –  DNS

•  Ideally you’d have all this in Cloudformation…. •  ….We set all this stuff up before Cloudformation was available •  AWS Cloudformer

41

To sum up: our approach •  If a service supports 2FA, it’s mandatory.

–  Currently Slack, Mailchimp, Github, AWS, and Onelogin •  If a service has an API for exporting a security config: try and script an

audit for it •  All third party services in Onelogin •  Bulk rotate shared passwords regularly •  Prepare for Armageddon:!

–  Backup CD config & AWS setup –  All data tiers backed up 3-2-1 (One offsite)

42

Lessons Learned •  Retro-fitting security is hard •  Be prepared to break lots of things (temporarily) •  Bring everyone along for the ride

–  Explain why it’s important –  Everyone gets the access they need –  #security for interested parties

•  Turn on AWS Cloudtrail and Config everywhere

43

Questions?

@neiljsaunders [email protected]

LONDON

Drury House 34-43 Russell Street

London WC2B 5HA

NEW YORK CITY

84 Wooster Street Suite 703 New York NY 10012

SYDNEY

22-36 Mountain Street Suite 1.10

Ultimo NSW 2007

© Beamly 2014. All Rights Reserved.

London Devops #9 - Security at a startup

Technology

Transcript of London Devops #9 - Security at a startup