Secure Application Development in the Age of Continuous Delivery

Secure Application Development in the Age of Continuous Delivery

LinuxCon/ContainerCon Europe 2016 – Berlin Germany

#whoami – Tim Mackey

Current roles: Senior Technical Evangelist; Occasional coder

• Former XenServer Community Manager in Citrix Open Source Business Office

Cool things I’ve done

• Designed laser communication systems

• Early designer of retail self-checkout machines

• Embedded special relativity algorithms into industrial control system

Find me

• Twitter: @TimInTech ( https://twitter.com/TimInTech )

• SlideShare: slideshare.net/TimMackey

• LinkedIn: www.linkedin.com/in/mackeytim

https://twitter.com/TimInTech

http://www.linkedin.com/in/mackeytim

Standard Development Workflow

DEVELOP BUILD PACKAGE DEPLOY PRODUCTION

BUG TRACKING

TEST

AUTOMATION

Understanding the Attacker Model

Vulnerability Management Implies Data Breach Management

89% of data breaches had a

financial or espionage motive

Legal costs and forensics dominate

remediation expenses

Source: Verizon 2016 Data Breach Report

Attackers Decide What’s Valuable …

But security investment is often not aligned with actual risks

Anatomy of a New Attack

Potential Attack

Iterate

Test against platforms

Document

Don’t forget PR department!

Deploy

Exploiting a Vulnerability

CLOSED SOURCE COMMERCIAL CODE

• DEDICATED SECURITY RESEARCHERS

• ALERTING AND NOTIFICATION INFRASTRUCTURE

• REGULAR PATCH UPDATES

• DEDICATED SUPPORT TEAM WITH SLA

OPEN SOURCE CODE

• “COMMUNITY”-BASED CODE ANALYSIS

• MONITOR NEWSFEEDS YOURSELF

• NO STANDARD PATCHING MECHANISM

• ULTIMATELY, YOU ARE RESPONSIBLE

Who is Responsible for Code and Security?

Knowledge is Key. Can You Keep Up?

glibc

Bug

Reported

July 2015

Vuln: CVE-2015-7547: glibc getaddrinfo stack-based

buffer overflow


glibc

Vuln

Introduced

May 2008

glibc

Bug

Reported

July 2015

CVE-2015-

7547

CVE

Assigned

Feb 16-2016

Low Security RiskVuln: CVE-2015-7547: glibc getaddrinfo stack-based

buffer overflow


glibc

Vuln

Introduced

May 2008

CVE-2015-

7547

CVE

Assigned

Feb 16-2016

glibc

Bug

Reported

July 2015

National

Vulnerability

Database

Vuln

Published

Feb 18-2016

Moderate Security Risk


buffer overflow


glibc

Vuln

Introduced

National

Vulnerability

Database

Vuln

Published

You

Find It

May 2008

CVE-2015-

7547

CVE

Assigned

Feb 16-2016 Feb 18-2016

glibc

Bug

Reported

July 2015

Patches

Available

You

Fix It

Highest Security Risk

Moderate Security Risk


buffer overflow

Understanding Vulnerability Impact

0

500

1000

1500

2000

2500

3000

3500

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

Open Source Vulnerabilities Reported Per YearBDS-Enhanced Vulnerability Data nvd

Reference: Black Duck Software KnowledgeBase, NVD

Vulnerability Disclosures Trending Upward

0

500

1000

1500

2000

2500

3000

3500

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

Open Source Vulnerabilities Reported Per Yearnvd BDS-Enhanced Vulnerability Data

Reference: Black Duck Software KnowledgeBase, NVD

Vulnerability Disclosures Trending Upward

Vulnerabilities Enter Product and Become Deployed


BUG TRACKING

TEST

AUTOMATION

Workflow Integrations Matter

Understanding the Deployment

Environment

Primary goals

• Protect against BIOS and firmware attacks

• Protect cryptographic host state

• Ensure valid hypervisor kernel

• Validate launch of critical VMs

• Attest to hosts’ trust state

Implemented by

• Intel Haswell and newer

• Cryptographic hashes stored in TPM

Intel TXT – Trusted Execution Protection - Foundational

Intel EPT – Extended Page Tables

Page 0

…

Page

13553

Page

13554

…

…

Page 126

Page 127

…

Page

64589

Page

64590

Page

64591

Page 0

…

Page 217

…

Page 31289

…

…

Page 78924

…

Page 97586

…

0→64589

13553→127

13554→64591

App Memory OS Memory

TLB CR3

Virtual Machine

126→31289

127→0

64589→97586

64590→217

64591→78924

Host Memory

EPT

Hypervisor

Hypervisor Memory Introspection – Enabled by EPT

Implementation Overview

• Critical memory pages are

assigned permissions in EPT

• Exception handler defined in

hypervisor

• Shadow EPT defined with elevated

privs

Protects Against Attack Techniques

• Rootkit injection

• Buffer overflow

• API hooking

VM Kernel Memory Layout

…

Kernel Code (R/X)

Driver Code (R/X)

…

Driver Data (R/W)

Kernel Code (R/X)

Kernel Data (R/W)

…

126→31289 (R/X)

127→0 (R/X)

64589→97586 (R/W)

64590→217 (R/X)

64591→78924 (R/W)

EPT#1

126→31289 (+W)

127→0 (+W)

64589→97586 (+X)

64590→217 (+W)

64591→78924 (+X)

EPT#2 (Shadow)

Exception

Handler

Guest Guest Guest Guest Guest

Critical

Memory

Access

Critical

Memory

Access

Critical

Memory

Access

Critical

Memory

Access

Critical

Memory

Access

Networking StorageCompute

Simplified Hypervisor Introspection Architecture Diagram

Xen Project Hypervisor

Control

Domain

(dom0)

Security

Appliance

(domU)

Memory

Introspection

Engine

Direct Inspect

APIs

Virtual Switches as Local Edge Protection – Silent Block

Guest

VM

SSL access

Attack silently blocked

Virtual Switch Rules

Ingress:

HTTPS public

Egress:

Dynamic port to origin

MySQL internal

Private CIDR internal

Port 22 access

Virtual Switches as Local Edge Protection – Traffic Monitor

Guest

VM

SSL access

Attack blocked with traffic log


Ingress:

HTTPS public

Egress:


MySQL internal


Port 22 accessovs Controller

Log SSH Port 22 access

Create port mirror for attackerTraffic

Monitor


Ingress:

HTTPS public

Egress:


MySQL internal


Mirror:

Port 22 to Traffic Monitor

All attacker traffic to monitor

Guest

VM

Virtual Switches as Local Edge Protection – Quarantine

Guest

VM

SSL access

Attack quarantined with full log


Ingress:

HTTPS public

Egress:


MySQL internal


Port 22 accessovs Controller

Log SSH Port 22 access

Create port mirror for attacker

Quarantine VM for attacker use

Trigger replacement VM for farm

Traffic

Monitor


Ingress:

HTTPS attacker

Egress:


Mirror:

Port 22 to Traffic Monitor

All attacker traffic to monitor

Production Container Usage

Are Containers Production Ready?

Container Use Cases

Application containers

• Hold a single application

• Can follow micro-services, cloud native design pattern

• Starting point for most container usage

• Short lifespan, many per host

System containers

• Proxy for a VM

• Insulate against core operating system

• Perfect for legacy apps

• Long lifespan, few per host

MyS

QL

Tom

cat

ngin

x

Kernel

MySQL

Tomcat

nginx

Kernel

Securing the Container

Contents and Environment

Baseline to Limit the Scope of Compromise

• Enable Linux Security Modules

• SELinux

• --selinux-enabled on Docker engine, --security-opt=“label:profile”

• AppArmor

• -- security-opt=“apparmor:profile”

• Apply Linux kernel security profiles

• grsecurity, PaX and seccomp protections for ALSR and RBAC

• Adjust privileged kernel capabilities

• Reduce capabilities with --cap-drop

• Beware –cap-add and –privileged=false, and CAP_SYS_ADMIN

• Use a minimal Linux host OS

• Red Hat Enterprise Linux Atomic Host 7, CoreOS, RancherOS, Intel Clear Linux, Alpine, etc

• Reduce impact of noisy neighbors

• Use cgroups to set CPU shares and memory

Red Hat Enterprise Linux Atomic Host 7

• What is Atomic Host?

• Optimized RHEL7 variation designed for use with Docker

• Uses SELinux for safeguards

• Provides atomic upgrade and rollback capabilities via rpm-ostree

• Pre-installed with Docker and Kubernetes

• Atomic App and Atomic Nulecule

• Provides a model for multi-container application definition

• Supports Docker, Kubernetes, OpenShift and Mesos

• OpenShift artifacts run natively or via atomic provider

• Provides security compliance scan capabilities

Container Source Trust

Red Hat Atomic Host

Ato

mic

App

Ato

mic

App

Ato

mic

App

Red Hat Registry

MyS

QL

Redis

Jenkin

s

Docker Hub

Docker

Conta

iner

Docker

Conta

iner

Docker

Conta

iner

Docker

Conta

iner

Docker

Conta

iner

Third Party and Custom

Problem: Who to trust, and why?

• Trusted source?

• Unexpected image contents

• Locked application layer

versions (e.g. no yum update)

• Layer dependencies

(monolithic vs micro-services)

• Validated when?

OpenSCAP vs. Black Duck Hub

OpenSCAP

• Profile driven compliance policy engine

• Vendor vulnerability data is but one component of policy

• Integrated directly with Red Hat Atomic

• Usage: atomic scan --scanner openscap {container id}

Black Duck Hub integration with Red Hat Atomic

• Broad vulnerability data for most open source components

• Covers vulnerability, license compliance and operational risk

• Integrated with Red Hat Atomic

• Rich tooling integration for development teams

• Installed via: atomic install blackducksoftware/atomic

• Usage: atomic scan --scanner blackduck {container id}

A DONATION HAPPILY GIVEN

TO THE DEMO GODS

Gating Builds for Vulnerabilities

DEVELOP BUILD PACKAGE

BUG TRACKING

VULNERABILITY

ASSESMENT

Control

Domain

NetworkingCompute Storage

Hypervisor

Container

VM

Minimal OS

Understanding Scope of Compromise – Protect From the Inside

Co

nta

ine

r

Co

nta

ine

r

Co

nta

ine

r

Container

VM

Minimal OS

Co

nta

ine

r

Co

nta

ine

r

Co

nta

ine

r

Security

Serv

ice

Co

nta

ine

r

Understanding Scope of Compromise – Protect From the Inside

Container VM

Minimal OS

Co

nta

ine

r

Co

nta

ine

r

Un

pa

tch

ed

Co

nta

ine

r

Co

mp

rom

ise

d

Co

nta

ine

r

Co

mp

rom

ise

d

Co

nta

ine

r

Vulnerability Assessment Post Deployment


BUG TRACKING

TEST

AUTOMATION

VULNERABILITY

ASSESMENT

Risk Mitigation Shrinks Scope of Compromise

Open source license compliance

• Ensure project dependencies are understood

Use of vulnerable open source components

• Is component a fork or dependency?

• How is component linked?

Operational risk

• Can you differentiate between “stable” and “dead”?

• Is there a significant change set in your future?

• API versioning

• Security response process for project

Vulnerability Analysis Compliments SAST/DAST

Vulnerability Analysis

- Identifies vulnerable dependencies

- 3000+ disclosures in 2015

- Most vulnerabilities found by researchersAll possible security vulnerabilities

Static and Dynamic Analysis

- Discover common security patterns

- Challenged by nuanced bugs

- Focuses on your code; not upstream

7 of the top 10 Software Companies

(44 of the top 100)

6 of the top 8Mobile Handset Vendors

6 of the top 10 Investment Banks

24Countries

250+Employees

1,800Customers

Who is Black Duck Software?

27Founded

2002

8,500WEBSITES

350BILLION LINES OF CODE

2,400LICENSE TYPES

1.5MILLION PROJECTS

76,000VULNERABILITIES

• Largest database of open source project

information in the world.

• Vulnerabilities coverage extended through

partnership with Risk Based Security.

• The KnowledgeBase is essential for identifying

and solving open source issues.

Comprehensive KnowledgeBase

Black Duck Hub Security Architecture

Hub Scan1 File and Directory Signatures2 Open Source

Component Identified

3

Hub Web ApplicationBlack Duck

KnowledgeBase

On Premise Black Duck Data Center

We Need Your Help

Knowledge is power• Know what’s running and why

• Define proactive vulnerability response process

• Don’t let technology hype cycle dictate security

Invest in defense in depth models• Don’t rely on perimeter security to do heavy lifting

• Do look at hypervisor & container trends in security

• Make developers and ops teams part of the solution

• Focus attention on vulnerability remediation

Together we can build a more secure data center

Free Tools to Help

Free Docker Container Security Scanner

• https://info.blackducksoftware.com/Security-Scan.html

14 Day Free Trial to Black Duck Hub

• https://info.blackducksoftware.com/Demo.html

https://info.blackducksoftware.com/Security-Scan.html

https://info.blackducksoftware.com/Demo.html

Secure Application Development in the Age of Continuous Delivery

Technology

Transcript of Secure Application Development in the Age of Continuous Delivery