Towards Interactive Network Forensics andIncident Response

Matthias VallentinUC Berkeley / ICSI

vallentin@icir.org

Boundary Tech TalkSan Francisco, CA

November 17, 2011

Motivation

What do the following activities have in common?I Network troubleshootingI Incident responseI Network forensics

→ Data-intensive analysis of past activity→ Interactive response times often critical

“How to build a platform that efficiently supports these activities?”

2 / 36

Motivation

What do the following activities have in common?I Network troubleshootingI Incident responseI Network forensics

→ Data-intensive analysis of past activity→ Interactive response times often critical

“How to build a platform that efficiently supports these activities?”

2 / 36

Motivation

What do the following activities have in common?I Network troubleshootingI Incident responseI Network forensics

→ Data-intensive analysis of past activity→ Interactive response times often critical

“How to build a platform that efficiently supports these activities?”

2 / 36

Outline

1. Incident Response and Network Forensics

2. Operational Network Monitoring using Bro

3. Building an Interactive Analytics Platform

3 / 36

About

I 4th-year PhD student at UC Berkeley, advised by Vern PaxsonI Working with researchers at ICSI/ICIR and the AMPlabI Interests

I Large-scale network intrusion detectionI High-performance traffic analysisI Network forensics and incident response→ with strong operational emphasis

I ProjectsI The Bro network security monitorI VAST: Visibility Across Space and TimeI HILTI: High-Level Intermediate Language for Traffic Inspection

4 / 36

Outline

1. Incident Response and Network Forensics

2. Operational Network Monitoring using Bro

3. Building an Interactive Analytics Platform

5 / 36

Use Case #1: Classic Incident Response

I Goal: fast and comprehensive analysis of security incidentsI Often begins with an external piece of intelligence

I “IP X serves malware over HTTP”I “This MD5 hash is malware”I “Connections to 128.11.5.0/27 at port 42000 are malicious”

I Analysis style: Ad-hoc, interactive, several refinements/adaptionsI Typical operations

I Filter: project, selectI Aggregate: mean, sum, quantile, min/max, histogram, top-k,

unique

⇒ Concrete starting point, then widen scope (bottom-up)

6 / 36

Use Case #2: Network Troubleshooting

I Goal: find root cause of component failureI Often no specific hint, merely symptomatic feedback

I “I can’t access my Gmail”I Typical operations

I Zoom: slice activity at different granularitiesI Time: seconds, minutes, days, . . .I Space: layer 2/3/4/7, host, subnet, port, URL, . . .

I Study time series data of activity aggregatesI Find abnormal activity

I “Today we see 20% less outbound DNS compared to yesterday”I Infer dependency graphs: use joint behavior from past to asses present

impact [KMV+09]I Judicious machine learning [SP10]

⇒ No concrete starting point, narrow scope (top-down)

7 / 36

Use Case #3: Combating Insider Abuse

I Goal: uncover policy violations of personnelI Analysis procedure: connect the dotsI Insider attack:

I Chain of authorized actions, hard to detect individuallyI E.g., data exfiltration

1. User logs in to internal machine2. Copies sensitive document to local machine3. Sends document to third party via email

I Typical operationsI Compare activity profiles

I “Jon never logs in to our backup machine at 3am”I “Seth accessed 10x more files on our servers today”

⇒ Relate temporally distant events, behavior-based detection

8 / 36

Outline

1. Incident Response and Network Forensics

2. Operational Network Monitoring using Bro

3. Building an Interactive Analytics Platform

9 / 36

Basic Network Monitoring

Internet Local NetworkTap

Monitor

I SitesI UC Berkeley (10 Gbps, 50,000 hosts)I NCSA, IL (8×10 Gbps, 10,000 hosts)I LBNL, Berkeley (10 Gbps, 12,000 hosts)I ICSI, Berkeley (100 Mbps, 250 hosts)I AirJaldi, India (10 Mbps, 500 hosts)

10 / 36

High-Performance Network Monitoring:The NIDS Cluster [VSL+07]

Internet Local NetworkTap

Frontend

Manager

PacketsLogsState

Proxy

......Worker Worker Worker

User

11 / 36

The Bro Cluster

I We run it operationally at:I UC Berkeley (26 workers)I LBNL (15 workers)I NCSA (10 4-core workers)

I Runs at numerous large sites:I IndustryI AcademiaI Government

Internet Local NetworkTap

Worker

Frontend

Worker Worker

Proxy

Worker Worker Worker

Manager

Proxy

...

Proxy

...

PacketsLogsState

12 / 36

The Bro Network Security Monitor

I Fundamentally different from other IDSI Real-time network analysis frameworkI Policy-neutral at the coreI Highly stateful

Key components1. Event engine

I TCP stream reassemblyI Protocol analysisI Policy-neutral

2. Script interpreterI “Domain-specific Python”I Generate extensive logsI Apply site policy

User Interface

Network

Event Engine

Script Interpreter

Packets

Events

Logs Notifications

13 / 36

From Packets to High-Level Descriptions of Activity

Event declaration

type connection: record { orig: addr, resp: addr, ... }

event connection_established(c: connection)event http_request(c: connection, method: string, URI: string)event http_reply(c: connection, status: string, data: string)

Event instantiation

connection_established({127.0.0.1, 128.32.244.172, ... })http_request({127.0.0.1, 128.32.244.172, ..}, "GET", "/index.html")http_reply({127.0.0.1, 128.32.244.172, ..}, "200", "<!DOCTYPE ht..")http_request({127.0.0.1, 128.32.244.172, ..}, "GET", "/favicon.ico")http_reply({127.0.0.1, 128.32.244.172, ..}, "200", "\xBE\xEF\x..")connection_established({127.0.0.1, 128.32.112.224, ... })

14 / 36

From Packets to High-Level Descriptions of Activity

Event declaration

type connection: record { orig: addr, resp: addr, ... }

event connection_established(c: connection)event http_request(c: connection, method: string, URI: string)event http_reply(c: connection, status: string, data: string)

Event instantiation

connection_established({127.0.0.1, 128.32.244.172, ... })http_request({127.0.0.1, 128.32.244.172, ..}, "GET", "/index.html")http_reply({127.0.0.1, 128.32.244.172, ..}, "200", "<!DOCTYPE ht..")http_request({127.0.0.1, 128.32.244.172, ..}, "GET", "/favicon.ico")http_reply({127.0.0.1, 128.32.244.172, ..}, "200", "\xBE\xEF\x..")connection_established({127.0.0.1, 128.32.112.224, ... })

14 / 36

Event Extraction with BroEvent and data model

I Rich-typed: first-class networking types (addr, port, subnet, . . . )I Deep: across the whole network stackI Fine-grained: detailed protocol-level informationI Expressive: nested data with container types (aka. semi-structured)

Transport

(Inter)Network

ApplicationMessages

Byte stream

Packets

LinkFrames

http_request, smtp_reply, ssl_certificate

new_connection, udp_request

new_packet, packet_contents

arp_request, arp_reply

15 / 36

After the Fact: Bro LogsI Policy-neutral by default: no notion of good or bad

I Forensic investigations highly benefit from unbiased informationI Flexible output formats: ASCII, binary, DB, custom

% more conn.log#fields ts id.orig_h id.orig_p id.resp_h id.resp_p proto service duration obytes ..1144876741.1198 192.150.186.169 53115 82.94.237.218 80 tcp http 16.14929 4351144876612.6063 192.150.186.169 53090 198.189.255.82 80 tcp http 4.437460 86611144876596.5597 192.150.186.169 53051 193.203.227.129 80 tcp http 0.372440 4611144876606.7789 192.150.186.169 53082 198.189.255.73 80 tcp http 0.597711 3371144876741.4693 192.150.186.169 53116 82.94.237.218 80 tcp http 16.02667 30271144876745.6102 192.150.186.169 53117 66.102.7.99 80 tcp http 1.004346 4221144876605.6847 192.150.186.169 53075 207.151.118.143 80 tcp http 0.029663 347

% more http.log#fields ts id.orig_h id.orig_p host uri status_code user_agent ..1144876741.6335 192.150.186.169 53116 docs.python.org /lib/lib.css 200 Mozilla/5.01144876742.1687 192.150.186.169 53116 docs.python.org /icons/previous.png 304 Mozilla/5.01144876741.2838 192.150.186.169 53115 docs.python.org /lib/lib.html 200 Mozilla/5.01144876742.3337 192.150.186.169 53116 docs.python.org /icons/up.png 304 Mozilla/5.01144876742.3337 192.150.186.169 53116 docs.python.org /icons/next.png 304 Mozilla/5.01144876742.3337 192.150.186.169 53116 docs.python.org /icons/contents.png 304 Mozilla/5.01144876742.3337 192.150.186.169 53116 docs.python.org /icons/modules.png 304 Mozilla/5.01144876742.3338 192.150.186.169 53116 docs.python.org /icons/index.png 304 Mozilla/5.01144876745.6144 192.150.186.169 53117 www.google.com / 200 Mozilla/5.0

16 / 36

After the Fact: Bro Logs

17 / 36

Log Analysis

I What do we do with Bro logs?I Process (ad-hoc analysis)I Summarize (time series data, histogram/top-k, quantile)I Correlate (machine learning, statistical tests)I Age (elevate old data into higher levels of abstraction)I Visualize

I How do we do it?I All eggs in one basket

I SIEM: Splunk, ArcSight, NarusInsight, . . . $$$I VAST

I In-situ processingI Tools of the trade (awk, sort, uniq, . . . )I MapReduce / Hadoop

18 / 36

Log Analysis

I What do we do with Bro logs?I Process (ad-hoc analysis)I Summarize (time series data, histogram/top-k, quantile)I Correlate (machine learning, statistical tests)I Age (elevate old data into higher levels of abstraction)I Visualize

I How do we do it?I All eggs in one basket

I SIEM: Splunk, ArcSight, NarusInsight, . . . $$$I VAST

I In-situ processingI Tools of the trade (awk, sort, uniq, . . . )I MapReduce / Hadoop

18 / 36

Outline

1. Incident Response and Network Forensics

2. Operational Network Monitoring using Bro

3. Building an Interactive Analytics Platform

19 / 36

From Ephemeral to Persistent Activity

I Bro eventsI Policy-neutral activityI EphemeralI Only inside the Bro process→ Can I haz access?

I BroccoliI Send/Receive Bro eventsI Written in CI Language bindings

I RubyI PythonI Perl

→ Send-them-while-they-are-hot

User Interface

Network

Event Engine

Script Interpreter

Packets

Events

Logs Notifications

3rd-party Application

BroccoliEvents

Com

m

(Broccoli = Bro client communications library)

20 / 36

From Ephemeral to Persistent Activity

Events

Events

Apache

Broccoli

OpenSSH

Broccoli

Events

User

Query

Result

Query Result

Bro

21 / 36

Today’s Open-Source Solutions for Analytics

22 / 36

Caveats in Real-Time Analytics

1. Getting poor performanceI Batch processing (MapReduce)I Architectural flaws (inflexible MQ)I Bloated runtime (Java)

2. Losing domain-specific contextI TypingI NestingI Causality

“Can we do better?”

23 / 36

Inspiration

1. DremelI Columnar storageI Nested data model

2. BigtableI Sharding: distributed tablets

3. GFSI Single master with meta dataI Locate chunks via master

4. SawzallI Aggregators: collection, sample, sum, maximum, quantile, top-k,

unique

5. FastBitI Bitmap indexes “work” for high-cardinality attributes

24 / 36

Design Philosophy Touch Stones [Lam11]Storage

I Keep data sorted → reduce seeks, easy random entryI Shard with access locality → minimize involved nodesI Store data in columns → don’t waste I/OI Use append-only disk format → avoid expensive index updates

ComputeI Use disk appropriately → large sequential readsI Trade CPU for I/O → type-specific, aggressive compressionI Use pipelined parallelism → hide latencyI Ship compute to data → aggregation serving tree

QueryI Make it user-friendly → declarative query interfaceI Provide query hooks → support complex analysis

25 / 36

Design Philosophy Touch Stones [Lam11]Storage

I Keep data sorted → reduce seeks, easy random entryI Shard with access locality → minimize involved nodesI Store data in columns → don’t waste I/OI Use append-only disk format → avoid expensive index updates

ComputeI Use disk appropriately → large sequential readsI Trade CPU for I/O → type-specific, aggressive compressionI Use pipelined parallelism → hide latencyI Ship compute to data → aggregation serving tree

QueryI Make it user-friendly → declarative query interfaceI Provide query hooks → support complex analysis

25 / 36

Design Philosophy Touch Stones [Lam11]Storage

I Keep data sorted → reduce seeks, easy random entryI Shard with access locality → minimize involved nodesI Store data in columns → don’t waste I/OI Use append-only disk format → avoid expensive index updates

ComputeI Use disk appropriately → large sequential readsI Trade CPU for I/O → type-specific, aggressive compressionI Use pipelined parallelism → hide latencyI Ship compute to data → aggregation serving tree

QueryI Make it user-friendly → declarative query interfaceI Provide query hooks → support complex analysis

25 / 36

VAST: Visibility Across Space and Time

I VisibilityI Deep understanding of the dataI Visualization: you know how to do that already. . .

I Across space:I Unify heterogeneous data formatsI One query language

I Apache logs, SSH logs, Bro events, sensor data, . . .I Across time:

1. From the ancient past (old historical data)2. To subscribing to data that may arrive in the future

26 / 36

Queries

I Two types1. Search: historical query2. Feed: live query→ use case: crawl archive first, then make query permanent

I Unify two ends of a spectrumLive Historical

Operation Push PullLatency O(|Xresult|) O(|Xdata|)Data location In-memory Disk (ideally cached)Flexibility Predefined Ad-hoc, adjustableCost Pay-As-You-Go Lumpsum

27 / 36

VAST: Architecture Overview

I Distributed architectureI Elasticity via MQ middle layerI Few component dependencies

I DFS: fault-tolerance, replicationI Archive: key-value store

I Contains serialized eventsI Index: sharded column-store

I Compressed bitmap indexesI In-memory store

I Caches tablets (LRU)I Flushes in batches

Archive

Index

DFS

Store

Ingest Query

28 / 36

VAST: Ingestion Architecture

1. Events arrive at Event Router1.1 Assign UUID x1.2 Put (x, event) in archive1.3 Forward event to Indexer

2. Indexer writes event into tabletand updates indexes

3. Tablet Manager flushes “ripe”tablets

I Capacity (space/rows)I Lifetime

Store

EventRouter

Tablets

Tablet Manager

Indexer

flush

ripe?

Index

DFS

put

Tablets

write

Archive

29 / 36

VAST: Query Architecture

1. User or NIDS issues query2. Query Manager distributes it

to relevant nodes3. Tablet Manager load tablets4. Query Proxy hits index

a Returns direct resultb Returns set of UUIDs

Store

Query Manager

Tablets

Tablet Manager

QueryProxy

flush

LRU

Index

DFS

get

Tablets

query

Archive load

30 / 36

Bitmap Indexes

I Column cardinality: # distinct valuesI One bitmap bi for each value i

I Sparse, but compressibleI WAH [WOSN01]I COMPAX [FSV10]I Consice [CDP10]

I Can operate on compressed bitmapsI No need to decompress

2

1

2

0

0

1

3

0

0

0

1

1

0

0

0

1

0

0

0

1

0

1

0

1

0

0

0

0

0

0

0

0

0

0

1

b1 b2 b3b0

Data Bitmap Index

31 / 36

Conclusion

1. Motivation: incident response, network troubleshooting, insider abuse2. The Bro network security monitor

I High-performance network monitoringI Expressive representation of activityI Publish/subscribe event model

3. Design sketch of a distributed analytics platform

32 / 36

Questions?

33 / 36

References I

A. Colantonio and R. Di Pietro.Concise: Compressed ’n’ Composable Integer Set.Information Processing Letters, 110(16):644–650, 2010.

Francesco Fusco, Marc Ph. Stoecklin, and Michail Vlachos.NET-FLi: On-the-fly Compression, Archiving and Indexing ofStreaming Network Traffic.Proceedings of the VLDB Endowment, 3:1382–1393, September 2010.

Srikanth Kandula, Ratul Mahajan, Patrick Verkaik, Sharad Agarwal,Jitendra Padhye, and Paramvir Bahl.Detailed Diagnosis in Enterprise Networks.In Proceedings of the ACM SIGCOMM 2009 Conference on DataCommunication, SIGCOMM ’09, pages 243–254, New York, NY, USA,2009. ACM.

34 / 36

References II

Andrew Lamb.Building Blocks for Large Analytic Systems.In 5th Extremely Large Databases Conference, XLDB ’11, Menlo Park,California, October 2011.

Robin Sommer and Vern Paxson.Outside the Closed World: On Using Machine Learning for NetworkIntrusion Detection.In Proceedings of the 2010 IEEE Symposium on Security and Privacy,SP ’10, pages 305–316, Washington, DC, USA, 2010. IEEE ComputerSociety.

35 / 36

References III

Matthias Vallentin, Robin Sommer, Jason Lee, Craig Leres, VernPaxson, and Brian Tierney.The NIDS Cluster: Scalably Stateful Network Intrusion Detection onCommodity Hardware.In Proceedings of the 10th International Conference on RecentAdvances in Intrusion Detection, RAID’07, pages 107–126.Springer-Verlag, September 2007.

Kesheng Wu, Ekow J. Otoo, Arie Shoshani, and Henrik Nordberg.Notes on Design and Implementation of Compressed Bit Vectors.Technical Report LBNL-3161, Lawrence Berkeley National Laboratory,Berkeley, CA, USA, 94720, 2001.

36 / 36