Cloud-based Log Analysis and Visualization

mobile-166 My syslog

DeepSec 2010, Vienna, Austria

Raffael Marty - @zrlram

© by Raffael MartyLogging as a Service

Raffael (Raffy) Marty

2

• Founder @ • Chief Security Strategist and Product Manager @ Splunk•Manager Solutions @ ArcSight• Intrusion Detection Research @ IBM Research• IT Security Consultant @ PriceWaterhouse Coopers

Applied Security VisualizationPublisher: Addison Wesley (August, 2008)

ISBN: 0321510100

© by Raffael MartyLogging as a Service

Agenda

3

•Introduction

•Beaver Challenge

•The Cloud

•Visualization

•Visualization Tools

•Visualization in the Cloud

•Visualization Use-Cases

•Visualization Resources

Raffael Marty - @zrlram

The Public Cloud

4

IaaS - InfrastructurePaaS - PlatformSaaS - Software

LaaS - Logging

What is really new and has changed?

Raffael Marty - @zrlram

Visibility and Big Data

5

Raffael Marty - @zrlram

Visibility

6

•Monitoring-Performance-Availability-Ephemeral Infrastructure

•Security-New Threats-New Vulnerabilities-Different Risk Distribution

IaaS - Similar to beforePaaS - Lack of InfrastructureSaaS - Blind?

Application Instrumentationand Logging

Raffael Marty - @zrlram

Big Data

7

•NoSQL•Distributed data stores•Distributed queues•Map reduce•ETL (Extract, Transform, Load)

•...

Logging as a Service

Raffael Marty - @zrlram

Information Visualization

8

•Better tools and capabilities

•Across disciplines•More instrumentation•Dichotomies

© by Raffael MartyLogging as a Service

Open Your Eyes

9

© by Raffael MartyLogging as a Service

Information Visualization?

A picture is worth a thousand log records.

Inspire

Pose a New Question

Explore and Discover

Support Decisions

Communicate Information

Increase Efficiency

Answer a Question

10

Visualization Tools

11

© by Raffael MartyLogging as a Service

Reporting vs. Visualization

12

•Reporting Libraries-HighCharts-Flot-Google Chart API-Open Flash Chart-HTML5

•Visualization Libraries-TheJIT-Graphael-Protovis-ProcessingJS-Flare

JavaScript vs. Flash vs. XYZ

© by Raffael MartyLogging as a Service

HighCharts

13

•Click-Through•On load-near real-time updates

•Zoom•AJAX data input via JSON

http://www.highcharts.com/

© by Raffael MartyLogging as a Service

Google Visualization API

• JavaScript•Based on DataTables()•Many graphs•Playground

- http://code.google.com/apis/ajax/playground

14

http://code.google.com/apis/visualization/interactive_charts.html

© by Raffael MartyLogging as a Service

ProtoVis•JavaScript based visualization library•Charting•Treemaps•BoxPlots•Parallel Coordinates•etc.

15

http://vis.stanford.edu/protovis/

© by Raffael MartyLogging as a Service

TheJIT•JavaScript InfoVis Toolkit•Interactive•Link Graphs

16

http://thejit.org/

© by Raffael MartyLogging as a Service

Processing•Visualization library•Java based•Interactive (event handling)•Number of libraries to

-draw in OpenGL-read XML files-write PDF files

•Processing JS-JavaScript-HTML 5 Canvas-Web IDE

17

http://processing.org/http://processingjs.org/

Data Visualization in the Cloud

18

© by Raffael MartyLogging as a Service

LaaS - Logging as a Service

19

• Log collection

• all data in one place

• Log storage and management

• index, storage, archive

• Extremely fast log search across all your data

• data source agnostic (no parsers)

• innovative Web shell

• API log access

• oAuth authentication

• always on

Benefits• No installation• Easy configuration• No maintenance

• Great scalability• 7x24 availability• Pay as you go

© by Raffael MartyLogging as a Service

AfterGlow Cloud

20

Grapher Loggly

JSON

CSV

DOT

Graph

Visualization Use-Cases

21

© by Raffael MartyLogging as a Service

Old Skewl

22

Yesterday Today - Cloud

© by Raffael MartyLogging as a Service

Traditional Style

23

Yesterday Today - Cloud

© by Raffael MartyLogging as a Service

The Analysis Approach

24

Overview first Zoom Details on demand

Principle by Ben Shneiderman

© by Raffael MartyLogging as a Service

NetFlow Visualization• Treemap• Protovis.JS• Size: Amount • Brightness: Variance• Color: Sensor• Shows: Scans - bright spots

• Thanks to Chris Horsley

25

© by Raffael MartyLogging as a Service

Firewall Treemap

26

© by Raffael MartyLogging as a Service

Firewall LogPort Source IP Destination IP

27

© by Raffael MartyLogging as a Service

IDS Signature TuningTop signatures

© by Raffael MartyLogging as a Service

Signatures Over Time

© by Raffael MartyLogging as a Service

IDS Sig Tuning - Treemap

30

Hierarchy: SourceDestinationSignatureNumber of Events

Color: ServiceSize: Number of alerts

© by Raffael MartyLogging as a Service

IDS Sig Tuning - Treemap

31

Hierarchy: SourceDestinationSignatureNumber of Events

Color: PrioritySize: Number of alerts

© by Raffael MartyLogging as a Service

IDS Sig Tuning - Treemap

32

Hierarchy: SignatureSourceService (Port)

Color: PrioritySize: Number of alerts

Visualization Resources

33

© by Raffael MartyLogging as a Service

Share, discuss, challenge, and learn about security visualization.

http://secviz.org

•List: secviz.org/mailinglist

•Twitter: @secviz

34

© by Raffael MartyLogging as a Service

Applied Security Visualization• Bridging the gap between security and visualization• Hands-on, end to end examples• Data processing and analysis

Chapters• Visualization• Data Sources• From Data to Graphs• Perimeter Threat

35

Addison Wesley (August, 2008)ISBN: 0321510100

• Compliance• Insider Threat• Visualization Tools

36

about.me/raffy

We are hiring!