Unlock Security Insight from Machine Data

Unlock Security Insight
from Machine Data

Narudom Roongsiriwong, CISSP

WhoAmI

Lazy BloggerJapan, Security, FOSS, Politics, Christian

http://narudomr.blogspot.com

5 Years In Log Analysis

Consultant, OWASP Thailand Chapter

Head of IT Security, Kiatnakin Bank PLC (KKP)

[email protected]

Objective

Lay foundation of Big Data analytics using information security scenarios for example

State the practical analytics from my experience

Show how to acquire each component to fulfill operational requirement.

Agenda

Know Your Machine Data

Know Your Context

Look for Insight

Identify Measure

Security Analysis Life Cycle

Implementation



Types of Data

Information from Each Data Type

Size of DataBytes per Event

Numbers of Events per Second, Minute, Hour, Day, Month

Percentage of Each Data Type Compared to Total Data Size

Time Series

Know Your Machine Data: Firewall

Types of DataAccess Control Log (Accepted/Denied Log)

Administrative Activity Log

System Status Log

Other Next Generation Firewall Logs; IDS, SIP, Connection Built/Teardown

Information from Each Type of DataAccess Control Log: Start Time, Action, Source IP/Port, Destination IP/Port, Protocol, etc.

Administrative Activity Log: Time, User, Action, Result, etc.

Cisco ASA: Built/Teardown Log

Apr 29 2013 12:59:50: %ASA-6-305011: Built dynamic TCP translation from inside:X.X.3.42/4952 to outside:X.X.X.130/12834Apr 29 2013 12:59:50: %ASA-6-302013: Built outbound TCP connection 89743274 for outside:X.X.X.43/443 (X.X.X.43/443) to inside:X.X.3.42/4952 (X.X.X.130/12834)Apr 29 2013 12:59:50: %ASA-6-305011: Built dynamic UDP translation from inside:X.X.1.35/52925 to outside:X.X.X.130/25882Apr 29 2013 12:59:50: %ASA-6-302015: Built outbound UDP connection 89743275 for outside:X.X.X.222/53 (X.X.X.222/53) to inside:X.X.1.35/52925 (X.X.X.130/25882)Apr 29 2013 12:59:50: %ASA-6-305012: Teardown dynamic UDP translation from inside:X.X.1.24/63322 to outside:X.X.X.130/59309 duration 0:00:30Apr 29 2013 12:59:50: %ASA-6-305011: Built dynamic TCP translation from inside:X.X.3.42/4953 to outside:X.X.X.130/45392Apr 29 2013 12:59:50: %ASA-6-302013: Built outbound TCP connection 89743276 for outside:X.X.X.1/80 (X.X.X.1/80) to inside:X.X.3.42/4953 (X.X.X.130/45392)Apr 29 2013 12:59:50: %ASA-6-302016: Teardown UDP connection 89743275 for outside:X.X.X.222/53 to inside:X.X.1.35/52925 duration 0:00:00 bytes 140Apr 29 2013 12:59:50: %ASA-6-305011: Built dynamic TCP translation from inside:X.X.3.42/4954 to outside:X.X.X.130/10879Apr 29 2013 12:59:50: %ASA-6-302013: Built outbound TCP connection 89743277 for outside:X.X.X.17/80 (X.X.X.17/80) to inside:X.X.3.42/4954 (X.X.X.130/10879)

Cisco ASA: Access Log Intelligence

Translate IP Address to Domain User

Know Your Machine Data: IPS/IDS

Type of DataIPS Event: Blocked, Alert

Packet Acquisition (PCAP)

Contextual Information (Intelligence)

System Status Log

Information from Each Type of DataIPS Event: Source IP/Port, Destination IP/Port, Name of Matched Rule, etc.

Packet Acquisition: Raw Data or Payload

Contextual Information: IP to Domain, IP to User, Application Detection, etc.

Cisco FirePower (SourceFire): eStreamer

The Cisco Event Streamer (also known as eStreamer) allows you to stream FireSIGHT System intrusion, discovery, and connection data from the Cisco Defense Center or managed device to external client applications.

Provides more intelligent information than IPS/IDS alert logs.

Know Your Machine Data: Windows

Type of DataSecurity

System

Application

Information from Each Type of DataTime Generated, Time Written, Event ID, Event Type, User, Computer, Keyword

Windows Server 2003 vs 2008 Event ID's

EVT vs EVTX

Know Your Machine Data: Web Server

Type of DataAccess Log

Error Log

Information from Each Type of DataAccess Log: Client IP, User ID, Finished Time, Request Method, URL, HTTP Version, Status Code, Returned Size

Error Log: Time, Log Level, Client IP Address, Error Message

Information vs Noise

Know Your Context

What Is Context?

Context is the information surrounding the information. Without context, information can be misinterpreted.

Context may be information of your environment.

Information of context is normally constant, rarely changed.

Context Example:
NAT & Port Forwarding

Context Example:
Proxy, IDS & Firewall

Context Example:
Multiple IP Address Server (Multi-Homed Server)

Other Context Examples

Security Policies & Compliance

System Information: OS, Patches, Middleware, Applications, etc.

Vulnerability Database

Risk Profile

Look for Insight

What is Insight?

The capacity to gain an accurate and deep intuitive understanding of a person or thing

Where Does Insight Comes From?

The best insights tend to come from sources that can be categorized

Insight ChannelsAnomalies: Deviations from the norm

Confluence: Macro trend intersection

Frustrations: Deficiencies in the system

Orthodoxies: Question conventional beliefs

Extremities: Learn from the behaviors of leading or laggard actors

Voyages: Learn how your stakeholders live, work, and behave

Analogies: Borrow from other industries or organizations

Harvard Business Review, November 2014 Issue

Anomalies
Deviations from the norm

Security insights frequently come from anomalies.

Confluence
Macro trend intersection

Two or more data sets are direct or reverse variation to each others.

Frustrations
Deficiencies in the system

Frustration is security risk and leads to an offense or policy violation, reflects in machine data.

Tell about source code uploaded to BitBucket.

Orthodoxies
Question conventional beliefs

Are there assumptions or beliefs in your environment that go unexamined?

Extremities
Learn from the behaviors of leading or laggard actors

Analyze Traffic from RussiaHow about the missing actors?

Voyages
Learn how your stakeholders live, work, and behave

Sometimes it is hard to figure out why data set seems strange until you see what are going on the fields.

Case: Share printer as Administrator on a domain member windows client.

Analogies
Borrow from other industries or organizations

Knowledge from the othersOther Industries

Other organization in the same industry

Forms of KnowledgeStandard

Best Practice

Security Pattern : A packaged reusable solution to a recurrent problem which embody the experiences and knowledge of many security designers.

Analysis or Research Papers

Methodologies or Algorithms

Identify the Measures

Context Analysis

Security PolicyDate/
Time/Source Match

System StatusSystem Exposure

Vulnerability DatabaseMessage Analysis

Intrusion PathAlertStats

Behavior
AnalysisFunctional
AnalysisStructural
Analysiswww.cust1.comhack1.comhack2.commail.cust1.comhack1.comhack3.comwww.cust2.comhack2.comhack3.comhack1.comwww.cust1.commail.cust1.comhack2.comwww.cust1.comwww.cust2.comhack3.commail.cust1.comwww.cust2.comSource &Target Correlation

Contexts

Conventional Cyber Attacks

Reconnaissance (Foot Printing)

Enumeration & Fingerprinting

Identification of Vulnerabilities

Attack Exploit the Vulnerabilities

Gaining Access

Escalating Privilege

Covering Tracks

Creating Back Doors

Objective Collecting as much information about the targetDNS Servers

IP Ranges

Administrative Contacts

Problems revealed by administrators

MethodsGather information from Search engines, forums, internet databases (whois, ripe, arin, apnic)

Use tools PING, whois, Traceroute, DIG, nslookup, sam spade

No log source affected

Reconnaissance (Foot Printing)

ObjectiveSpecific targets determined

Identification of Services / open ports

Operating System Enumeration

MethodsBanner grabbing

Responses to various protocol (ICMP &TCP) commands

Port / Service Scans TCP Connect, TCP SYN, TCP FIN, etc.

Tools Nmap, FScan, Hping, Firewalk, netcat, tcpdump, ssh, telnet, SNMP Scanner

Enumeration & Fingerprinting

Primary log sources affectedFirewall Access Log

IPS/IDS Alert Log

Secondary log sources affectedOS Security Log

Enumeration & Fingerprinting Detection

Objective: Finding target vulnerabilitiesInsecure Configuration

Weak passwords

Unpatched vulnerabilities in services, Operating systems, applications

Possible Vulnerabilities in Services, Operating Systems

Insecure programming

Weak Access Control

MethodsUnpatched / Possible Vulnerabilities Tools, Vulnerability information Websites

Weak Passwords Default Passwords, Brute force, Social Engineering, Listening to Traffic

Insecure Programming SQL Injection, Listening to Traffic

Weak Access Control Using the Application Logic, SQL Injection

Identification of Vulnerabilities

Identification Detection

Primary log sourcesIPS/IDS alert logs

OS security logs

Web server access logs

Secondary log sourcesHost-Based IDS

Web Application Firewall

Database Firewall

Attack Exploit the Vulnerabilities

Network Infrastructure AttacksExploit network equipment

Weaknesses in TCP / IP, NetBIOS

Flooding the network to cause DOS

Operating System AttacksAttacking Authentication Systems

Exploiting Protocol Implementations

Exploiting Insecure configuration

Breaking File-System Security

Application Specific AttacksExploiting implementations of HTTP, SMTP protocols

Gaining access to application Databases

SQL Injection

Spamming

Attack Detection

Network Infrastructure AttacksFirewall logs: access, administration and system status

IPS/IDS logs: alert and system status

Operating System AttacksIPS/IDS alert logs

OS security logs

Special Security S/W logs Host-Based IDS

Application Specific AttacksWeb server logs access and error

IPS/IDS alert logs

Special Security Device & S/W logs Host-Based IDS, Web Application Firewall, Database Firewall

After exploitation success, attempt to access the target

TechniquesPassword eavesdropping

File share brute forcing

Password file grab

Buffer overflows

Gaining Access

Gaining Access Detection

TechniqueDetection form Log Sources

Password eavesdroppingNone

Buffer overflows

File share brute forcingOS file audit logs (not installed by default, Linux's auditd for example)


Password file grab

If only user-level access was obtained in the last step, the attacker will now seek to gain complete control of the system

Techniques Password cracking

Known exploits

Detection: Privilege User Creation or LoginOS security logs

Escalating Privileges

Objective: After successful compromising, hiding this fact from system administrators.

TechniquesClear logs

Hide tools

Detection: Log service stop, log file deleted or unauthorized changeOS security logs***


Covering Tracks

Objective: Ensure that privileged access is easily regained.

Techniques Create rogue user accounts

Schedule batch jobs

Infect startup files

Plant remote control services

Install monitoring mechanisms

Replace apps with Trojans

DetectionOS security logs***

OS file audit logs***


Creating Back Doors

Measure for Host Scanning*

ContextWe have firewall separated Internet and internal network

We have IP network x.x.x.x/26 (64 IP)

Attack PatternAttackers use one source IP try to connect to many destination IP from the Internet.

Possible MeasureFound accepted/denied access control log from Firewall with one source IP to many IP addresses > 20 IP addresses in one minutes

*For example only, the most effective way is to implement the IDS probing firewall's interface connected to the Internet

Measure for Port Scanning*

ContextWe have firewall separated Internet and internal network

Attack PatternAttackers use one source IP try to connect to one destination IP on various ports from the Internet.

Possible MeasureFound accepted/denied access control log from Firewall with one source IP to one IP address on different ports > 20 ports in one minutes

*For example only, the most effective way is to implement the IDS probing firewall's interface connected to the Internet

Measure for Centralized HTTP Botnet

HTTP C&C Server

Bot Master

Botnet

Botnet

Check for new command

Check for new command

Command

Command

Measure for Centralized HTTP Botnet

ContextWe have firewall separated Internet and internal network, outbound only port 80 & 443

Attack PatternThe bots connect to them periodically to get new commands from the bot master.

The instructions of the bots tend to be short. The lengths of command packets are typically small size of 1KB or even less

Possible MeasureFound accepted log from Firewall to one destination IP address with byte-in size less than 1K for 3 or more events per hour.



Look for Insights

Identify Measures

Detect Incidents

Verify Incidents

False Positive vs False Negative

Good Measure Minimize False Detection

Implementation

Implementation

E = Event Generator

C = Collection

D = Data Storage with Indexes

A = Analysis Tool

K = Knowledge Base

R = Reaction & Reporting

DCCEEEEEAKR

Event Generator

SensorIDS

Any system providing logs

Agents

PollerSNMP

DCCEEEEEAKR

Collection + Data Storage with Indexes

CollectionGather information from different sensors

Filter

Parse useful information (tag or normalize)

Aggregate

Data Storage with IndexesStore raw or formatted data with index

DCCEEEEEAKR

Analysis + Knowledge Base

AnalysisAnalyze events stored in data storage

Correlation algorithms, false-positive message detection, mathematical representation

Knowledge BaseContext Information

Intrusion Path

System Model

Security Policy

DCCEEEEEAKR

Reaction and Reporting

Subjective ConceptDashboard

Report

Security Policy Enforcement Strategy

Legal Constraints

Contractual SLAs

DCCEEEEEAKR

Solution#1

ComponentImplementation

CollectionSYSLOG DaemonBash script with grep+sed+awk

Data Storage with IndexesCSV Files

Analysis ToolMicrosoft Excel

Knowledge BaseMicrosoft Excel

Reaction & ReportingMicrosoft Excel

The Good: Low Cost

The Bad: Automation only for collection

The Ugly: Analysis once a day

Solution#2


CollectionWindows Service (In-house)

Data Storage with IndexesMS SQL

Analysis ToolWindows Client Application (In-house)

Knowledge BaseMS SQL

Reaction & ReportingWindows Client Application (In-house)

The Good: Built-in security surveillance process

The Bad: Unable to handle more than 1 GB/day and lost some information from normalization

The Ugly: Searching specific event using grep on raw data faster than from database 10 times or more

Solution#3


CollectionSplunk Forwarder

Data Storage with IndexesSplunk Indexer

Analysis ToolSplunk Search Head

Knowledge BaseSplunk Built in tables, RDBMS in the future

Reaction & ReportingSplunk Search Head

The Good: Scalable

The Bad: Expensive!!!

Useful Skills

Data InterpretationNetwork, System, Application

Information Security Knowledge

Search Skill

Regular Expression

Copyright 2000-2011 e-Cop. All rights reserved worldwide.Click to edit Master title style

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelClick to edit Master text stylesSecond levelThird levelFourth level

Fifth level

6/18/16

Unlock Security Insight from Machine Data

Data & Analytics

Transcript of Unlock Security Insight from Machine Data