Statistical Flow analysis
description
Transcript of Statistical Flow analysis
Section 4.1
Network Forensics
TRACKING HACKERS THROUGH CYBERSPACE
STATISTICAL FLOW ANALYSIS
PURPOSE• Identify compromised hosts
• Send out more traffic
• Use usual ports
• Communicate with known malicious systems
• Confirm / Disprove data leakage
• Volume of exported data
• Individual profiling
• Reveal
• Normal working hours
• Periods of inactivity
• Sources of entertainment
• Correlate activity exchanges
PROCESS OVERVIEW• Defined
• “Flow record—A subset of information about a flow. Typically, a flow record includes the source and destination IP address, source and destination port (where applicable), protocol, date, time, and the amount of data transmitted in each flow.” (Davidoff & Ham, 2012)
1. PG 161
FLOW RECORD PROCESSING SYSTEM• Flow record processing systems include the following components:
• Sensor—The device that is used to monitor the flows of traffic on any given segment and extract important bits of information to a flow record.
• Collector—A server (or multiple servers) configured to listen on the network for flow record data and store it to a hard drive.
• Aggregator—When multiple collectors are used, the data is typically aggregated on a central server for analysis.
• Analysis—Once the flow record data has been exported and stored, it can be analyzed using a wide variety of commercial, open-source, and homegrown tools. 1
SENSORS
• Sensor types
• Network Equipment
• Many switches support flow record creation and export
• Cisco - NetFlow format
• Sonicwall – IPFIX and NetFlow
• Be cautious of “sampling” which is not comprehensive data
• Standalone appliances
• Used if existing network software does not support flow data
• Software
• Argus – Audit Record Generation and Utilization System
• Softflowd
• Yaf – Yet Another Flowmeter
SENSOR SOFTWARE• Argus
• Two packages• Argus Server• Argus Client
• Libpcap- based• Supports BPF filtering• Documentation specifically mentions forensic investigation• Argus’ compressed format over UDP
• Softflowd• Passively monitor traffic• Exports record data in NetFlow format• Linux and OpenBSD• Libpcap- based
• Yaf• Libpcap and live packet transfer• IPFIX format over SCTP, TCP or UDP• Supports BPF filters
SENSOR PLACEMENT• Investigators often do not have much control over placement
• Infrastructures should be set up with flow monitoring in mind but usually are not
• Factors to consider
• Duplication is inefficient and must be minimized
• Time synchronization is crucial
• Most flow records are collected on external devices such as firewalls but this ignores internal network traffic which can be valuable
• Resources are important when planning, prioritize
• Do not over load your network capacity
MODIFYING THE ENVIRONMENT• Leverage existing equipment
• Switches, routers, firewalls, NIDS / NIPS
• Upgrade network equipment
• If existing equipment will not work deploy replacements
• Deploy additional sensors
• Use port mirroring to send packets to standalone sensor
• Network tap another option
FLOW RECORD EXPORT PROTOCOLS• Proprietary – Cisco’s NetFlow
• Open source – IPFIX
• Relatively new and not yet matured – better tools on the horizon
NETFLOW• Maintains a cache that tracks the state of all active flows observed
• Completed flows marked as “expired” and exported as a “NetFlow Export” packet to a collector
• Newer versions (NetFlow v9) are transport-layer independent: UDP, TCP and SCTP
• Older versions only support UDP and IPv4
IPFIX• Extends NetFlow v9
• Handles bidirectional flow reporting
• Reduces redundancy
• Better interoperability
• Extensible flow record data using data templates
• Template defines data to be exported
• Sensor uses template to construct flow data export packets
SFLOW• Supported by many devices – not Cisco
• Conduct statistical packet sampling
• Does not support recording and processing every packet
• Scales very well
• Generally not very good for forensic analysis
COLLECTION AND AGGREGATION• Placement factors to consider
• Congestion• Flow records generate network traffic and can intensify congestion• Choose location where this will cause low network impact
• Security• Export flow records on separate VLAN if possible• Isolate physical cables• Encrypt using IPSec or TLS
• Reliability• Consider using TCP or SCTP over UDP
• Capacity• One sensor or many?
• Analysis strategy• Can affect all of the above, plan accordingly
COLLECTION SYSTEMS• Commercial options
• Cisco NetFlow Collector
• Manage Engine’s NetFlow Analyzer
• WatchPoint NetFlow Collector
COLLECTION SYSTEMS CONTINUED• Open source options
• SiLK – System for Internet Level Knowledge• Command-line • Most powerful – biggest learning curve• Collector specific tools – flowcap and rwflowpack
• Flow-tools• Modular and easily extensible• Only accepts UDP input
• Nfdump / NfSen• Collector daemon – nfcapd• UDP network socket or pcap files
• Argus• Supports Argus format and NetFlow v 1-8
• NetFlow v9 and IPFIX not yet supported
ANALYSIS• Defined
• “Statistics—“The science which has to do with the collection, classification, and analysis of facts of a numerical nature regarding any topic.” (The Collaborative International Dictionary of English v.0.48).” (Davidoff & Ham, 2012)
• Purpose
• Store a summary of information about the traffic flowing across the network
• Forensic data carving does not apply
• Still very useful
FLOW RECORD TECHNIQUES• Goals and resources
• This should shape your analysis
• Access available time, staff, equipment and tools
• Starting indicators – triggering event
• Example evidence:
• IP address of compromised or malicious system
• Time frame of suspect activity
• Known ports of suspect activity
• Specific flows which indicate abnormal or unexplained activity
FLOW RECORD TECHNIQUES CONTINUED• Analysis techniques
• Filtering
• Baselining
• “Dirty Values”
• Activity pattern matching
FILTERING• Important to narrow down a large pool of evidence
• Remove extraneous data
• Start by isolating activity relating to specific IP address/es
• Filter for known patterns of behavior
• Use small percentages of data for detailed analysis
BASELINING• Advantage of flow record data vs full traffic capture
• Dramatically smaller allowing for longer retention
• Build a profile of “normal” network activity
• Network baseline
• General trends over a period of time
• Host baseline
• Historical baseline can identify anomalous behavior
• Most flow patterns will change dramatically if host is compromised or under attack
“DIRTY VALUES”• Suspicious keywords
• IP addresses
• Ports
• Protocols
ACTIVITY PATTERN MATCHING• Elements
• IP address• Internal network or Internet-exposed network• Country of origin• Who are they registered too?
• Ports• Assigned / well-known ports link to specific applications• Is system scanning or being scanned?
• Protocols and Flags• Layer 3 and 4 are often tracked in flow record data
• Connection attempts• Successful port scans• Data transfers
• Directionality• Data coming in (something downloaded) or going out (something uploaded)
• Volume of data transferred• Lots of small packets can indicate port scanning• Large amounts of data usually cause for concern
SIMPLE PATTERNS• Many-to-one IP addresses
• DOS attack• Syslog server• “Drop box” data repository on destination IP• Email server (at destination)
• One-to-many IP addresses• Web server• Email server (at source)• SPAM bot• Warez server• Network port scanning
• Many-to-many IP addresses• Peer-to-peer file sharing• Widespread port scanning
• One-to-one IP addresses• Targeted attack• Routine Server communication
COMPLEX PATTERNS• Fingerprinting
• Matching complex flow record patterns to specific activities
• Example:
• TCP SYN port scan
• One source IP address
• One or more destination IP addresses
• Destination port numbers increase incrementally
• Volume of packets surpass a specified value within a given period of time
• TCP protocol
• Outbound protocol flags set to “SYN”
FLOW RECORD ANALYSIS TOOLS• flowtools
• SiLK
• Argus
• FlowTraq
• Nfdump / NfSen
SiLK• Rwfilter
• Extracts flows of interest• Filters by time and category• Partitions them by protocol attributes• Generally as functional as BPF
• Rwstats, rwcounts, rwcut, rwuniq• Basic manipulation utilities
• Rwidsquery• Can be fed a Snort rule or alert file and it will figure out which flow matches it and writes an
rwfilter to match it• Rwpmatch
• Libpcap-based program that reads in SiLK-format flow metadata and an input source and save only the packets that match the metadata
• Advanced SiLK• Includes a Python interpreter “PySiLK”
FLOW-TOOLS• Variety
• Flow export data collection
• Storage
• Processing
• Sending tools
• “flow-report”
• ASCII text report based on stored flow data
• “flow-nfilter”
• Filter based on primitives specific to flow-tools
• “flow-dscan”
• Identifies suspicious traffic based on flow export data
ARGUS CLIENT TOOLS• Ra
• Reads• Filters• Prints• Supports BPF filtering
• Racluster• Exports based on user-specified criteria
• Rasort• Sorts based on user-specified criteria
• Ragrep • Regular expression and pattern matching
• Rahisto
• Generated frequency distribution table for user-selected metrics: flow duration, src and dst port numbers, byte transfer, packet counts, average duration, IP address, ports, etc
FLOW TRAQ• Commercial tool by ProQueSys
• Supports many formats and sniffs traffic directly
• Users can
• Filter
• Search
• Sort
• Produce reports
• Designed for forensics and incident response
NFDUMP• Part of the nfdump suite• Includes
• Aggregate flow record fields by specific fields• Limit by time range• Generate statistics
• IP addresses• Interfaces• Ports
• Anonymize IP addresses• Customize output format• BPF-style filters
NFSEN• Graphical, web-based interface for nfdump
ETHERAPE• Libpcap-based graphical tool
• Visually displays activity in real time
• Colors designate traffic protocol
• HTTP
• SMB
• ICMP
• IMAPS
• Does not take flow records as input
Works Cited
Davidoff, S., & Ham, J. (2012). Network Forensics Tracking Hackers Through Cyberspace. Boston: Prentice Hall.