NetFlowftp.cisco.cz/.../2013-10-03-NetFlow-JiriTesar.pdf2013/10/03 · • The NetFlow v9 Exporter...

Cisco Confidential © 2012 Cisco and/or its affiliates. All rights reserved. 1

NetFlow Technology Overview

Jiří Tesař, CCIE #14558 Systems Engineering

Cisco Connect Club

[email protected]

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

•  Developed and patented at Cisco® Systems in 1996

•  NetFlow is the defacto standard for acquiring IP operational data

•  Provides network and security monitoring, network planning, traffic analysis, and IP accounting

Network World Article – NetFlow Adoption on the raise http://www.networkworld.com/newsletters/nsm/2005/0314nsm1.html


Metering Process Export Process

IETF Scope

NetFlow v5, NetFlow v8, NetFlow v9,

IPFIX Capacity Planning Security Performance Analysis Visibility


Monitoring Feature

Export Protocol

Traditional Netflow

Netflow Version 5

Netflow Version 9 IPFIX

Flexible Netflow MMON ART

Need by AVC


Version Major Advantage Limits/Weaknesses

V5 Defines 18 exported fields Simple and compact format Most commonly used format

IPv4 only Fixed fields, fixed length fields only Single flow cache

V9 Template-based IPv6 flows transported in IPv4 packets MPLS and BGP nexthop supported Defines 104 fields, including L2 fields Reports flow direction

IPv6 flows transported in IPv4 packets Fixed length fields only Uses more memory Slower performance Single flow cache

Flexible NetFlow (FNF) Template-based flow format (built on V9 protocol) Supports flow monitors (discrete caches) Supports selectable key fields and IPv6 Supports NBAR data fields

Less common Requires more sophisticated platform to produce Requires more sophisticated system to consume

IP Flow Information Export (IPFIX) AKA NetFlow V10

Standardized – RFC 5101, 5102, 6313 Supports variable length fields, NBAR2 Can export flows via IPv4 and IPv6 packets

Even less common Only supported on a few Cisco platforms

NSEL (ASA only) Built on NetFlow v9 protocol State-based flow logging (context) Pre and Post NAT reporting

Missing many standard fields Limited support by collectors


NetFlow v9 160+ fields to choose from including IPv6 and payload sections


NetFlow EnabledDevice

Traffic

•Input Interface

•TOS byte (DSCP)

•Layer 3 protocol

•Destination port

•Source port

•Destination IP address

•Source IP address

•Input Interface

•TOS byte (DSCP)

•Layer 3 protocol

•Destination port

•Source port

•Destination IP address

•Source IP address

Create a flow from the packetattributes

…

152811000Address, ports…

Bytes/packetPacketsFlow Information

…

152811000Address, ports…

Bytes/packetPacketsFlow Information

NetFlow CacheInspect Packet

NetFlow Export Packets

Reporting

Cisco NetFlow Collector


•  Key fields are unique per flow record

•  Non-key fields are attributes or characteristics of a flow

•  If packet key fields are unique, new entry in flow record is created

•  Otherwise, update the non-key fields, i.e. packet count

Key Fields Packet 1

Source IP 1.1.1.1

Destination IP 2.2.2.2

Source port 23

Destination port 22078

Layer 3 Protocol TCP - 6

TOS Byte 0

Non-key Fields Packet 1

Length 1250

1 2 1 2

Key Fields Packet 2

Source IP 3.3.3.3


Source port 80

Destination port 22079


TOS Byte 0

Non-key Fields Packet 2

Length 519

Source IP Dest. IP Dest. I/F Protocol TOS … Pkts

1.1.1.1 2.2.2.2 E1 6 0 … 11000

Source IP Dest. IP Dest. I/F Protocol TOS … Pkts

3.3.3.3 4.4.4.4 E1 6 0 … 50

1.1.1.1 2.2.2.2 E1 6 0 … 11000

Netflow Cache After Packet 1 Netflow Cache After Packet 2


1. Create and update flows in NetFlow cache Srclf SrclPadd Dstlf DstlPadd Protocol TOS Flgs Pkts Src

Port Src Msk

Src AS

Dst Port

DstMsk

Dst AS NextHop Bytes/

Pkt Active Idle

Fa1/0 173.100.21.2 Fa0/0 10.0.227.12 11 80 10 11000 00A2 /24 5 00A2 /24 15 10.0.23.2 1528 1745 4

Fa1/0 173.100.3.2 Fa0/0 10.0.227.12 6 40 0 2491 15 /26 196 15 /24 15 10.0.23.2 740 41.5 1

Fa1/0 173.100.20.2 Fa0/0 10.0.227.12 11 80 10 10000 00A1 /24 180 00A1 /24 15 10.0.23.2 1428 1145.5 3

Fa1/0 173.100.6.2 Fa0/0 10.0.227.12 6 40 0 2210 19 /30 180 19 /24 15 10.0.23.2 1040 24.5 14

•  Inactive timer expired (15 sec is default) •  Active timer expired (30 min (1800 sec) is default) •  NetFlow cache is full (oldest flows are expired) •  RST or FIN TCP flag

2. Expiration

Srclf SrclPadd Dstlf DstlPadd Protocol TOS Flgs Pkts Src Port

Src Msk

Src AS

Dst Port

DstMsk

Dst AS NextHop Bytes/

Pkt Active Idle

Fa1/0 173.100.21.2 Fa0/0 10.0.227.12 11 80 10 11000 00A2 /24 5 00A2 /24 15 10.0.23.2 1528 1800 4

3. Aggregation 4. Export version

5. Transport protocol

ie: Protocol-port aggregation scheme becomes

Aggregated flows—export Version8 or 9 Export packet

Payload (flows)

Non-aggregated flows—export Version 5 or 9 Protocol Pkts SrcPort DstPort Bytes/Pkt

11 11000 00A2 00A2 1528

Hea

der

30 Flows per 1500 byte export packet


Source IP

Dest. IP

Source Port

Dest. Port Protocol TOS Input

I/F … Pkts

3.3.3.3 2.2.2.2 23 22078 6 0 E0 … 1100

Traffic Analysis Cache

Flow Monitor 1

Traffic

Non-Key Fields

Packets

Bytes

Timestamps

Next Hop Address

Security Analysis Cache

Flow Monitor 2

Key Fields Packet 1

Source IP 3.3.3.3


Source Port 23

Destination Port 22078


TOS Byte 0

Input Interface Ethernet 0

Key Fields Packet 1

Source IP 3.3.3.3


Input Interface Gi0/1

SYN Flag 0

Non-Key Fields

Packets

Timestamps

Source IP

Dest. IP Input I/F Flag … Pkts

3.3.3.3 2.2.2.2 Gi0/1 0 … 11000


•  The NetFlow v9 Exporter sends packets to Collectors

•  There are “Template FlowSets”, which define the following data formats by a set of Field Types and their lengths, and “Data FlowSets”, which are sets of actual statistics data

•  The NetFlow v9 Exporter sends Templates separately (and less frequently) than data sets

•  There are another sets of Template/Data FlowSets called “Option Template/Data FlowSets”, by which an Exporter can send a Collector meta-data related to NetFlow process and so on.

Packet Header

FlowSet Template

FlowSet Data

FlowSet Template

FlowSet Data

FlowSet Data

NetFlow v9

IPFIX

RFC3954


•  A single record per monitor •  Potentially multiple monitors per interface •  Potentially multiple exporters per monitor

Interface

Monitor “A” Monitor “B”

Record “X” Exporter “M”

Record “Y”

Exporter “N”

Monitor “C”

Exporter “M”

Record “Z”

12


Different Flow monitors for detecting different information:

SiSi SiSi

WAN DATA CENTER

CAMPUS

BRANCH

Security Flows • Protocol • Ports • IP Addresses • TCP Flags • Packet Section

Multicast Flows • Protocol • Ports • IP Subnets • Packet Replication

ISP Peering Flows • Dest. AS • Dest. Traffic Index • BGP Next Hop • DSCP

IP Flows • IP Subnets • Ports • Protocol • Interfaces • Egress/Ingress


Service Provider

Network Infrastructure Optimization and Planning

Peering Arrangements

Traffic Engineering

Accounting and Billing

Security Monitoring and Incident (DDoS) Detection

Data at ANY granularity to understand network use: who, what, where, when and how


Enterprise

Internet Access Monitoring

User Monitoring/Profiling

Application Monitoring

Billing for Departments

Security Monitoring and Incident (DDoS) Detection

Data at ANY granularity to understand network use: who, what, where, when and how


NetFlow (L2, L3, L4)

Application Id (Extracted Field)

QoS Classification QoS Treatment

Forwarding Status

Performance Metrics

DPI Metadata

QoS CEF

Performance Agent PfR

Performance Monitoring

Which applications are running in my

network?

Do applications get the right QoS treatment?

Do applications get the right QoS treatment?

Do applications get the right service

from the network?


IntegrationInterface

Source IP Address

Source Port

Destination Port

NetFlow ü  Monitors data in Layers 2 thru 4 ü  Determines applications by combination of

Port or Port/IP Addressed ü  Flow information who,

what, when, where

NBAR ü  Examines data from

Layers 3 thru 7 ü  Utilizes Layers 3 and 4

plus packet inspection for classification ü  Stateful inspection of

dynamic-port traffic ü  Packet and byte counts

Protocol

Link Layer Header

Deep Packet (Payload) Inspection

ToS NetFlow

NBAR

Destination IP Address

IP Header

TCP/UDP Header

Data Packet


Key Fields Packet #1

Source IP 10.1.1.1


Source Port 20457

Destination Port 23

Layer 3 protocol 6

TOS byte 0

Ingres Interface Ethernet 0

Src. IP Dest. IP Src. Port Dest. Port Layer 3 Prot. TOS Byte Ingress Intf.

10.1.1.1 173.194.34.134. 20457 80 6 0 Ethernet 0

Key Fields Packet #2

Source IP 10.1.1.1


Source Port 30307

Destination Port 80

Layer 3 protocol 6

TOS byte 0

Ingres Interface Ethernet 0

Src. IP Dest. IP Src. Port Dest. Port Layer 3 Prot. TOS Byte Ingress Intf. App Name Timestamps

Byttes Packets

10.1.1.1 173.194.34.134 20457 80 6 0 Ethernet 0 HTTP

10.1.1.1 72.163.4.161 30307 80 6 0 Ethernet 0 Youtube

NetFlow cache

News

15.0(1)M IOS XE 3.1S

flow record app_record! match ipv4 source address! match ipv4 destination address! match …..! match application name!!

First packet of a flow will create the Flow entry using the Key Fields” Remaining packets of this flow will only update statistics (bytes, counters, timestamps)


SrcIPadd DstIPadd TOS pkts bytes

10.1.0.5 172.16.10.19 0x00 1 64

10.1.0.5 172.16.0.20 0x00 10 800

10.1.0.95 172.16.10.19 0x00 200 16000

10.1.0.34 172.16.10.4 0x0 100 4500

10.1.0.121 172.16.10.4 0x00 1 64

10.1.0.333 172.16.10.2 0x00

367 23488

10.1.0.100 172.16.0.2 0x00

111 7104

10.1.0.121 172.16.10.21 0x00

5 350

10.1.0.34 172.16.10.2 0x00

35 200

10.1.0.95 172.16.10.2 0x00

30 200

# show flow monitor <monitor>

Top Talkers provide quick, easy, and granular traffic analysis by displaying a subset of flow monitor in real time

§  Top 4 IPv4 destinations sorted by number of bytes:

Switch# show flow monitor <monitor> aggregate ipv4 destination address sort counter bytes top 4

§  Top 5 sources of 1-packet flows: Switch# show flow monitor <monitor> cache filter counter packet 1 aggregate ipv4 source address sort highest flow packet top 5

enable users to select flows based on specific values for any fields

enable users to aggregate on a subset of the key and non-key fields

enable users to control how the displayed cache entries are sorted on any field and show in order or reverse order

DesIPadd flows bytes pkts

172.16.10.2 12 1358370 6708

172.16.10.19 2 44640 1116

172.16.10.20 2 44640 1116

172.16.10.4 1 22360 559

SrcIPadd flows bytes pkts

10.1.0.5 135 8640 135

10.1.0.100 100 6400 100

10.1.0.95 95 6080 95

10.1.0.121 80 5120 80

10.1.0.34 79 5056 79

Benefits and Applications Security

See if traffic patterns are consistent with a DoS or other undesirable behavior

Traffic load Identify heavily used parts of the network so you can redistribute load accordingly

Traffic analysis Baseline network traffic for capacity planning and network engineering

Granularity Flow information displayed per monitor and per interface (port or VLAN)


srcIf SrcIPadd DstIf DstIPadd TLL

Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 0

Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 10

Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 200

srcIf SrcIPadd DstIf DstIPadd bytes

Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 34346

Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 300

Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 1000

*MAR 29 2010 12:29:02.604 UTC: %HA_EM-6-LOG: my-ttl-applet: flow record with zero TTL

Netflow cache

Example I: Malformed Packets Detection & Reporting TTL = 0 triggers an EEM event

syslog message generated based on pre-configured policies

Attacker sending malformed pkts with TTL=0

Netflow cache

Example II : Anomaly Flow Detection and Mitigation

*Feb 18 01:24:30.455: %LINK-5-CHANGED: Interface FastEthernet 1/0, changed state to administratively down

NetFlow ED triggers policies to monitor flow rate. Typically, voice conversations are 64kbps

interface Fa1/0 is shut down when the flow rate exceeds 1Mbps

Compromised phone sending traffic with high rate

•  Quick Instant, on board traffic anomaly detection and reaction

•  Detailed Granular view of flow info enables a wide range of applications

•  Flexible Custom policies written in CLI or TCL

•  Event-driven NF event detector triggers policies locally on network devices instead


Cisco Network

StealthWatch FlowCollector

StealthWatch Management

Console

NetFlow

StealthWatch FlowSensor

StealthWatch FlowSensor VE

Users/Devices

Cisco ISE

NetFlow

StealthWatch FlowReplicator

Other tools/collectors

NBAR NSEL

StealthWatch Labs Information Center

Reputation Feed (Optional)


Who is 10.10.101.89?

Policy Start Active Time

Alarm Source Source Host Groups

Target Details

Desktops & Trusted Wireless

Jan 3, 2013 Suspect Data Loss

10.10.101.89 Atlanta, Desktops

Multiple Hosts Observed 5.33G bytes. Policy maximum allows up

to 500M bytes.


Policy Start Active Time

Alarm Source Source Host Groups

Source User Name

Device Type

Target

Desktops & Trusted Wireless

Jan 3, 2013 Suspect Data Loss

10.10.101.89 Atlanta, Desktops

John Chambers Apple-iPad Multiple Hosts


•  Flow Action field can provide additional context

•  State-based NSEL reporting is taken into consideration in StealthWatch’s behavioral analysis

•  Concern Index points accumulated for Flow Denied events

•  NAT stitching


•  NetFlow is a mature Cisco IOS feature (in Cisco IOS since 1996)

•  NetFlow provides input for accounting, performance, security, application visibility, and billing applications

•  Cisco standardizes on NetFlow/IPFIX: NetFlow 9 and Flexible NetFlow consistency across many devices, including in hardware now

NetFlow v9 eases the exporting of additional fields

Flexible NetFlow is a major enhancement

•  NetFlow is deployable today!

•  NetFlow has IETF and industry leadership

26


TNF = Traditional (Non-Flexible) Netflow and FNF = Flexible Netflow

Cisco 800 Series

Cisco 1800 1900

Series

Cisco 3800 3900

Series

Cisco 2800 2900

Series

Catalyst 6K Sup2T

Cisco 12000 Series ASIC Cisco 7x00

Series

Cisco 4500 Sup7

Cisco IOS Software Releases

Enterprise & aggregation/edge

Cisco IOS Software Release 12.2S

Cisco 7200/ 7300 Series

Access

Core Release 12.0S/IOS-XR

ASR9000 CRS-1 CRS-3 ASIC

ASR1000 QFP based

DataCenter

Nexus 7000

Nexus 1000V

ASR1000 QFP based

Cat 6K Sup2T

FNF FNF

FNF FNF FNF FNF

FNF

FNF

FNF

FNF FNF

FNF FNF

Cisco 7600 Series

FNF

Catalyst 6K < Sup2T

NO FNF support Hardware limitation

Catalyst 29xx Catalyst 3750

FNF TNF TNF

TNF TNF

TNF TNF TNF TNF

TNF TNF

Cisco 4500 <= Sup5

TNF

NO FNF support Hardware limitation

TNF

Catalyst 3750X Next Gen Cat3K

FNF


•  System Scalability. Up to ~512K (with 99% utilization efficiency) cached flows for Forwarding Engine. Per direction, per DFC => 13 million flow entries

•  Sampled NetFlow. Effective hardware-based sampling to improve and preserve NetFlow table utilization

•  Ingress & Egress NetFlow. Useful for example to track packets de-capsulated after tunneling mechanisms

•  Per Interface or Sub-Interface activation

•  Bridged NetFlow. Capability of creating and tracking bridged IP flows

•  TCP Flags are now exported as part of the flow information. Very useful to understand TCP flow directions and to detect denial of service attacks

•  IPv4, IPv6 and Layer 2 Flows support

•  Export version 5 (the most used) and export version 9 (the most flexible) are both supported

•  VRF aware export

•  Hitless ISSU

For Your Reference


•  System Scalability. Up to ~1M cached flows for Forwarding Engine, in hardware

•  Sampled NetFlow. Effective hardware-based sampling

•  Ingress & Egress NetFlow.



•  Hardware based Export hardware acceleration for export


•  IPv4, Application layer (NBAR) Flow support in XE 3.1.1S


•  Hitless ISSU in IOS XE 3.2.0S

For Your Reference


•  System Scalability. Up to ~500K (with 95% utilization efficiency) cached flows for Forwarding Engine


•  Egress NetFlow. Useful for example to track packets de-capsulated after tunneling mechanisms







•  Hitless ISSU and process restartability

•  Flexible NetFlow CLI look & feel

For Your Reference


•  System Scalability. Up to ~128K cached flows


•  Ingress & Egress NetFlow, IPv4, IPv6 and Layer 2 Flows support

•  Per Interface, Sub-Interface or VLAN activation




•  VRF aware export, Hitless ISSU

•  Note: on SUP2, Netflow Lite: packet sampling + no caching, exported with IPFIX

For Your Reference


3KX-SM-10G Service Module •  System Scalability. Up to ~32K cached flows


•  Ingress & Egress NetFlow

•  Per Interface, Sub-Interface or Vlan activation




•  Export version 9


•  3KX-SM-10G

For Your Reference

Both Full flow accounting and sampled NetFlow accounting are supported


•  System Scalability. Up to ~1M cached flows per Line Card

•  Sampled NetFlow. Effective hardware-based sampling from 1:1 to 1:64K – 100kpps/LC (ingress + egress)


•  Per Interface or Sub-Interface support

•  TCP Flags Very useful to understand TCP flow directions and to detect denial of service attacks

•  Export version 9 – 50K Flows/s export per LC

•  IPv4, IPv6, MPLS Flows support


•  Hitless ISSU and process restartability

•  Flexible NetFlow Pre-defined aggregation only

For Your Reference


NetFlow in Hardware on Catalyst 3850 •  System Scalability NetFlow V4 / V6

WS-C3850-24: up to 24k /12k cached Flows WS-C3850-48: up to 48k / 24k cached Flows



•  Per Interface activation, NetFlow support on all ports


•  Key Fields: IP address, MAC address, TOS, TCP Flags, and VLan


•  Multicast Flow support


•  Dynamic top talker support

•  If stacked, Individual stack members export their own NetFlow records directly to the Collector

•  15.0(1)EX

New Platform

Both Full flow accounting and sampled NetFlow accounting are supported Feature se: IP Base or IP Services)


NetFlow in Hardware on Catalyst 2960X/2960-XR •  NetFlow-Lite is natively supported on all downlink and uplink ports

•  NetFlow-Lite uses sampled flows to provide statistics for network traffic accounting, network monitoring and network planning. A flow is created using a flow record, which define the unique keys of the flow. NetFlow-Lite provides valuable information about network users and applications, peak usage times, and traffic routing.

•  NetFlow-Lite is supported on Mixed Stack (stack of Cisco Catalyst 2960-S and 2960-X/XR series switches). But NetFlow-Lite monitor can be attached only on Cisco Catalyst 2960-X/XR Series ports

•  Only NetFlow Version 9 is supported for NetFlow exporter using the export-protocol command option.

•  16K NetFlow-Lite Flows are supported

•  Only Ingress flow monitors are supported

•  The Flow monitors can be attached to physical interfaces and VLAN interfaces

•  The Flow monitor can’t be attached to logical interfaces like EtherChannel or Layer2 VLANs.

New Platform

Sampled NetFlow accounting Feature se: LAN Base or IP Lite


NetFlow in Hardware on cat 5760 •  System Scalability: NetFlow v4/V6 Up to ~72K / 36k cached

flows



•  Per Interface , SSID activation


•  Key Fields: IP address, MAC address, TOS, TCP Flags, security fields and VLan


•  Multicast Flow support


•  Dynamic top talker support

•  15.0(1)EX

• 

New Platform


•  NetFlow http://www.cisco.com/go/netflow

•  Cisco network accounting services Comparison of Cisco NetFlow versus other available accounting technologies http://www.cisco.com/warp/public/cc/pd/iosw/prodlit/nwact_wp.htm

•  Cisco IT case study http://business.cisco.com/prod/tree.taf%3Fasset_id=106882&IT=104252&public_view=true&kbns=1.html

•  A complete white paper http://www.cisco.com/univercd/cc/td/doc/cisintwk/intsolns/netflsol/ nfwhite.htm

•  NetFlow product manager: Jean Charles Griviaud [email protected]

37

Thank you.

NetFlowftp.cisco.cz/.../2013-10-03-NetFlow-JiriTesar.pdf2013/10/03 · • The NetFlow v9 Exporter...

Documents

Transcript of NetFlowftp.cisco.cz/.../2013-10-03-NetFlow-JiriTesar.pdf2013/10/03 · • The NetFlow v9 Exporter...