Dissecting Firepower-d2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/BRKSEC-3455.pdf · Dissecting...

Dissecting Firepower-NGFW(FTD) “Installation & Troubleshooting”

Veronika Klauzova

BRKSEC-3455

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/clus17/#BRKSEC-3455Cisco Spark spaces will be available until July 3, 2017.

Haitham Jaradat

John Groetzinger


Related sessions - You don’t want to miss at #CLUS

4BRKSEC-3455

TECSEC-3301

Firepower Data-Path

Troubleshooting

John Groetzinger

BRKSEC-2020

Firepower NGFW

Deployment in the Data

Center and Enterprise

Network Edge using

FTD

Steven Chimes

BRKSEC-2050

Firepower NGFW

Internet Edge

Deployment Scenarios

Jeff Fanelli

For YourReference


Related sessions - You don’t want to miss at #CLUS

5BRKSEC-3455

TECSEC-2004

Troubleshooting FTD

like a TAC Engineer

Ben Ritter

Kevin Klous

BRKSEC-3035

Firepower Platform

Deep Dive

Andrew Ossipov

BRKSEC-3020

Troubleshooting ASA

Firewalls

Kevin Klous

For YourReference


Your presenter throughout FTD journey

• Firepower TAC engineer

6BRKSEC-3455

Veronika Klauzova

• Originally from

• Working in

• Slavic countries accent

• Introduction

• Hardware & Software review

• Installation and Configuration

• Device registration troubles

• FTD Data-Flow: life of a packet

• Troubleshooting & Tools

• Conclusion

Agenda


Abstract-Review

• The session will cover both operational and maintenance aspects of all relevant Firepower-NGFW functions from “Installation” to “Operation” to “Troubleshooting” with a focus on interactive demonstration of the detailed topics.

• Upon successful completion of this session, the attendee will be able to:

• describe the FTD system architecture

• describe packet flow processing

• perform installation and configuration of FirePOWER Threat Defense(FTD)

• verify and troubleshoot traffic flows traversing FTD

8BRKSEC-3455

For YourReference


All content and demos are based on the following

• Firepower 4100 series system

• FXOS Version 2.1(1.77)

• Firepower Threat Defense 6.2.0.2 version (Released in May 2017)

• Firepower Management Center 6.2.0.2 version (Released in May 2017)

9BRKSEC-3455

Hardware & Software Review


NGFW evolution

BRKSEC-3455 11

LTRSEC-1000

FTD Deployment Hands-on-lab

Dax Mickelson

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 12BRKSEC-3455

What platforms can run FTD Software



Platform FTD Support

ASA 5500X-Series (5506X-5555X with SSD) Yes

Firepower 4100 series Yes



Virtual options (VMware, KVM, AWS, Azure) Yes

Cisco ISR 4000/ISR-G2 (UCS-E module) Yes

For YourReference


Firepower 4100 – closer look

16BRKSEC-3455

Front view

Rear view

PowerConsole

MGMT

8 x optic SFP+ ports

2 x 2.5” SSD Bays

2 x Power Supply Module Bays6 x Hot-Swap Fans units

2x optional NetMods


Firepower 8350 – do not run FTD software

18BRKSEC-3455


Firepower Chassis Manager

19BRKSEC-3455


Firepower Management Center

20BRKSEC-3455


Firepower Device Manager


Firepower Threat Defense

22BRKSEC-3455

DETECTION ENGINE / Snort

DATA-PATH / LINA

Packet Data Transport System (PDTS)

FXOS

FTD CLI modes


FTD CLI modes

There are three CLIs while dealing with a ftd deployment:

• FXOS CLI

• CLISH

• LINA CLI

Moving between different CLI‘s:

24BRKSEC-3455

firepower#

>

Firepower-module1>

connect ftd

system support diagnostic-cli

CTRL + a, d

exit

FXOS -> CLISH

CLISH -> LINA

LINA -> CLISH

CLISH -> FXOS

> expert $ sudo su #


Firepower Threat Defense – CLI MODES

25BRKSEC-3455

> expert $ sudo su #

> system support diagnostic-cli

firepower#

firepower> enable

>

CTRL + a, d

Firepower-module1> connect ftd


Converged FTD CLISH

• Available over SSH on data and management interface/s

• No switching back and forth between FP and ASA sub-modes

26BRKSEC-3455

> system support diagnostic-cli

firepower> enable

firepower# show cpu

Ctrl + a + d

> show cpu

> show cpu system

Linux 3.10.62-ltsi-WR6.0.0.27_standard (ftd.cisco.com) 02/07/17 _x86_64_

Time CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle

14:32:43 all 20.46 0.00 0.19 0.00 0.00 0.00 0.00 0.00 0.00 79.35

> show cpu

CPU utilization for 5 seconds = 0%; 1 minute: 0%; 5 minutes: 0%>

BEFORE 6.1

6.1+

Installation and Configuration


Preparing Firepower 4100 for an installation

28BRKSEC-3455

KSEC-FPR4100-2-A# scope fabric-interconnect a

KSEC-FPR4100-2-A /fabric-interconnect # set out-of-band gw 10.62.148.1 ip 10.62.148.38

netmask 255.255.255.0

Warning: When committed, this change may disconnect the current CLI session

KSEC-FPR4100-2-A /fabric-interconnect* #

KSEC-FPR4100-2-A /fabric-interconnect* # commit

KSEC-FPR4100-2-A /fabric-interconnect # exit

Setup Management IP address

Verify basic connectivity

KSEC-FPR4100-2-A# connect local-mgmt

KSEC-FPR4100-2-A(local-mgmt)# ping cisco.com

ping: unknown host cisco.com

KSEC-FPR4100-2-A(local-mgmt)# ping 72.163.4.161

64 bytes from 72.163.4.161: icmp_seq=1 ttl=231 time=156 ms

64 bytes from 72.163.4.161: icmp_seq=2 ttl=231 time=156 ms



29BRKSEC-3455

KSEC-FPR4100-2-A# scope system

KSEC-FPR4100-2-A /system # scope services

KSEC-FPR4100-2-A /system/services # show dns

KSEC-FPR4100-2-A /system/services #

Verify DNS configuration settings in FXOS CLI

Verify and configure DNS settings from FCM

KSEC-FPR4100-2-A /system/services # show dns

Domain Name Servers:

IP Address: 173.38.200.100

IP Address: 8.8.8.8



30BRKSEC-3455

Verify and configure Network Time Synchronization (NTP)

KSEC-FPR4100-2-A# show clock

Tue May 16 16:10:42 UTC 2017

KSEC-FPR4100-2-A# show ntp-overall-status

NTP Overall Time-Sync Status: Time Synchronized


Brief installation steps on Firepower 4100 series

31BRKSEC-3455

Add FTD to Firepower Management Center

Upgrade the supervisor (FXOS) software bundle

Configure FTD Management and Data Interfaces

Install FTD application image

Provision FTD Settings (mode, IP settings, FMC info)


Upload new supervisor (FXOS) software to FCM


Upgrade the supervisor (FXOS) software bundle


Configure FTD Data & Management Interfaces

34BRKSEC-3455


FTD logical device creation

35BRKSEC-3455


FTD installation on 4100(1)For YourReference


FTD installation on 4100(2)For YourReference


FTD installation on 4100 (working hard)For YourReference


FTD Installation „Local Console“ monitoringKSEC-FPR4100-2-A /ssa/slot # connect module 1 console

Telnet escape character is '~'.

Trying 127.5.1.1...

Connected to 127.5.1.1.

Escape character is '~'.

CISCO Serial Over LAN:

Close Network Connection to Exit [ OK ]

Executing S47install_default_sandbox_EO.pl [ OK ]

Executing S50install-remediation-modules [ OK ]

Executing S51install_health_policy.pl [ OK ]

Executing S52install_system_policy.pl [ OK ]

Executing S53change_reconciliation_baseline.pl [ OK ]

Executing S70remove_casuser.pl [ OK ]

Executing S70update_sensor_objects.sh [ OK ]

Executing S85patch_history-init [ OK ]

Executing S90banner-init [ OK ]

Executing S96grow_var.sh [ OK ]

Executing S96install_vmware_tools.pl [ OK ]

(output truncated)


FTD installation on 4100 (finished)


Device registration

Having trouble registering device?


Device Registration

44BRKSEC-3455

FMC FTDEncrypted

Tunnel

192.168.0.0/24


Device Registration

45BRKSEC-3455

FMC FTDEncrypted

Tunnel

192.168.0.0/24 10.10.10.0/24


Device Registration

46BRKSEC-3455

FMC FTDControl channel

Events channel

Encrypted

Tunnel

• Connection Events

• IPS Events

• Malware Events

• File Events

• SSL Events

• Keep-Alive messages


Device Registration

47BRKSEC-3455


Events channel

Encrypted

Tunnel

root@fmc-2:/# netstat -lnta | grep 8305

ftd-4100-2:/# netstat -lnta | grep 8305

tcp 0 0 10.62.148.90:8305 10.62.148.85:60563 ESTABLISHED

tcp 0 0 10.62.148.85:60563 10.62.148.90:8305 ESTABLISHED

tcp 0 0 10.62.148.85:54849 10.62.148.90:8305 ESTABLISHED

tcp 0 0 10.62.148.90:8305 10.62.148.85:54849 ESTABLISHED


Device Registration

48BRKSEC-3455


Events channel

Encrypted

Tunnel

TCP 8305

> configure manager add <FMC IP address> <shared

key> <NAT ID>


Trouble 1: FTD has DHCP IP address what now?

49BRKSEC-3455 49

FTD – add FMC details

Important Note:

NGFW will initiate Registration communication!

FMC - Add FTD into FMC WebUIFMC

mgmt0

eth0

MGMT interface with DHCP IP address

MGMT interface with static IP address

> configure manager addFTD

• Add manager/FMC IP address in CLI

<FMC static IP address>

• Shared Key (needs to match with FMC side)

<shared key> <NAT ID>

• NAT ID (needs to match with FMC side)

1. Keep Host entry EMPTY

2. Registration/Shared Key

3. ACP

4. License

5. NAT ID (required when host entry not used)


Trouble 2: FMC has DHCP IP address what now?

50BRKSEC-3455 50

FTD

• Add manager/FMC IP address in CLI

• Shared Key (needs to match with FMC side)

• NAT ID (needs to match with FMC side)

Important Note:

FMC will initiate Registration communication!

FMC - Add FTD into FMC WebUI

1. Keep Host entry (IP address of FTD)

2. Registration/Shared Key

3. ACP

4. License

5. NAT ID (optional)

FMC

mgmt0

eth0

MGMT interface with static IP address

MGMT interface with DHCP IP address

> configure manager add DONOTRESOLVE <shared

key> <NAT ID optional>FTD


Device registration “headache” error message

51BRKSEC-3455

"Could not establish a connection with sensor. Make sure the registration keys match, that the software versions are compatible, and that the network is not blocking the connection."


Device registration trouble #3

52BRKSEC-3455

FTD

> configure manager add

FMC

10.62.148.92

Manager successfully configured.

Please make note of reg_key as this will be required while adding Device in FMC.

>

key cisco123



53BRKSEC-3455

FTD FMC

configure manager add <FMC IP> <REG KEY> <NAT ID>

> configure manager add 10.62.148.92 key cisco123


Please make note of reg_key as this will be

required while adding Device in FMC.

>

CORRECT COMMAND SYNTAX



54BRKSEC-3455

FTD

> show managers

Host : 10.62.148.90

Registration Key : ****

Registration : pending

RPC Status :

>

FMC



55BRKSEC-3455

FTD

# tail -n 14 /etc/sf/sftunnel.conf

host 10.62.148.90;

ip 10.62.148.90;

reg_key cisco12345;

FMC

#tail –f /ngfw/var/log/messages

May 28 18:04:57 fmc-vklauzov SF-IMS[2769]: [3315]

sftunneld:sf_ssl[WARN] Accept: Failed to authenticate peer

'10.62.148.90’



56BRKSEC-3455

FTD

#tail –f /ngfw/var/log/messagesMay 28 18:04:57 fmc-vklauzov SF-IMS[2769]: [3315] sftunneld:sf_ssl[WARN]

Accept: Failed to authenticate peer '10.62.148.90’

# tail -n 14 /etc/sf/sftunnel.conf

host 10.62.148.90;

ip 10.62.148.90;

reg_key cisco12345;

FMC


Device registration trouble 5

57BRKSEC-3455

FTD

> configure manager add 10.62.148.92 cisco123



>

FMC

Internet

For YourReference



58BRKSEC-3455

FTD

> configure manager add 10.62.148.92 cisco123



>

FMC

Internet

is full of NAT devices

For YourReference



59BRKSEC-3455

FTD

# tail -f /ngfw/var/log/messages | grep sftunnnel

(no new logs for encrypted communication channel used for registration)

#

> capture-traffic

Please choose domain to capture traffic from:

0 - management0

Selection? 0

Options: -n port 8305

18:36:47.642198 IP 10.62.148.90.54216 > 10.62.148.85.8305: Flags [S]

18:36:47.642218 IP 10.62.148.85.8305 > 10.62.148.90.54216: Flags [R.]

For YourReference



60BRKSEC-3455

FTD

# tail -f /ngfw/var/log/messages | grep sftunnnel

(no new logs for encrypted communication channel used for registration)

#

> capture-traffic


0 - management0

Selection? 0


18:36:47.642198 IP 10.62.148.90.54216 > 10.62.148.85.8305: Flags [S]

18:36:47.642218 IP 10.62.148.85.8305 > 10.62.148.90.54216: Flags [R.]

> pmtool status

sftunnel (system) - User Disabled

Command: /ngfw/usr/local/sf/bin/sftunnel -d -f

/etc/sf/sftunnel.conf

PID File: /ngfw/var/sf/run/sftunnel.pid

Enable File: /ngfw/etc/sf/sftunnel.conf

For YourReference



61BRKSEC-3455

FTD

> show network

==================[ management0 ]===================

State : Enabled

Channels : Management & Events

MTU : 9000

----------------------[ IPv4 ]----------------------

Address : 10.62.148.85

FMC

# ifconfig eth0 | grep MTU

UP BROADCAST RUNNING MULTICAST MTU:1500

For YourReference



62BRKSEC-3455

FTD

> show network

==================[ management0 ]===================

State : Enabled


MTU : 9000

----------------------[ IPv4 ]----------------------

Address : 10.62.148.85

FMC



For YourReference



63BRKSEC-3455

FTD

> show network

==================[ management0 ]===================

State : Enabled


MTU : 9000

----------------------[ IPv4 ]----------------------

Address : 10.62.148.85

FMC



For YourReference



64BRKSEC-3455

FTD

> capture-traffic


0 - management0

1 - Router

Selection? 0


FMC

# tcpdump -i eth0 port 8305 -n

IP 10.62.148.85.38530 > 10.62.148.90.8305: Flags [S],

seq 2011406652, win 17920, options [mss

8960,sackOK,TS val 73165282 ecr 0,nop,wscale 7],

length 0

IP 10.62.148.90.53985 > 10.62.148.85.8305: Flags [S],



length 0

IP 10.62.148.85.49249 > 10.62.148.90.8305: Flags [S],



length 0

For YourReference



65BRKSEC-3455

FTD

> capture-traffic


0 - management0

1 - Router

Selection? 0


FMC

# tcpdump -i eth0 port 8305 -n

IP 10.62.148.85.38530 > 10.62.148.90.8305: Flags [S],



length 0

IP 10.62.148.90.53985 > 10.62.148.85.8305: Flags [S],



length 0

IP 10.62.148.85.49249 > 10.62.148.90.8305: Flags [S],



length 0

For YourReference



66BRKSEC-3455

FTD

# scp 10.62.148.85-1e149ee0-3f8f-11e7-b625-b451664b5209-troubleshoot.tar.gz [email protected]:/var/tmp/

10.62.148.85-1e149ee0-3f8f-11e7-b625-b451664b5209-

troubleshoot.tar.gz 1% 3MB 1KB/s 01:01

For YourReference



67BRKSEC-3455

FTD

# scp 10.62.148.85-1e149ee0-3f8f-11e7-b625-b451664b5209-troubleshoot.tar.gz [email protected]:/var/tmp/

10.62.148.85-1e149ee0-3f8f-11e7-b625-b451664b5209-

troubleshoot.tar.gz 1% 3MB 1KB/s 01:01

For YourReference


Device-RegistrationCommon-Fail-Scenarios Summary

68BRKSEC-3455

1 Invalid Syntax 6 Low bandwidth between FMC and

FTD

2 Mismatch Between Keys 7 Process down

3 NAT ID not configured 8 MTU changes

4 FTD has DHCP IP address

what now?

9 Blocked TCP 8305 port on

network

5 FMC has DHCP IP address

what now?

10 NAT ID mismatch

For YourReference

FTD Data-Flow: life of a packet


Firepower 4100 architecture overview

Security Engine (FTD)

Smart NIC

Internal Switch Fabric

Internal NM NM 1 NM 2


Firepower 4100 architecture overview

Data-Path

Detection Engine / Snort

PDTS

FXOS

Security Engine (FTD)


Packet-Flow

Data-Path / LINA

RX

L3, L2

hopsPre-Filter L3/L4

ACL

Egress

Interface NAT

TX

ALGchecks

Ingress

Interface

Existing

Conn


PDTSYES

NO

DAQ

VPN DecryptQoS, VPN Encrypt


Packet-Flow

Data-Path / LINA

RX

L3, L2


ACL

Egress

Interface NAT

TX

ALGchecks

Ingress

Interface

Existing

Conn


PDTSYES

NO

DAQ


Lina rule-id matched


Packet-Flow

PDTS

Data-Path / LINA


SI (DNS/URL), Identity

SI (IP) SSL L7 ACL File/AMP

Snort Verdict (trust, fast-forward, deny/blacklist)

IPS


Packet-Flow

PDTS

Data-Path / LINA




Snort Verdict

IPS


Data-Path

Data-Path / LINA

RX

L3, L2


ACL

Egress

Interface NAT

TX

ALGchecks

Ingress

Interface

Existing

Conn


PDTSYES

NO

DAQ



Data-Path – Do we receive any packets?

firepower# sh int eth 1/7

Interface Ethernet1/7 "INSIDE", is up, line protocol is up

MAC address 5897.bdb9.73ee, MTU 1500

IP address 172.16.1.1, subnet mask 255.255.255.0

Traffic Statistics for "INSIDE":

180 packets input, 14853 bytes

155 packets output, 12628 bytes

25 packets dropped

1 minute input rate 1 pkts/sec, 94 bytes/sec

Number of packets dropped in ASP ‘show asp drop‘

BRKSEC-3455 77


Data-Path – Do we receive any packets?

DATA-PATH

> show capture in

1: 15:52:55.249834 172.16.1.56 > 20.20.20.33: icmp: echo request

2: 15:52:55.250643 20.20.20.33 > 172.16.1.56: icmp: echo reply

BRKSEC-3455 78


Data-Path

Data-Path / LINA

RX

L3, L2


ACL

Egress

Interface NAT

TX

ALGchecks

Ingress

Interface

Existing

Conn


PDTSYES

NO

DAQ



Data-Path – Existing Connection

• LINA part checks whether the connection belongs to existing flow or not

• If packet is part of already established flow, then appliance skip basics checks and process the packet in Fast-Path – and continue with checks at DAQ level

80BRKSEC-3455

firepower# show cap in2 packet-number 46 trace detail46: 19:28:20.056012 0050.56b6.0b33 5897.bdb9.73ee 0x8100 Length: 58

802.1Q vlan#208 P0 172.16.2.13.49182 > 20.20.20.11.80: . [tcp sum ok] 2790183968:2790183968(0) ack

1176461110 win 231 (DF) (ttl 128, id 16898)

...

Type: FLOW-LOOKUP

Found flow with id 34550, using existing flow

firepower# sh logging | include 34550

%ASA-6-302013: Built inbound TCP connection 34550 for in2:172.16.2.13/49182

(172.16.2.13/49182) to OUTSIDE:20.20.20.11/80 (20.20.20.11/80)

%ASA-6-302014: Teardown TCP connection 34550 for in2:172.16.2.13/49182 to

OUTSIDE:20.20.20.11/80 duration 0:00:28 bytes 1073752075 Flow closed by inspection

firepower#

Unique Connection ID


Data-Path

Data-Path / LINA

RX

L3, L2


ACL

Egress

Interface NAT

TX

ALGchecks

Ingress

Interface

Existing

Conn


PDTSYES

NO

DAQ



Data-Path – Egress Interface

• Determination of Egress Interface

• Routing table / route lookup – ‘in’ entries of the ASP routing table will be checked to determine the egress interface

• UN-NAT (destination NAT) – egress interface will be choosen based on NAT rule

82BRKSEC-3455

firepower# show asp table routing

firepower# show capture <name> packet-number 10 trace detail

firepower# packet-tracer

Data-Path / LINA CLI:


Data-Path

Data-Path / LINA

RX

L3, L2


ACL

Egress

Interface NAT

TX

ALGchecks

Ingress

Interface

Existing

Conn


PDTSYES

NO

DAQ



Data-Path – Pre-Filter Policy

Flow-offload feature

• Help to offload the flows to Smart NIC for faster throughput and low latency

• Decision to offload is made by DATA-PATH (in future release also Snort would do this)

• Flow state tracking done by DATA-PATH

• Supported in Clustering deployments, but no offload mode compatibility checks

• Supported in HA failover mode – offload flags are replicated to standby

Motivation:

• Data center FTD deployments with FAT a.k.a. Elephant Flows

• Latency issues in current data plane processing due to x86 CPU complex involvement

84BRKSEC-3455



Use cases

• High performance computing research sites

• High frequency trading

• GRE tunneled packets

Configuration

• Enabled by default on FTD (no GUI option to enable/disable feature)

• Flows that match pre-filter policy rule with Fast-Path action or Access Control Policy rule with TRUST action will be selected for flow offload

85BRKSEC-3455



• Limitations 6.1 release

• Flows processed by Detection-Engine/Snort cannot be offloaded, only Data-Path flows

• Flow offload not supported for FTD when interfaces are configured as inline-set

• DATA-PATH

• Handle decisions to offload based on policies setup by user

• Handle connection establishment and tear-down of offloaded flows

86BRKSEC-3455



87BRKSEC-3455

Actions

• Analyze: sends traffic for inspection to Snort

• Block: drops the traffic

• Fastpath: allows traffic and bypass further inspection,

process the rule in hardware, offload the traffic


Data-path policy vs. Snort policy

• Distributed evaluation of policy between LINA and SNORT

88BRKSEC-3455

Access-control policy

Pre-filter policy


Data-path policy vs. Snort policy

• AC rules that are evaluated by Snort are pushed down to LINA as PERMITACL rules

• Pre-filter rules are presented as Global ACL’s to LINA

89BRKSEC-3455

Permit ACL (appID, URL, User)

Global ACL (5-tuple)

Outer-headers packet

Inner-headers packet


Data-path / LINA “backend” ACL’s

• New type of ACL (Advanced ACL) is introduced for Access control

• Permit/Trust/Deny actions (within show access-list cmd)

• Permit means that packet is punted to Snort

• Trust means to skip Snort/Detection engine checks

• Lina can send start and end of flow events and Snort sends them to FMC

• Lina rule-id uniquely identify a rule and sends to Snort to perform NGFW policy evaluation

90BRKSEC-3455



91BRKSEC-3455

firepower# show running-config access-l | exclude remark

access-list CSM_FW_ACL_ advanced trust icmp any any rule-id 268434442 event-log both

access-list CSM_FW_ACL_ advanced trust tcp any any eq ftp rule-id 268434444 event-log

both

This is example of configuration that triggers flow offload!


92BRKSEC-3455

Advanced Snort / FirePOWERSI (DNS/URL), Identity

DETECTION ENGINE

DATA-PATH

Packet Data Transport System (PDTS) & DAQ

L7 ACL File/AMP


IPSSSLSI (IP)

L3/L4 ACL

NOIngress

Interface

Egress

Interface NAT TXALGchecks

RX

QoS, VPN Encrypt

L3, L2

hops

Existing

ConnPre-Filter

YES

VPN Decrypt

SMART NIC firepower# show flow-offload flow

2 in use, 2 most used, 16% offloaded

TCP intfc 106 src 20.20.20.11:80 dest 172.16.2.14:49191, timestamp

2265924877, packets 191614, bytes 264712022

TCP vlan 208 intfc 107 src 172.16.2.14:49191 dest 20.20.20.11:80, timestamp

2265924879, packets 26301, bytes 1788781

Traffic that matches pre-filter rule with FAST-PATH Action

Will be offloaded to Hardware



93BRKSEC-3455

Verify that flow-offload is enabled

Clear connection table in hardware / flow offloaded flows

firepower# clear flow-offload flow all

This command will not remove connection from DATA-PATH, you have to run clear conn command to do so.

firepower# show flow-offload info

Current running state : Disabled

User configured state : Enabled

For YourReference



94BRKSEC-3455

BRKSEC-3455

%ASA-6-805001: Offloaded TCP Flow for connection 34892 from

in2:172.16.2.14/49193 (172.16.2.14/49193) to OUTSIDE:20.20.20.11/80

(20.20.20.11/80)

%ASA-6-805001: Offloaded TCP Flow for connection 34892 from

OUTSIDE:20.20.20.11/80 (20.20.20.11/80) to in2:172.16.2.14/49193

(172.16.2.14/49193)

Syslog message when flow is offloaded and no longer offloaded

%ASA-6-805002: TCP Flow is no longer offloaded for connection 34892 from

in2:172.16.2.14/49193 (172.16.2.14/49193) to OUTSIDE:20.20.20.11/80

(20.20.20.11/80)

%ASA-6-805002: TCP Flow is no longer offloaded for connection 34892 from

OUTSIDE:20.20.20.11/80 (20.20.20.11/80) to in2:172.16.2.14/49193

(172.16.2.14/49193)

For YourReference


Data-Path

Data-Path / LINA

RX

L3, L2


ACL

Egress

Interface NAT

TX

ALGchecks

Ingress

Interface

Existing

Conn


PDTSYES

NO

DAQ



Access Control rule actions

• Allow – allow packet/s to go through further IPS/File policy evaluation (if configured)

• Trust – push traffic through hardware (Fast-Path traffic), no further Snort checks needed


Data-Path – L3/L4 ACL

97BRKSEC-3455

FMC

Data-Path

5-TUPLE

firepower# show access-list | i icmp

access-list CSM_FW_ACL_ line 9 remark rule-id 268441864: L7 RULE: icmp traffic

access-list CSM_FW_ACL_ line 10 advanced permit icmp any any rule-id 268441864

(hitcnt=335) 0xa2dc10fa

root@ftd:/var/sf/detection_engines/ae4faffe-d1b2-11e6-8ea4-817d227fa40c# cat ngfw.rules | grep 268441864

268441864 fastpath any any any any any any any 1 (log dcforward both)

FirePOWER


Data-Path – L3/L4 ACL

98BRKSEC-3455

FMC

Data-Path

5-TUPLE

firepower# show access-list | i icmp

access-list CSM_FW_ACL_ line 9 remark rule-id 268441864: L7 RULE: icmp traffic

access-list CSM_FW_ACL_ line 10 advanced permit icmp any any rule-id 268441864

(hitcnt=335) 0xa2dc10fa

Why AC rule with 5-tuple information is not marked as TRUST flow?


Packet-Flow

PDTS

Data-Path / LINA




Snort Verdict

IPS


Detection engine / Snort – Security Intelligence

• Ability to block dangerous / malicious traffic aka “bad guys”

• SI feed is updated by Cisco TALOS team periodically

• SI whitelist is intentionally processed by rest of the ACP rules

• 2 default SI Lists: Global Whitelist and Blacklist

100BRKSEC-3455


User Story #1 – Security Intelligence (1)

• Problem description: URL website blocked

101BRKSEC-3455

Analysis -> Connections -> Security Intelligence Events


User Story #1 – Security Intelligence (2)

102BRKSEC-3455

Why whitelisted traffic has been not allowed/trusted immediately?


User Story #2 – Security Intelligence

• Problem description: Inability to access local web servers from outside network

103BRKSEC-3455

No sings of drops:

• Connection Events

• IPS Events

• Malware Events

• SI events

root@firepower:/Volume/home/admin# cd /var/sf/iprep_download/

# grep "72.4.119.2\|#" *

d8eea83e-6167-11e1-a154-589de99bfdf1:#Global-Blacklist

d8eea83e-6167-11e1-a154-589de99bfdf1:72.4.119.2

# cat d8eea83e-6167-11e1-a154-589de99bfdf1

#Global-Blacklist

72.163.4.161


Lesson learned …

104BRKSEC-3455


Packet-Flow

PDTS

Data-Path / LINA




Snort Verdict

IPS


Detection Engine / Snort – L7 ACL

• Order of operation: rules are being processed from top to bottom

• Differentiate ACP rule operations between (AND operand) and within columns (OR operand)

• Adaptive profiling needs to be enabled (in order to determine App ID) – “on by default”

106BRKSEC-3455


Detection Engine / Snort – L7 ACL

• Identification of App ID occurs usually within 3-5 packets or after SSL handshake

107BRKSEC-3455

> system support firewall-engine-debug

172.16.1.10-60467 > 20.20.20.10-21 6 AS 1 I 7 no match rule order 3, 'FTP to be

allowed', app s=-1 c=-1 p=-1 m=-1

20.20.20.10-53156 > 172.16.1.10-21 6 AS 1 I 46 Starting with minimum 3, 'FTP to be

allowed', and SrcZone first with zones 2 -> 1, geo 0 -> 0, vlan 0, sgt tag:

untagged, svc 165, payload 4002, client 2000000165, misc 0, user 9999997, icmpType

0, icmpCode 0

20.20.20.10-53156 > 172.16.1.10-21 6 AS 1 I 46 match rule order 3, 'FTP to be

allowed', action Allow

or 65 535


Packet-Flow

PDTS

Data-Path / LINA




Snort Verdict

IPS


Traffic IN but not OUT

109BRKSEC-3455

firepower# sh cap

capture i type raw-data trace detail interface INSIDE [Capturing - 114 bytes]

match icmp any any

capture o type raw-data trace detail interface OUTSIDE [Capturing - 0 bytes]

match icmp any any


Snort: IPS policy

• “Troubleshooting thoughts”

• Connection inspected by SNORT?

• “show conn” – Flag ‘N’

• Packet captures (capture and capture-traffic) shows incoming traffic on ASA/LINA side, diverted traffic flows are send to the SNORT, but NO outgoing or there are missing packets after SNORT inspection on outside interface?

• Connection events are triggering? -> FMC Connection table view

• Is the right AC rule being evaluated? -> NGFW debugs

• IPS events are not populated? -> Create custom ICMP rule or enable “ICMP echo-reply” rule

1:408 to confirm IPS events are generally working

110BRKSEC-3455


Snort: IPS policy

• In IPS policy rule to “Drop and Generate” action

• Interface should be in the “Inline” mode

• IPS policy needs to have “Drop when Inline” option enabled

111BRKSEC-3455


How FTD is blocking traffic?

112BRKSEC-3455

firepower# sh cap i packet-number 1 trace

1: 09:09:18.644467 172.16.1.17 > 20.20.20.100: icmp: echo

request

Type: SNORT

Result: DROP

Snort Verdict: (black-list) black list this flow

Action: drop

Drop-reason: (snort-drop) Snort requested to drop the frame


Preprocessor

• Special Attention when packets are blocked, but there are no IPS events.

113BRKSEC-3455

Change Rule State:

Drop and Generate


Inline-normalization

114BRKSEC-3455


IPS policy troubleshooting was never easier as in 6.2+

Type: SNORT

Result: DROP

Packet: TCP, ACK, seq 3806011039, ack 3309256170

Firewall: allow rule, id 268434444, allow

IPS Event: gid 1, sid 1000000, drop

Snort detect_drop: gid 1, sid 408, drop

AppID: service HTTP (676), application unknown (0)

Firewall: allow rule, id 268434444, allow

Snort: processed decoder alerts or actions queue, drop

IPS Event: gid 1, sid 1000000, drop

Snort detect_drop: gid 1, sid 1000000, drop

NAP id 2, IPS id 1, Verdict BLACKLIST, Blocked by IPS

Snort Verdict: (black-list) black list this flow

Action: drop Drop-reason: (ips) Blocked or blacklisted by the IPS preprocessor

Capture with trace detail / packet tracer:


Data-Path

Data-Path / LINA

RX

L3, L2


ACL

Egress

Interface NAT

TX

ALGchecks

Ingress

Interface

Existing

Conn


PDTSYES

NO

DAQ



Data-path: Inspection

firepower# show service-policy flow tcp host 20.20.20.11 host 172.16.2.100 eq 21

Global policy:

Service-policy: global_policy

Class-map: inspection_default

Match: default-inspection-traffic

Action:

Input flow: inspect ftp

Class-map: class-default

Match: any

Action:

Output flow: Input flow: set connection random-sequence-number

disable

set connection advanced-options UM_STATIC_TCP_MAP


Data-Path

Data-Path / LINA

RX

L3, L2


ACL

Egress

Interface NAT

TX

ALGchecks

Ingress

Interface

Existing

Conn


PDTSYES

NO

DAQ



Data-path: NAT, L2 and L3 next hop

Remaining checks are same as on the standalone ASA:

• Determination of NAT IP header – in capture trace phase ‘NAT’ with translated IP addresses details

• Based on the packet processing step “Egress Interface” determination the ‘out’ entries will be now checked in ASP routing table

• Using packet capture trace detail option we can see phase “ROUTE-LOOKUP” with the next-hop IP address IP address details

120BRKSEC-3455


Data-Path

Data-Path / LINA

RX

L3, L2


ACL

Egress

Interface NAT

TX

ALGchecks

Ingress

Interface

Existing

Conn


PDTSYES

NO

DAQ



DATA-PATH

Packet processing - TX ring

122BRKSEC-3455

Advanced Snort / FirePOWERSI (DNS/URL), Identity

> show capture out


2: 15:52:55.250627 20.20.20.33 > 172.16.1.56: icmp: echo reply

> show capture in


2: 15:52:55.250643 20.20.20.33 > 172.16.1.56: icmp: echo reply

FTD Troubleshooting tools


What are main FTD processes and what they do?snort inspects network traffic (pass,

block and alert)

sftunnel secure tunnel between

managed device and FMC

ids_event_processor sends intrusion events to

managing device (FMC)

diskmanager,

Pruner

managing disk space and

clean up old files

ids_event_alerter sends intrusion events to

Syslog or SNMP serverLina Responsible for Firewall

functionality like ACL, NAT, Routing etc.

wdt-util used for fail-to-wire /

hardware bypass

Snmpd,

ntpd

SNMP monitoring,responsible for time

synchronization

SFDataCorrelator processing events pm (process

manager)

responsible for launching

and monitoring of all FTD

relevant processes and

restarting them in case of

failure

124BRKSEC-3455


Process Management - basics

125BRKSEC-3455

Process name

Category

Status

Process ID

FMC Root CLI:

fmc-vklauzov:/# pmtool status | grep " - " | head

SFDataCorrelator (normal) - Running 15278

mysqld (system,gui,mysql) - Running 15109

httpsd (system,gui) - Waiting

sftunnel (system) - Running 19857


Process Management - basics

126BRKSEC-3455

FMC Root CLI:

root@fmc-2:/# pmtool disablebyid sftunnel

root@fmc-2:/# pmtool status | grep " - " | grep sftunnel

sftunnel (system) - User Disabled

root@fmc-2:/# pmtool enablebyid sftunnel

root@fmc-2:/# pmtool status | grep " - " | grep sftunnel

sftunnel (system) - Running 1720


Data-path and Snort capture points

127BRKSEC-3455


DATA-PATH

data-path inbound

data-path outbound

snort inbound/outbound

1.

2.

3.firepower# capture in

firepower# capture out

> capture-traffic


Data-path inbound/outbound - The Wires Never Lie!

firepower# capture in interface INSIDE match icmp any any trace detail

Capture nameInterface name

protocol

SourceDestination

Data-path/lina (diagnostic cli):


Snort Capture - The Wires Never Lie! (1)

129BRKSEC-3455

CLISH:

> capture-traffic

Options: -s 0 -w capture.pcap icmp and host 172.16.1.17

IP 172.16.1.17 > 20.20.20.100: ICMP echo request,id 24538,seq 1,length 64

Berkeley Packet Filter syntax – same as for tcpdump capturing tool

-s 0 means snaplength, in other words no limit for packet size

-w filename.pcap indicates to which file you want to write output of data captured by specified filter

capture is written to /ngfw/var/common/ folder

Copy file out to SCP server:

file secure-copy <IP address of server> <username> <location where to copy the file> capture.pcap


Snort Capture - The Wires Never Lie! (2)

130BRKSEC-3455

CLISH:

firepower# sh cap inside

802.1Q vlan#208 P0 172.16.2.11 >

20.20.20.11: icmp: echo request

LINA CLI:

firepower# sh cap outside

172.16.2.11 > 20.20.20.11: icmp: echo

request

IN OUT

LINA CLI:

NON-VLAN TAGGED TRAFFIC VLAN TAGGED TRAFFIC

> capture-traffic

Options: -v -n -e (icmp and host 172.16.2.11) or (vlan and icmp and host 172.16.2.11)

00:50:56:b6:0b:33 > 58:97:bd:b9:73:ee, ethertype 802.1Q (0x8100), length 78: vlan 208,

p 0, ethertype IPv4, (tos 0x0, ttl 128, id 5366, offset 0, flags [none], proto ICMP

(1), length 60)


Which ACP rule is being evaluated?

>system support firewall-engine-debug

Please specify an IP protocol: icmp

Please specify a client IP address: 172.16.1.17

Please specify a server IP address: 20.20.20.100

Monitoring firewall engine debug messages172.16.1.17-8 >

20.20.20.100-0 1 AS 1 I 44 New session

172.16.1.17-8 > 20.20.20.100-0 1 AS 1 I 44 using HW or preset

rule order 2, 'allow and inspect', action Allow and prefilter

rule 0

172.16.1.17-8 > 20.20.20.100-0 1 AS 1 I 44 allow action

131BRKSEC-3455

• Tool that provides the Access Control Rule evaluation status for each flow as we receive packets in real time.

• NGFW debug needs to have specified at least one filtering condition.


Access Control Policy Rule Hit Counters

132BRKSEC-3455

> show access-control-config

===================[ ciscolive ]====================

Description :

Default Action : Allow

Default Policy : Balanced Security and Connectivity

Logging Configuration

DC : Disabled

Beginning : Disabled

End : Disabled

Rule Hits : 10

Variable Set : Default-Set

... (output omitted) ...

# watch ´/usr/local/sf/bin/sfcli.pl show firewall | grep "ciscolive\| Rule\:\|Rule Hits "´

===================[ ciscolive ]====================

Rule Hits : 10

------------------[ Rule: allow ]-------------------

Rule Hits : 14

Policy name

Rule name



133BRKSEC-3455


===================[ ciscolive ]====================

Description :




DC : Disabled


End : Disabled

Rule Hits : 14




===================[ ciscolive ]====================

Rule Hits : 14

------------------[ Rule: allow ]-------------------

Rule Hits : 14

Policy name

Rule name



134BRKSEC-3455


===================[ ciscolive ]====================

Description :




DC : Disabled


End : Disabled

Rule Hits : 19




===================[ ciscolive ]====================

Rule Hits : 19

------------------[ Rule: allow ]-------------------

Rule Hits : 14

Policy name

Rule name



135BRKSEC-3455


===================[ ciscolive ]====================

Description :




DC : Disabled


End : Disabled

Rule Hits : 26




===================[ ciscolive ]====================

Rule Hits : 26

------------------[ Rule: allow ]-------------------

Rule Hits : 14

Policy name

Rule name


ACP Rule Hit Counters – FMC WebUI

• Analysis -> Custom -> Custom Workflows -> Create Custom Workflow and use Table “Connection Events”

• Add page and fill in fields like: “Access Control Policy”, “Access Control Rule”, “Count”, “Initiator IP”, “Responder IP”

• Add Table view

136BRKSEC-3455


ACP Rule Hit Counters – FMC WebUI vs CLISH

137BRKSEC-3455

Why the hit counters do not match?


------[ Rule: DNS and icmp ]------

Action : Allow

Destination Ports : protocol 6, port 53

protocol 17, port 53

protocol 1

protocol 6, port 80


DC : Enabled

Beginning : Enabled

End : Enabled

Rule Hits : 28


(truncated)

Event Path


Types of Events

• Network Discovery Events• information about a host based on traffic seen from the host

• Connection Events• when a session matches an AC rule with logging

• Intrusion Events• when an IPS rules trigger (Drop and Generate Event)

• File Events• when a file is captured

• Malware Events• when a file is captured and it is detected to be Malware

139BRKSEC-3455


FTD Detection Engine Logging• When an event is generated in detection engine, it is written to :

/ngfw/var/sf/detection_engine/<uuid>/instance-*/

• Intrusion events – snort-unified.log.1497179589

• Connection/File events – unified_events-2.log.1497179650

• Malware events – unified_events-1.log.1497179650

• Network Discovery events – unified_events-2-rna.log.1497179650

BRKSEC-3455 140

Decode Linux Epoch Time

date -d@1497179589

Sun Jun 11 11:13:09 UTC 2017


FTD Detection Engine Logging

• Determine detection engine UUID:

141BRKSEC-3455

ftd-4100-2:/# de_info.pl

________________________________________________________________________

DE Name : Primary Detection Engine (1e149ee0-3f8f-11e7-b625-b451664b5209)

DE Type : ids

DE Description : Primary detection engine for device 1e149ee0-3f8f-11e7-b625-

b451664b5209

DE Resources : 12

DE UUID : 4dec8fce-3f8f-11e7-b0f0-d383664b5209

________________________________________________________________________________

# cd /ngfw/var/sf/detection_engines/4dec8fce-3f8f-11e7-b0f0-d383664b5209/instance-1/


Event Path for IPS event

142BRKSEC-3455

NGFWFMC

IDS Event Service:

TOTAL TRANSMITTED MESSAGES <4> for IDS Events service

RECEIVED MESSAGES <b> for service IDS Events service

SEND MESSAGES <2> for IDS Events service

HALT REQUEST SEND COUNTER <0> for IDS Events service

STORED MESSAGES for IDS Events service (service 0/peer 0)

STATE <Process messages> for IDS Events service

REQUESTED FOR REMOTE <Process messages> for IDS Events service

REQUESTED FROM REMOTE <Process messages> for IDS Events service

> sftunnel_status

/ngfw/var/sf/detection_engine/<uuid>/instance-*/snort-unified.log


Event Path for Malware/Connection event

143BRKSEC-3455

NGFWFMC

Priority UE Channel 0 service – high priority queueTOTAL TRANSMITTED MESSAGES <4> for UE Channel service

RECEIVED MESSAGES <2> for UE Channel service

SEND MESSAGES <2> for UE Channel service

HALT REQUEST SEND COUNTER <0> for UE Channel service

STORED MESSAGES for UE Channel service (service 0/peer 0)

STATE <Process messages> for UE Channel service

REQUESTED FOR REMOTE <Process messages> for UE Channel service

REQUESTED FROM REMOTE <Process messages> for UE Channel service

> sftunnel_status


unified_events-1.log.<timstamp> -- malware

unified_events-2.log.<timestamp> – connection


Event Path for Network Discovery event

144BRKSEC-3455

NGFWFMC

Priority UE Channel 1 service – low priority queueTOTAL TRANSMITTED MESSAGES <4> for UE Channel service

RECEIVED MESSAGES <2> for UE Channel service

SEND MESSAGES <2> for UE Channel service

HALT REQUEST SEND COUNTER <0> for UE Channel service

STORED MESSAGES for UE Channel service (service 0/peer 0)

STATE <Process messages> for UE Channel service

REQUESTED FOR REMOTE <Process messages> for UE Channel service

REQUESTED FROM REMOTE <Process messages> for UE Channel service

> sftunnel_status

/ngfw/var/sf/detection_engine/<uuid>/instance-*/unified_events-2-rna.log.<timestamp>

Mysteries of IPS events logging


IPS-logging

146BRKSEC-3455

NGFW20.20.20.1010.10.10.20

ICMP request ICMP request

ICMP reply

IPS block SID 1:408:8


IPS-logging

147BRKSEC-3455NGFW

FMC

Syslog Servers

TC

P 8

30

5

Se

cu

red

ch

an

ne

l

eth0

management0

IPS event/s


snort-unified.log.1497179014

# date -d@1497179014

Sun Jun 11 11:03:34 UTC 2017

1.

2.


IPS-logging

148BRKSEC-3455NGFW

FMC

Syslog Servers

TC

P 8

30

5

Se

cu

red

ch

an

ne

l

eth0

management0

IPS event/s

20.20.20.1010.10.10.20


Possible root cause?

149BRKSEC-3455


IPS alerting configuration review (1)

• IPS Policy -> Advanced Settings

150BRKSEC-3455


IPS alerting configuration review (2)

151BRKSEC-3455


System processes review

152BRKSEC-3455

> pmtool status

d002ce08-55e0-11e7-a28f-534987204de8-alert (de) – Down 31729

Command: /ngfw/usr/local/sf/bin/ids_event_alerter

PID File: /ngfw/var/sf/detection_engines/d002ce08-55e0-11e7-a28f-534987204de8/ids_event_alerter.pid

Enable File: /ngfw/var/sf/detection_engines/d002ce08-55e0-11e7-a28f-534987204de8/ids_alert.conf

> pmtool enablebyid d002ce08-55e0-11e7-a28f-534987204de8-alert

d002ce08-55e0-11e7-a28f-534987204de8-alert (de) – Running 41324

Command: /ngfw/usr/local/sf/bin/ids_event_alerter

PID File: /ngfw/var/sf/detection_engines/d002ce08-55e0-11e7-a28f-534987204de8/ids_event_alerter.pid

Enable File: /ngfw/var/sf/detection_engines/d002ce08-55e0-11e7-a28f-534987204de8/ids_alert.conf


IPS-logging

153BRKSEC-3455NGFW

FMC

Syslog Servers

TC

P 8

30

5

Se

cu

red

ch

an

ne

l

eth0

management0

IPS event/s


snort-unified.log.1497179014

# date -d@1497179014

Sun Jun 11 11:03:34 UTC 2017

Conclusion


Take the chance and drive your FTD installation to a success

• Plan your desired hardware based on capabilities and performance

• Plan your desired feature-set and functionality

• Plan your desired operations mode (there are choices)

• Plan a pilot-phase with extra timing for all operational tasks

• Upgrades/Downgrades

• Backup/Restore

• Replacement/RMA

• Practice basic troubleshooting steps

• Have a look at new features and functionality inside a testbed

155BRKSEC-3455

We wish you every success operating and troubleshooting your new NG-Firewall


• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card.

• Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us.

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.

for BRKSEC-3455

http://www.ciscolive.com/us

http://ciscolive.com/Online

http://ciscolive.com/Online

http://www.ciscolive.com/Online


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

157BRKSEC-3455


Reminder - You don’t want to miss at #CLUS

158BRKSEC-3455

TECSEC-3301

Firepower Data-Path

Troubleshooting

John Groetzinger

BRKSEC-2020

Firepower NGFW

Deployment in the Data

Center and Enterprise

Network Edge using

FTD

Steven Chimes

BRKSEC-2050

Firepower NGFW

Internet Edge

Deployment Scenarios

Jeff Fanelli

For YourReference


Reminder - You don’t want to miss at #CLUS

159BRKSEC-3455

TECSEC-2004

Troubleshooting FTD

like a TAC Engineer

Ben Ritter

Kevin Klous

BRKSEC-3035

Firepower Platform

Deep Dive

Andrew Ossipov

BRKSEC-3020

Troubleshooting ASA

Firewalls

Kevin Klous

For YourReference

Thank you

Veronika Klauzova

BRKSEC-3455

Thank you for attenting

BRKSEC-3455

Dissecting Firepower-d2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/BRKSEC-3455.pdf · Dissecting...

Documents

Transcript of Dissecting Firepower-d2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/BRKSEC-3455.pdf · Dissecting...