VSX Troubleshooting - CPUG
Transcript of VSX Troubleshooting - CPUG
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
VSX Troubleshooting
Q u i c k g u i d e
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Agenda
How VSX is built (in brief)
Management scheme
Gateway architecture
Licensing
Issues to fix
Tools and methods
2
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Reference Note
Pictures from Check Point publicly available documents are used in this presentation
Information from Check Point troubleshooting documentation used in this presentation
3
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Management Side
V S X A r c h i t e c t u r e
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
MGMT model of VSX
Three tear infrastructure
Two types
SmartCenter
Provider-1
5
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
SmartCenter Model
Nothing special
6
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Provider-1 Model
Virtual Systems are managed by different CMAs
Special so called “Main CMA” to manage VSX cluster objects
“Target CMAs” to manage particular VSs
7
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Provider-1 Model
8
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
MGMT DB objects
Two types instead of one as for regular FW
network_object - security aspects of a Virtual Device
vs_slot_objects - networking aspects of a Virtual Device
network_object - on the Target CMA
vs_slot objects - on Main CMA
9
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
vs_slot objects
Special DB table called vs_slot_objects
Network interfaces of VS
Routes of VS
Other VS specific attributes, such as reference to hosting VSX object
Vital info for creation and change of VS
10
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Network Configuration Scripts
Configuration changes are pushed to VSX from MGMT with NCS
NCS are generated on MGMT
On VSX GW NCS are parsed and executed
11
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Types of NCS
local.vs - NCS file - last configuration change
local.vsall - NCS file, full configuration, executed on startup
local.vskeep - contains list of existing VSIDs, used at system startup
12
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
local.vs
Interfaces lists of each vs_slot: interfaces - interfaces’ configuration to be interfaces_installed - existing interfaces configuration
Each vs_slot object has 2 attributes containing routes lists: routes - routing table to be routes_installed existing routing table
local.vs file is created by comparing and calculating the difference of “interfaces” to “interfaces_installed” and “routes” to “routes_installed”
13
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
local.vsall and local.vskeep
Each Virtual Device has 2 NCS files kept on the management: VS_name.vsnew - NCS file containing interfaces VS_name.vsrt - NCS file containing routes
These files are updated each time configuration change is done using SmartDashboard.
local.vsall is created by concatenating all the files of all the Virtual Devices.
local.vskeep is created by going over vs_slo_objects table and writing all the Virtual Devices VSIDs to it.
14
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Location of NCS files
All .vsnew and .vsrt files - in $FWDIR/conf/vs_repository/VSX_NAME of Main CMA
MGMT: local.vs, local.vsall and local.vskeep files - in $FWDIR/state/VSX_NAME/VSX/
VSX GW: local.vs, local.vsall and local.vskeep files in $FWDIR/state/__tmp/VSX/ If scripts processed successfully, they copied to to $FWDIR/state/local/VSX/ directory
15
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Provider-1 forwarding
16
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
VSX private network
“Funny IPs”
For internal communication
Default cluster private network: 192.168.196.0/22
Cluster Private Network can be changed: - In SmartDashboard, if there is no VDon the VSX - By using “vsx_util change_private_net”
17
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
gateway Side V S X A r c h i t e c t u r e
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Important note
VS is NOT a virtual machine
Common file space
Common kernel
VRFs and kernel contexts are different
19
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
VRFs
A Multiple routing domain (VRF) has separate:
VRF IDInterfacesUnicast routing tableRouting cacheMulticast forwarding cacheARP tableLoopback interfaceSockets
VRFs enable overlapping IP addresses.
20
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
VRF: CLI changesMind context when working with ip route, iputils, ping, traceroute, arp, route, ifconfig and netstat:
- traceroute –Z vrfid - ip route vrf vrfid - “-z vrfid” for the rest (arp –z 1, netstat –z 2 -rn)
Use “all” instead of “vrfid” to show information for all VRFs
VRF context can be changed by “vsx set <vsid>” or “vrfctl –s <vrf>” commands.
VRF 0 - physical machine
21
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
VSX user mode
Processes:
Multi context: fwd, cpd, cplogd and fibmgr
Single context: vpnd and gated
System resources like CPU and HDD space are shared
22
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
File Structure
VS folders - CTX under $FWDIR / $CPDIR
CTX00xxx - Virtual Device ID xxx.
VSX machine (VSID 0) - $FWDIR / $CPDIR
23
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Creating VS object 1
License validation
Update context database with Virtual Device information $CPDIR/conf/ctxdb.C)
Create Virtual Device registry entries (OTP for SIC certificate)
Create Virtual Device directories and soft-links:
$CPDIR/CTX/CTX00xxx/conf
$FWDIR/CTX/CTX00xxx/log, database, …
24
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Creating VS object 2
Create initial policy
Create the OS VRF instance
Create the VS instance in the FW kernel and load security policy
Send a message that notifies cpd and fwd that a new context was added. cpd adds the new context to its db.
25
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Troubleshooting techniques
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Useful knowledge
Management debugging (ref. P-1 lecture)
Gateway architecture and troubleshooting techniques
ClusterXL
SecureXL
27
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Things to Check First
Licensing on both MGMT and GW sides
Connectivity between VSX and MGMT
All the jazz:
- local time settings - static routes - IP addressing - mind funny IPs- etc...
28
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Management Debugging
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Management Issues
Provisioning
Changes
vsx_util operations
policy installation
30
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Important
Do not lock Main CMA while working with VSX on Target CMAs
31
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Debbuging fwm
TDERROR_ALL_ALL - might be too much
vsx provisioning and vsx_util: TDERROR_ALL_VSXM
Policy installation: TDERROR_ALL_INSTMGR
32
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
How to set debug flags
Mind context!!!
fw debug fwm on TDERROR_ALL_VSXM=INFO
Or export TDERROR_ALL_VSXM=INFOand restart fwm process
33
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Debug output
$FWDIR/log/fwm.elg
34
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Turning it off
fw debug fwm TDERROR_ALL_VSXM=0
fw debug fwm off
35
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Which CMA?
Most of the cases - Main CMA
Policy installation - Target CMA
36
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Gateway Debugging
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Common Issues
Connectivity
Policy
Interfaces
Clustering
38
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
To check first
Connectivity
Topology of VSX cluster and adjacent networks
Local times
Licenses
39
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Overvew
vsx stat -v
VSX Gateway Status==================Name: test1Security Policy: StandardInstalled at: 25Jul2010 3:42:11
SIC Status: TrustNumber of Virtual Systems allowed by license: 25Virtual Systems [active / configured]: 7 / 7Virtual Routers and Switches [active / configured]: 0 / 0Total connections [current / limit]: 4994 / 135000
Virtual Devices Status====================== ID | Type & Name | Security Policy | Installed at | SIC Stat-----+-------------------------+-------------------+-----------------+--------- 1 | S test1_XXXXXXXXXXXX1...| Standard | 25Jul2010 3:42 | Trust 2 | S test1_XXXXXXXXXXXX2...| Standard | 25Jul2010 3:42 | Trust 3 | S test1_XXXXXXXXXXXX3...| Standard | 25Jul2010 3:42 | Trust 4 | S test1_XXXXXXXXXXXX2...| Standard | 25Jul2010 3:42 | Trust 5 | S test1_XXXXXXXXXXXX2...| Standard | 25Jul2010 3:42 | Trust 6 | S test1_XXXXXXXXXXXX2...| Standard | 25Jul2010 3:42 | Trust 7 | S test1_XXXXXXXXXXXX2...| Standard | 25Jul2010 3:42 | Trust
40
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Tools
tcpdump -i <IF name>
fw monitor [–v <vsid>] -e <Your filter> Example: fw monitor –v 2 –e “port(80) and ip_p=17, accept;”
Note: changing context does NOT help “fw monitor” to limit output
41
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Acceleration
Most of fw monitor output is accelerated, so you will see just the first packet.
fwaccel [-vs <vsid>] [conns | templates | stat | on | off]
Mind VS number
42
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Cluster Issues
Get statuscphaprob [-vs vsid] stat
Interfaces statuscphaprob -a [-vs vsid] if
Problem notification list:cphaprob [-vs vsid] list
Force member UP or DOWN, for failover tests:clusterXL_admin up/down
43
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Kernel Debug
One kernel for all!
Massive output, mind performance
Some express debugs:fw ctl zdebug drop | grep <your filter> - to see drop reason on specific traffic
Mind kernel buffer size
44
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
System Tools
arp, route, netstat, ifconfig - to have “-z X” flag where X is VS number
-z all prints info for all VSs
ping -z...
traceroute -Z... (Capital letter)
45
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
VS Policy
To fetch the last installed policy:fw [-vs <vsid>] fetch local
Fetching the last policy that failed to be installedfw fetchlocal -d $FWDIR/state/__tmp/FW1/
To unload policy:fw [-vs <vsid>] unloadlocal
Unload policy for all VSs:fw vsx unloadall
46
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
VS configuration
To fetch configuration: fw vsx fetch
For specific VS: fw vsx fetchvs –vs 2
To see NCS script for a specific VS: fw vsx showncs <vsid>
47
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Other tips
Double check topology
If you cannot figure connectivity issues, especially some traffic degradation, suspect ClusterXL before others
48
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Questions And
Answers
(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland
Thank You For Your Time!