CiscoLive BRKAPP-3003

BRKAPP-3003

Troubleshooting the Application ControlEngine (ACE)

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 2

Core Message

To understand the architecture and flow management is to understanding how to troubleshoot the Application Control Engine


Session ObjectiveAt the End of the Session, You Will Be Able To: ACE Architecture

Understand the ACE architecture and connectivity through ACE

Verify software images, licenses and image recovery

Use the real-time “TCP-DUMP” command

Implement management traffic protection

Understand access-list list on ACE

Flow Management Understand the difference between “L4” and “L7” processing

Check for possible asymmetric flows

Understand high availability from the show commands

Provide layer 7 troubleshooting

Ability to monitor performance and troubleshoot resources


ACE ArchitectureDiscuss the Architecture

Functions of control plane and data plan

Common debugging commands

Packet Capturing on and logging

Traffic Forwarding on ACE

Management Traffic Protection

Flow ManagementConnection Handling on ACE

Health Monitoring on ACE

High Availability on ACE

Layer 7 Troubleshooting and Performance

Session Agenda

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 5

ACE Architecture


ACE Module Hardware Architecture

SwitchFabricInterface

16G

DaughterCard 1

DaughterCard 2

8G

8G

SSLCrypto

10G

NetworkProcessor 1

NetworkProcessor 2

10G10G

2GClassificationDistributionEngine(CDE)

Consoleport

SupConnect

100M

ControlPlane

Data Plane


Network Processer Micro-Engines

Receive + Fastpath (+ Transmit)

IP Reassembly + Timers + Syslog

Inbound Connection Manager

Outbound Connection Manager

Connection Close Management

TCP

HTTP

Application fixups

SSL Record Layer

Static and user-configurable REGEX

TCP Normalization + FixUps

Rx FastPath

FastPath

FastPath

FastPath

FastPath

IP FragTimers

ICM

OCM CCM TCP HTTP

HTTP SSLRecord

RegEx FixUpsTCP Norm.

CPU “Xscale”


Separation of Data and Management Traffic

Control-PathDevice controlConfiguration manager (CLI, XML API, SSH, …)Server health monitoring (native probes, TCL scripts)SYSLOGs, SNMP, …ARP, DHCP relayHigh-Availability

Control path and data path run on separate processors

Data-PathConnection management

TCP termination

Access lists

SSL Offload

Regular expression matching

Load Balancing & forwarding


Traffic to the ACE


Traffic Flow to the CDE

The ACE has no native ports. The Switch Fabric Interface forwards packets to the CDE

A packet comes in over the Switch Fabric Interface marked with the VLAN and the L2 information

This is the TenGigabit Ethernet link (Te?/1, where ? is the slot number)

Packets entering/leaving the ACE will traverse this link, using VLAN tagging to indicate the VLAN

The CDE (Classification and Distribution Engine) fills out the IMPH header and forwards traffic to the appropriate blade subsystem (e.g., CP, NP1, NP2…)


Traffic Flow to the CDE - Continue

The CDE hashes incoming packets to be forwarded to either NP1 or NP2 based on the following:

TCP/UDP – hash of source/destination port

Non TCP/UDP IP – hash on source/destination IP address

NonIP – hash on source/destination L2 MAC

All forwarding is done on the NPs. These constitute two parallel forwarding paths which maintain independent connection state and forward independently


Traffic to the ACE – Control Plane

Traffic directed to the ACE itself is received on the Control Plan. Useful statistics are:

“Show netio stats” and “show fifo stats” counting traffic into/out of the CP

“Show netio clients” show applications which have registered to receive traffic from the CDE

There are a number of useful context-specific commands These are for ACE terminated traffic, and do not measure traffic forwarded by the ACE!! show ip traffic

show [protocol] statisticsprotocol can be arp, udp, tcp, icmp


ACE in a Nutshell


ACE in a Nutshell Cisco ACE provides many advanced load balancing features

Features consisting of interface and application security, server offload, and application load balancing


Virtual Context Setup

Virtual contexts are virtualized ACEs. Each virtual context has independent configuration and dedicated resources assigned. One context can pull resources from another

Every ACE device contains a special virtual context called "Admin“. It is recommended that you create separate virtual contexts for load balancing

The capacity of each ACE virtual context is determined by its resource class


Common Debugging


Common Debugging VIP is not responding

when trying to connect

If you try ping the VIP you must configure loadbalance vip icmp-reply


Common Debugging

Show command on the Catalyst 6500 Supervisor

show versionshow clockshow moduleshow powershow asic slot <n>show interface TenGigabitEthernet <n>/1show interface TenGigabitEthernet <n>/1 trunkshow svclc vlan-group[no] power enable <module>

Make sure the module status is OK


Common Debugging Show command available on the ACE

show versionshow clockshow ft group statusshow ip int brshow int vlan <n>show arpshow service-policyshow serverfarmshow rservershow probeshow connshow statshow ip trafficshow resource usageshow np 1 me-stats “-s norm”show np 1 me-stats “-s norm –M1”

System Information

L2, L3

Performance,ResourcesDebuggingFlows

L4, L7

This provides the DELTA


Looking at the Normalization counters

Shows the DROP counters in Fast Path and TCPswitch/Admin# show np 1 me-stats "-s norm" | i Drop[Drops] L2 invalid DA mac: 0[Drops] L4 port is zero 0[Drops] TCP invalid conn miss flags: 0[Drops] TCP invalid flags: 0[Drops] TCP urgent pointer denied: 0[Drops] TCP non-zero reserved field: 0[Drops] TCP syn data denied: 0[Drops] TCP non-syn options on syn: 0[Drops] TCP syn options on non-syn: 0[Drops] TCP no of denied options: 0[Drops] TCP option length wrong: 0[Drops] fp TCP invalid ack in syn-ack: 0[Drops] fp TCP invalid ack for syn-ack: 0[Drops] fp TCP ack past seq: 0[Drops] fp TCP window left edge: 0[Drops] fp TCP window right edge: 0[Drops] fp TCP data past FIN: 0[Drops] fp TCP FIN has wrong seq: 0[Drops] fp TCP RST has wrong seq: 0[Drops] fp TCP RST has wrong ack: 0[Drops] fp TCP ack > FIN_ACK exp: 0[Drops] fp TCP exceeded MSS: 18


Show Module from the Catalyst 6500 Supervisor

cat6k#show modMod Ports Card Type Model Serial No.--- ----- -------------------------------------- ------------------ -----------1 1 Application Control Engine 10G Module ACE10-6500-K9 SAD093508042 48 48 port 10/100 mb RJ45 WS-X6348-RJ-45 SAD04450L445 2 Supervisor Engine 720 (Active) WS-SUP720-3BXL SAD08300D5L

Mod MAC addresses Hw Fw Sw Status--- ---------------------------------- ------ ------------ ------------ -------1 0001.0002.0003 to 0001.0002.000a 0.504 8.6(0.252-En 3.0(0)A1(2) Ok2 00d0.d32e.1b42 to 00d0.d32e.1b71 1.5 5.4(2) 8.5(0.46)RFW Ok5 000f.f7be.b17c to 000f.f7be.b17f 4.0 8.1(3) 12.2(PP_R31_ Ok

Mod Sub-Module Model Serial Hw Status---- --------------------------- ------------------ ----------- ------- -------5 Policy Feature Card 3 WS-F6K-PFC3BXL SAD083006N2 1.3 Ok5 MSFC3 Daughterboard WS-SUP720 SAD082905VE 2.1 Ok

Mod Online Diag Status---- -------------------1 Pass2 Pass5 Pass

Module status shows OK


Verifying Version and Licenses

switch/Admin# show versionCisco Application Control Software (ACSW)Softwareloader: Version 12.2[118]system: Version A2(1.0) [build 3.0(0)A2(1.0)system image file: [LCP] disk0:c6ace-t1k9-mz.A2_1.bininstalled license: ACE-08G-LIC ACE-VIRT-250 ACE-SSL-15K-K9

HardwareCisco ACE (slot: 1)cpu info:number of cpu(s): 2cpu type: SiBytecpu: 0, model: SiByte SB1 V0.2, speed: 700 MHzcpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz

Installed Licenses


Available System Memory and Uptime

switch/Admin# show version – Continuation of output[...]

memory info:total: 958004 kB, free: 335372 kBshared: 0 kB, buffers: 3540 kB, cached 0 kB

cf info:filesystem: /dev/cftotal: 499744 kB, used: 447136 kB, available: 52608 kB

last boot reason: reload command by adminconfiguration register: 0x1ACE kernel uptime is 7 days 23 hours 42 minute(s) 25 second(s)

Displays ACE module uptime


What Licenses Are Installed

View the current licenses installedswitch/Admin# show licenseACE-250CTX-08G-SSL-15K.lic:SERVER this_host ANYVENDOR ciscoINCREMENT ACE-08G-LIC cisco 1.0 permanent 1 \

VENDOR_STRING=<count>1</count> HOSTID=ANY \

NOTICE="<LicFileID>20060523161924670</LicFileID><LicLineID>1</LicLineID> <PAK></PAK>" SIGN=76DA7526434AINCREMENT ACE-SSL-15K-K9 cisco 1.0 permanent 1 \


NOTICE="<LicFileID>20060523161924670</LicFileID><LicLineID>7</LicLineID>

<PAK></PAK>" SIGN=1077701CF92CINCREMENT ACE-VIRT-250 cisco 1.0 permanent 1 \


Shows the license file installed


Installing New Licenses on ACE

Copy license file to disk0: on the ACEswitch/Admin# dir disk0:

636 Apr 17 16:04:04 2007 ACE-250CTX-08G-SSL-20K.lic236 Apr 17 16:06:54 2007 ACE-16G-LIC.lic

switch/Admin# license ?install Install the licenseuninstall Uninstall the licenseupdate Update existing license

License commands available on the ACE

License commands available on the ACE. Reload only required when increase throughput on the ACE10switch/Admin# license install disk0:ACE-16G-LIC.licInstalling license... done


ACE File System

Use the dir command to view directory listing for filesswitch/Admin# dir ?core: Directory or filenamedisk0: Directory or filenameimage: Directory or filenameprobe: Directory or filenamevolatile: Directory or filename

The internal File system is mapped as below/mnt/cf - Image:

Also the following compressed file systems are used/TN-HOME = disk0:/TN-CONFIG = Startup config /TN-LOGFILE = Internal Storage for audit logs/TN-CERTKEY-STORAGE : internal storage for Cert and Keys/TN-COREFILE = core:


ACE File System

Load debug plug-in to access ACE file systemStartup configuration located at /mnt/cf/TN-CONFIG

ACE will generate / fix any missing or corrupted file systems during boot

When to use the format command?

If you receive the following error

Warning!! This will erase everything in the compact flash including startup configs for all the contexts and reboot the system!!

switch/Admin# write memory ERROR!config filesystem is not mounted on compact flash


Working with Core Files

If ACE creates a core file you can locate the files from the core directory

All cores files are stored in dir core: (core names are self explanatory)

switch/Admin# dir core:99756 Apr 5 17:57:05 2007 ixp2_crash.txt13047 Apr 5 17:56:59 2007 loadBalance_core_log.tar.g

Ixpx_crash.txt will have some details on the core dump

If it is a kernel crash , then a file named crashinfo wil be available in core:

Show version wil show last reload reason


Invoke Context

To display the context running configuration information from the Admin context, use the invoke context command

invoke context context_name show running-config

switch/Admin# invoke context BreakingPoint show running-config write memory Generating configuration....

switch/Admin# invoke context Exchange2010 show running-config | include 192.168.1.1Generating configuration....ip address 192.168.1.11ip address 192.168.1.12

alias 192.168.1.1 255.255.255.0Sandbox-Pod2-ACE20-1/Admin#


System Logging


Logging Features Each virtual context generates logs independently and

send to specified destinationsSyslog server, console, telnet/ssh, buffer, flash, supervisor, SNMP, NAT

Rate limiting of syslog messages is recommended. Never log to the console using level 7

ACE will log connection setup/teardown at the connection speed

Access-List deny entries are logged

Use the terminal monitor command to display log message when not using console

Useful commands to troubleshoot syslogging:show logging statistics

show logging queue | last


Basic Configuration to Enable Logging Enable logging on the ACE

logging enablelogging monitor 7no logging message 111008no logging message 111009logging timestampdo terminal monitor

It is recommended to disable or changing the severity level of some syslog messages. Use logging message syslog_id [level severity_level] command

To enable the logging of connection setup and teardown messages, use the logging fastpath command. Use the logging rate-limit to limit the rate at which the ACE generates messages in the syslog


Real-Time “TCP Dump”



Supportability and analysis of load balanced traffic is a major requirement in today's load balanced environment

ACE can capture real-time packet information for the network traffic that passes through the ACE

The attributes of the packet capture are defined byan ACL

The ACE buffers the captured packets, and you can copy the buffered contents to a file in flash memory on the ACE or to a remote server

User can also display the captured packet information on your console or terminal; capture can also be exported to capture to Ethereal



To enable the packet capture on ACE use the capturecommand

capture c1 interface vlan 211 access-list FILTER bufsize 64

Buffer in Kbytes(can be circular)

Pre-defined ACL toidentify relevant traffic

Interface to applycapture

One capture session per context

Capture triggered at flow setup

Capture configured on client interface where flow is received


Real-Time “TCP Dump” ACE can capture traffic based on a configured access-list

and interface

Follow the following procedure to capture traffic on ACE:1. Specify an ACL

2. Capture on an interface or globally

access-list FILTER line 10 extended permit tcp any any eq wwwcapture c1 interface vlan 211 access-list FILTER

Show capture status show status and buffer size

switch/Admin# show capture c1 status Capture session : c1 Buffer size : 64 KCircular : no Buffer usage : 1.00%Status : stopped


Real-Time “TCP Dump” Start the capture on the ACEswitch/Admin# capture c1 start23:40:37.236868 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 58: 172.16.11.190.443 > 209.165.201.11.1180: S 1389739009:1389739009(0) ack 617249474 win 17408 <mss 1460> (ttl 255, id 2401, len 44, bad cksum 0!)23:40:37.239102 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 54: 172.16.11.190.443 > 209.165.201.11.1180: . ack 71 win 17408 (ttl 255, id 2402, len 40, bad cksum 0!)switch/Admin# capture c1 stop

To copy the packet capture to disk0: use the copy capture

switch/Admin# copy capture c1 disk0: c1

Maximum buffer size is 5MB of data


Traffic Forwarding on ACE


ACE Load Balancer Policy Lookup Order

There can be many features applied on a given interface, so feature lookup ordering is important

The feature lookup order followed by datapath in ACE is as follows:1. Access-control (permit or deny a

packet)

2. Management traffic

3. TCP normalization/connection parameters

4. Server load balancing

5. Fix-ups/application inspection

6. Source NAT

7. Destination NAT

The policy lookup order is implicit, irrespective of the order in which the user configures policies on the interface


ACE in Router Mode

IP subnets cannot overlap within a context but can across two contexts

Non-Load balanced traffic is routed. ACE needs to ARP for destination before forwarding packet

Client MAC ACE MAC

Client IP VIP

Random Port VIP Port

ACE MAC SelectedServer MAC

Client IP Server IP

Random Port Server Port


ACE in Bridge Mode

Non-Load balanced connection are bridged from client to server vlan

Client MAC ACE MAC

Client IP VIP

Random Port VIP Port

Client MAC SelectedServer MAC

Client IP Server IP

Random Port Server Port


Checking VLAN Configuration Show interface provides you with valuable information switch/Admin# show interface vlan 211vlan210 is up

Hardware type is VLANMAC address is 00:16:36:fc:b3:36Virtual MAC address is 00:0b:fc:fe:1b:02Mode : routedIP address is 172.16.10.21 netmask is 255.255.255.0FT status is activeDescription:WAN SideMTU: 1500 bytesLast cleared: neverAlias IP address is 172.16.10.23 netmask is 255.255.255.0Peer IP address is 172.16.10.22 Peer IP netmask is 255.255.255.0Assigned on the physical port, up on the physical port

499707 unicast packets input, 155702918 bytes1485258 multicast, 5407 broadcast0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops497610 unicast packets output, 46804782 bytes6 multicast, 8201 broadcast0 output errors, 0 ignored


MAC Addresses

Virtual MAC (VMAC) is used for the alias IP, VIP address Alias IP and Virtual IP (VIP) are associated with a

VMAC only if high availability is configured Active context responds to ARPs for alias IP with

VMAC One unique VMAC per FT Group 00:0b:fc:fe:1b:XX

(XX=FT group number in hex) Packets destined to the VMAC are blocked on standby

context


MAC Addresses

The VMAC is a function of ft-group-id. Therefore different cards must have different ft-group-ids Use the show interface internal iftable to locate the

VMAC Each ACE supports 1,024 shared VLANs, and uses

only one bank of MAC addresses randomly selected at boot time ACEs may select the same address bank so avoid this

conflict use the shared-vlan-hostid command


Key Things to Know About ARP on ACE

For unicast packets, if the destination MAC is unknown ACE will drop the packet, instead of flooding it

So IP-address-to-MAC mapping and outgoing interface needs to happen first

ARP entries are populated as follows:With ARP requests

Learning through incoming ARP requests

Gratuitous ARP packets

Layer 2 mode:No MAC learning

So ARP is the way to learn IP to MAC and interface mapping


How to Read the ARP Table

Each virtual context maintains its own ARP tableswitch/Admin# show arpContext Exchange=======================================================================IP ADDRESS MAC-ADDRESS Interface Type Encap NextArp(s) Status=======================================================================172.16.11.1 00.00.0c.07.ac.0a vlan211 GATEWAY 226 87 sec up172.16.11.19 00.12.43.dc.83.bb vlan211 INTERFACE LOCAL _ up172.16.11.190 00.12.43.dc.83.bb vlan211 VSERVER LOCAL _ up192.168.1.1 00.0a.b8.66.60.85 vlan411 INTERFACE LOCAL _ up192.168.1.11 00.50.56.12.11.01 vlan411 RSERVER 230 87 sec up192.168.1.12 00.50.56.12.11.01 vlan411 RSERVER 229 87 sec up192.168.20.254 00.0a.b8.66.60.85 bvi2 INTERFACE LOCAL _ up==================================================================Total arp entries 11

ARP table shows the type of ARP entry from Gateway, Interface, VSERVER, RSERVER


Admin Context Resource Reservation



If Admin context is not configured correctly admin could be starved of all resources

When configuring resource allocations in ACE, it is possible to allocate 100% of resources to non-Admin contexts, so that the Admin context is no longer reachable via ICMP, telnet, SNMP, etc

It also appears that in some cases this will cause FT between a pair of HA ACE modules to fail, and create an active/active situation

Highly recommended to put some safe guard in place to ensure that the Admin context always receives at least a small percentage of resources


Admin Context Resource Reservation Shows starved admin context

switch/Admin# show arpContext Admin =============================================================================IP ADDRESS MAC-ADDRESS Interface Type Encap NextArp(s) Status ============================================================================= 10.87.102.225 00.00.00.00.00.00 vlan621 GATEWAY - * 2 req up10.87.102.229 00.0b.fc.fe.1b.01 vlan621 ALIAS LOCAL _ up10.87.102.230 00.0a.b8.71.2f.ef vlan621 INTERFACE LOCAL _ up172.16.0.1 00.0a.b8.71.2f.ef vlan999 INTERFACE LOCAL _ up172.16.0.2 00.05.9a.3b.92.e9 vlan999 LEARNED 18 * 2 req up============================================================================= Total arp entries 5

switch/Admin# ping 10.87.102.225 Pinging 10.87.102.225 with timeout = 2, count = 5, size = 100 .... No response received from 10.87.102.225 within last 2 sec No response received from 10.87.102.225 within last 2 sec 2 packet sent, 0 responses received, 100% packet loss

Unable to reach ACEs default gateway


Admin Context Resource Reservation Shows starved resources and drops for throughputswitch/Admin# show resource usage context Admin

Allocation Resource Current Peak Min Max Denied -------------------------------------------------------------------------------Context: Admin conc-connections 9 9 0 0 0 mgmt-connections 2 12 0 0 0 proxy-connections 0 0 0 0 0 xlates 0 0 0 0 0 bandwidth 0 4715 0 0 3704068

throughput 0 4247 0 0 3704068 mgmt-traffic rate 0 468 0 125000000 0

connection rate 0 7 0 0 8 ssl-connections rate 0 0 0 0 0 mac-miss rate 0 1 0 0 0 inspect-conn rate 0 0 0 0 0 acl-memory 26816 26880 0 0 0 sticky 0 0 0 0 0 regexp 0 0 0 0 0 syslog buffer 1024 4096 0 1024 0 syslog rate 0 7 0 0 118

No resources reserved



Shows heartbeats missed increasing. Heartbeats are not reaching the peer. Possibility for both Aces to go Active/Active

switch/Admin# sh ft stats HA Heartbeat Statistics------------------------Number of Heartbeats Sent : 1095573 Number of Heartbeats Received : 1095239 Number of Heartbeats Missed : 2987 Number of Unidirectional HB's Received : 2640 Number of HB Timeout Mismatches : 0 Num of Peer Up Events Sent : 1 Num of Peer Down Events Sent : 1 Successive HB's miss Intervals counter : 0 Successive Uni HB's recv counter : 0



Below shows the problem why ACE is starved of all resources

resource-class admin limit-resource all minimum 0.10 maximum equal-to-min

Suggest the following reserved resources for Adminresource-class Admin

limit-resource conc-connections min 5.00 max equal-to-min

limit-resource mgmt-connections min 5.00 max equal-to-min

limit-resource rate bandwidth min 5.00 max equal-to-min

limit-resource rate ssl-connections min 5.00 max equal-to-min

limit-resource rate mgmt-traffic min 5.00 max equal-to-min

limit-resource rate conc-connections min 5.00 max equal-to-min


Access-Control Lists


ACL Merge Process and Enhancements New ACL merge enhancements added to the ACE

ACL merge is responsible for merging all the features and generating a single merged list for an given interface. ACL compiler is responsible for programming the merged list into MTrie data structure – “Fast retrieval of data”

ACL memory usage has been optimized to better support incremental changes

The new implementation provides a consistent ACL memory usage during system bootup time and during incremental changes after the system comes up

This feature also provides an early detection of failure if the configuration needs more ACL resources than what system can support


View Total Action Nodes

Use the show np 1 access-list resource to view action nodes

switch/Admin# show np 1 access-list resource ACL Tree Statistics for Context ID: 3=======================================ACL memory max-limit: NoneACL memory guarantee: 0.00 %MTrie nodes(used/guaranteed/max-limit):

6 / 0 / 262143 (compressed) 2 / 0 / 21999 (uncompressed)

Leaf Head nodes (used/guaranteed/max-limit): 3 / 0 / 262143

Leaf Parameter nodes (used/guaranteed/max-limit): 7 / 0 / 524288

Policy action nodes used: 4memory consumed: 4696 bytes resource-limited 128 bytes other 4824 bytes total.min-guarantee: 0 bytes total.max-limit: 78610432 bytes total, 0 % consumed

The total policy action nodes counts for ACE:

ACE Module - 200k

ACE 4710 Appliance - 40k


Troubleshooting Secure Socket Layer (SSL)


Troubleshooting SSL

Configuration of SSL on ACE is relatively simply. However if you experience issue how do you troubleshoot?

Make sure the certificate and key used in ssl-proxy are valid. Use the crypto verify commandswitch/Admin# crypto verify RSA2048.key RSA2048.certKeypair in RSA2048.key matches certificate in RSA2048.cert

Check the size and location of the key. Use the show crypto key commandswitch/Admin# show crypt key allFilename Bit Size Type-------- -------- ----RSA2048.key 2048 RSA


Troubleshooting SSL

Looking at the certificate details. Use the show crypto certificate command switch/Admin# show crypto certificate cisco-sample-certCertificate:

Data:Version: 3 (0x2)Serial Number:

ad:e4:e2:f1:50:b7:ce:bdSignature Algorithm: sha1WithRSAEncryptionIssuer: C=IN, ST=KA, L=BLR, O=CISCO, OU=ADBU, CN=SSL-TESTValidity

Not Before: Apr 3 09:50:55 2009 GMTNot After : Apr 1 09:50:55 2019 GMT

Subject: C=IN, ST=KA, L=BLR, O=CISCO, OU=ADBU, CN=SSL-TESTSubject Public Key Info:

Public Key Algorithm: rsaEncryptionRSA Public Key: (1024 bit)

Modulus (1024 bit):00:cf:a2:60:66:5b:ce:b6:38:6f:94:df:0d:1c:61:26:af:7a:05:49:ed:8d:93:3b

Exponent: 65537 (0x10001)


Troubleshooting SSL – CRL Download Check to make sure you can download the CRLswitch/Admin(config-ssl-proxy)# do show crypto crl test2 detailtest2:URL: http://119.60.60.23/test.crlLast Downloaded (Cached): Sat Aug 8 16:14:24 2009 UTCTotal Number Of Download Attempts: 1Failed Download Attempts: 0Successful Loads: 1 Failed Loads: 0Hours since Last Load: 0 No IP Addr Resolutions: 0Host Timeouts: 0 Next Update Invalid: 0Next Update Expired: 0 Bad Signature: 0CRL Found-Failed to load: 0 File Not Found: 0Memory Outage failures: 0 Cache Limit failures: 0Conn failures: 0 Internal failures: 0Not Eligible for download: 3 HTTP Read failures: 0HTTP Write failures: 0

Looking for all best-effort CRLs in the system and their download status. Use the show crypto crl best-effort command

http://119.60.60.23/test.crl�


Advanced SSL Debugging This command provides the current crypto statistics

switch/Admin# sh np 1 me-stats "-s cryptoCrypto Statistics: (Current)------------------ARC4 operations: 376572 0TCP msgs received: 285260 0APP msgs received: 235151 0Nitrox messages forwarded to XScale: 381041 0SSL ctx allocated: 47758 0SSL ctx freed: 47758 0SSL received bytes: 61070430 0SSL transmitted bytes: 283256220 0SSL received application bytes: 7679113 0SSL transmitted application bytes: 275120867 0SSL received non-application bytes: 53391317 0SSL transmitted non-application bytes: 3292887 0Bulk flush operations: 95037 0ME records sent to XScale: 285808 0ME records received from XScale: 47723 0ME hw responses: 471516 0First segments received: 47400 0Handshake failure alert: 94 0CM close: 446 0


Advanced SSL Debugging

The show stats crypto server command provides statistics of the SSL handshake

switch/Admin# show stats crypto server+---- Crypto server termination statistics -----++------- Crypto server alert statistics --------++--- Crypto server authentication statistics ---++------- Crypto server cipher statistics -------++------ Crypto server redirect statistics ------++---- Crypto server header insert statistics ---+

These statistics provide details of the SSL packets for example; which version client interacted with ACE, which cipher is used, whether re-handshake happened, whether session id reuse happened and which SSLalerts are received or sent by ACE


Connection Handling on ACE


Flow ManagementLevel of Flow Processing Type of Processing Feature of Function

Layer 3 and Layer 4 Balance of first packet Basic Load Balancing

Applies to TCP/UDP for layer 4 rules Source IP Sticky

Applies to all other IP protocols TCP/IP Normalization

Select server or farm based on source IP

Layer 7 TCP Splicing Terminate TCP Connection HTTP Layer 7 rules of first request (URL LB)

Buffer request, inspect, LB Cookie Sticky (Persistence)

Create Hardware Shortcut Generic TCP Payload Parsing

Layer 7 Re-proxy TCP Splicing + ability to parse subsequent HTTP requests within the same TCP

HTTP Layer 7 rules with HTTP 1.1 connections keepalive(“persistence rebalance”)

Layer 7 Full-Proxy Fully terminate clients connection SSL Offload

TCP re-use

HTTP 1.1 Pipelining

Protocol Inspection (FTP,SIP)


Internal Mapping of TCP/UDP Flows

TCP and UDP Flows = 2 X Internal Half Flows

switch/Admin# show conn

conn-id np dir proto vlan source destination stat-------------+--+----+--------+-----+--------------------------+-------------------------------+---------+

9 1 In TCP 211 209.165.201.11:1867 172.16.11.190:80 ESTAB

6 1 Out TCP 411 192.168.1.11:80 209.165.201.11:1867 ESTAB

Client IP:port VIP Address

Server IP Returning half flow automatically created for both TCP and UDP flows

INIT, SYNACK,ESTAB, CLOSED

SYN_SEEN, SYN_SEEN,ESTAB, CLOSED

Non TCP shows as “--”

Use conn-id to track flow through ACE

Check the Network Processor


Troubleshooting Connections

Use the show stats connection command to show connections statistics.

Use the clear stats connection command to clear these counters

switch/Admin# show stats connection+------------------------------------------++------- Connection statistics ------------++------------------------------------------+Total Connections Created : 288232Total Connections Current : 2Total Connections Destroyed: 283404Total Connections Timed-out: 892Total Connections Failed : 3934

Note: ACE does not destroy connection. These are connections closed correctly!!!


Troubleshooting Connections Use the show stats loadbalance command to view the

loadbalance statistics

To clear the load balance statistical information stored in the ACE buffer, use the clear stats loadbalance command switch/Admin# show stats loadbalance+------------------------------------------------------------++------- Loadbalance statistics ----------------------++------------------------------------------------------------+Total version mismatch : 0Total Layer4 decisions : 0Total Layer4 rejections : 0Total Layer7 decisions : 24Total Layer7 rejections : 0Total Layer4 LB policy misses : 0Total Layer7 LB policy misses : 0Total times rserver was unavailable : 0Total ACL denied : 0


Troubleshooting VIPswitch/Admin# show service-policy client-vips detailStatus : ACTIVEDescription: ------------------------------------------Interface: vlan 211 service-policy: client-vipsclass: VIP-HTTPSVIP Address: Protocol: Port:172.16.11.190 tcp eq 443 loadbalance:L7 loadbalance policy: HTTPS-POLICYVIP Route Metric : 77VIP Route Advertise : DISABLEDVIP ICMP Reply : ENABLED-WHEN-ACTIVEVIP State: INSERVICEcurr conns : 22 , hit count : 22dropped conns : 0 client pkt count : 0 , client byte count: 0 server pkt count : 0 , server byte count: 0 max-conn-limit : 0 , drop-count : 0 conn-rate-limit : 0 , drop-count : 0 bandwidth-rate-limit : 0 , drop-count : 0 L7 Loadbalance policy : HTTPS-POLICYclass/match : class-defaultLB action :

primary serverfarm: backend-sslbackup serverfarm : -

hit count : 22dropped conns : 0

First command you show use view connection to VIP


Troubleshooting Serverfarmswitch/Admin# show serverfarm HTTPS-FARM detailserverfarm : HTTPS-FARM, type: HOSTtotal rservers : 4active rservers: 4description : -state : ACTIVEpredictor : ROUNDROBINfailaction : -back-inservice : 0partial-threshold : 0num times failover : 0num times back inservice : 0total conn-dropcount : 0

----------connections-----------real weight state current total failures

---+---------------------+--------+---------------------+-----------+------rserver: linux-1

192.168.1.11:0 8 OPERATIONAL 0 0 0max-conns : - , out-of-rotation count : -min-conns : -conn-rate-limit : - , out-of-rotation count : -bandwidth-rate-limit : - , out-of-rotation count : -retcode out-of-rotation count : -

Best command for checking server status and load


Layer 7 Troubleshooting


Layer 7 Policy Hits

Expanding the show service-policy using the detail option to provide hit count for layer 7 matches

switch/Admin# show service-policy client-vips detail

Status : ACTIVEDescription: ------------------------------------------Interface: vlan 211

service-policy: client-vipsL7 Loadbalance policy : pslb

class-map : curl1 LB action :

serverfarm: s1 hit count : 3dropped conns : 0

class-map : curl2 LB action :

serverfarm: s2 hit count : 0 dropped conns : 0

Shows hit count for layer 7 load balanced policy


Match URL Hit Count

Expanding the show service-policy using the url-summary option to provide visibility on which match http url are getting hit

switch/Admin# show service-policy url-summaryService-Policy: VIRTUAL-HOSTING-01 L3-Class: WEB-SSL L7-Class: A1

match http url /ECBACCOUNTINQUIRY_V5/.* hit: 42

Service-Policy: VIRTUAL-HOSTING-01 L3-Class: WEB-SSL L7-Class: A2match http url /AADSLICER/.* hit: 93match http url /ANALYSISHELP/.* hit: 102match http url /BOXIR2/.* hit: 67match http url /BUSINESSOBJECTS/.* hit: 78match http url /DSWSBOBJE/.* hit: 84

Use the show service-policy <service-policy-name> class-map <L3-class map-name> url-summary to provide better granularity


Troubleshooting HTTP Statistics

To effectively troubleshoot HTTP use the show stat http commandswitch/Admin# show stats http

+------------------------------------------++-------------- HTTP statistics -----------++------------------------------------------+LB parse result msgs sent : 6288 , TCP data msgs sent : 9143 Inspect parse result msgs : 0 , SSL data msgs sent : 6041

TCP fin/rst msgs sent : 135 , Bounced fin/rst msgs sent: 19 SSL fin/rst msgs sent : 13 , Unproxy msgs sent : 0 Drain msgs sent : 3107 , Particles read : 37917 Reuse msgs sent : 1539 , HTTP requests : 3145 Reproxied requests : 0 , Headers removed : 1549 Headers inserted : 1598 , HTTP redirects : 2 HTTP chunks : 0 , Pipelined requests : 0 HTTP unproxy conns : 0 , Pipeline flushes : 0 Whitespace appends : 0 , Second pass parsing : 0 Response entries recycled : 3032 , Analysis errors : 0 Header insert errors : 1509 , Max parselen errors : 0 Static parse errors : 9 , Resource errors : 0 Invalid path errors : 0 , Bad HTTP version errors : 0


Troubleshooting HTTP Cookies ACE parses HTTP requests for cookies with the name

given in the configuration and can skip a certain number of bytes and look for another specific number of bytes.

If the cookie is not found, then the ACE looks for a string in the URL, starting with one of the characters /?&#+ and followed by a "=", then parses that value.

If no cookie or HTTP URL cookie exists ACE defaults to the predictor for that farm

ACE can parse up to HTTP headers (includes cookies) up to 64kB (default header max parse length is 2048k)

Make sure that the sticky timeout matches the session timeout on the application


Troubleshooting TCP Connection Re-Use When using TCP conn re-use,"Connection: keep-alive" is

inserted and "Connection: close" is removed from the clients HTTP request, to avoid closing the server connection early

User needs to create a layer 7 class-map and configure Source Nat when using TCP conn re-use

class-map type http loadbalance match-any L7-RE-USE2 match http url .*

Use the show stats http | include Reuse counters to check if see if TCP Re-uses is getting used

switch/Admin# show stats http | include ReuseReuse msgs sent : 1 , HTTP requests : 4


Fundamentals for ACE probing ACE probes are fundamental to the system. It is key to not

oversubscribe the ACE health monitoring system

Use the show resource internal socket to determine how many open sockets the ACE has open. This is a Admin commandswitch/Admin# show resource internal socketApplication MaxLimit Current Creates Frees--------------------------------------------------------------SYSTEM 4000 0 0 0CRITICAL 50 0 0 0AAA 256 0 0 0MGMT 256 0 0 0XINETD 512 1 12 11HEALTH_MON 2500 532 193494 192962USER_TCL 200 0 0 0SYSLOG 256 10 14 4VSH 256 0 0 0OverAll - 650 194812 194162Non Reg App Usage: 107


Health Monitoring Process

If you see probing issues check the health monitoring process. The show proc cpu command provide very useful information

switch/Admin# show proc cpuCPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35%PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process972 1072965 613352 1749 35.9 18.5% 21.67% 20.90% arp_mgr

HM process is only consuming 1.40%. Why is the control plane CPU running at 30%? Check what process is running at 30%

switch/Admin# show proc cpuCPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35%PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process987 90257 57805 1561 0.0 1.40% 1.46% 1.43% hm988 90198 58952 1530 0.0 1.49% 1.49% 1.44% hm989 851 2947 288 0.0 0.0 % 0.1 % 0.0 % hm990 0 2 56 0.0 0.0 % 0.0 % 0.0 % hm



Use the show probe detail command to determine the status of the probe or possible last failure

switch/Admin# show probe detail – Cut output

--------------------- probe results --------------------probe association probed-address probes failed passed health------------------- ---------------+----------+----------+----------+-------rserver : CAS1

10.7.53.55 24 24 0 FAILED

Socket state : CLOSEDNo. Passed states : 0 No. Failed states : 1No. Probes skipped : 0 Last status code : 403No. Out of Sockets : 0 No. Internal error: 0Last disconnect err : Received invalid status codeLast probe time : Wed Nov 25 18:48:16 2009Last fail time : Wed Nov 25 18:25:16 2009Last active time : Never


High Availability on ACE


High Availability Basic building blocks

FT PEEROnly one FT peer per ACE device

1:1 peer relationship

FT GROUPOne FT group per ACE virtual context

FT VLANDesignated VLAN between the redundant peers

All HA related traffic sent over this VLAN

FT VLAN can be trunked between two Catalyst 6500 Chassis

Cannot be used for normal traffic

Admin Context

Context A

Context B

Context A

Context B

ACE2 (FT PEER)

FT VLAN

FT Group 2

FT Group 3

ACE1 (FT PEER)

FT Group 1


High Availability Control Traffic

TCP Connection between FT PeersState Machine (Election, Preempt, Relinquish)

Configuration sync

State Sync for ARP

HA KeepAlives

Heartbeats between FT peersHeartbeats are sent over UDP

Monitors the health of the peer

Heartbeat interval and count are configurable


ACE High Availability State Machine

Active Standby Election (assuming both peers are initialized at same time)

Based on a priority scheme Member with highest priority becomes ACTIVEOther member enters the STANDBY_CONFIG state If priorities are equal, member with the higher IP address wins

STANDBY_CONFIG StateStartup Configuration Sync from Active to StandbyRunning Configuration Sync from Active to StandbyKnob to turn on/off



STANDBY_BULK StateARP Sync (knob to turn on/off)

Connection Table Sync

Sticky Database Sync (knob to turn on/off)

STANDBY_HOT StateStandby FT group member is ready to take over

Incremental Configuration Sync from Active to Standy

Incremental State Sync from Active to Standby

STANDBY_COLD StateDue to error during Config Sync or Incremental Config Sync

No Config or State Sync happens from Active to Standby



Mismatch in software versionFT Peer may become INCOMPATIBLE (SRG Check)

ACTIVE ACTIVE state on both FT group members

Mismatch in Virtual Context LicensesConfiguration Sync (all types) for Admin context is disabled

State Sync for Admin context will continue to happen

For matching user contexts – Configuration State Sync will work

Mismatch in Other LicensesConfiguration and State Sync will work

After switchover, new Active will handle traffic as per its licenses


ACE Redundancy Query VLAN Query VLAN can be configured as an alternate path for

pinging the peer when no heat beat is being received from redundant peer

If configured, upon receiving a PEER_DOWN message from the heat beat process, the ACE data-plane tries to do a ping to the destination via the Query VLAN

If Ping fails, the Standby will transition to the ACTIVE state

If Ping succeeds, the Standby will transition to a STANDBY_COLD state

To configure a query interface, enter the following:

switch/Admin(config-ft-peer)# query-interface vlan 110


Common Debugging - Concussion

This session should provide you with some directions on where to start troubleshooting ACE!!

CiscoLive BRKAPP-3003

Documents

Transcript of CiscoLive BRKAPP-3003