Aol Internal Networking Manual

8/6/2019 Aol Internal Networking Manual

1/36

Introduction

Purpose and Objectives

The purpose of this document is to provide the AOL Network Engineering Staff, Management and anyother pertinent persons a detailed review, analysis and recommended best practices document for the

implementation of layer 4 through 7 switching configurations. This is an evolving document, providing

information from which an open forum of discussion can take place.Scope

The scope of this document covers a review of SLB concepts, the Foundry ServerIron product line,general and advanced configuration parameters, health check operation and implementation, , and a

case study section for easy reference. This document focuses primarily on the operation and

configuration of the Foundry Networks ServerIron, though the concepts of SLB covered within are not

limited to the Foundry ServerIron product. 90% of the SLB enabled services deployed within AOL isgeneral in nature.

Server Load Balancing Review

Basic SLB ReviewServer Load Balancing is a technique used by Layer 4 devices to spread the load of a particular

service across many servers, AKA a server farm. These services can include, but are not limited Web,

FTP, SSL, etc. The following drawing depicts a typical SLB topology deployed within AOL:?

Figure 1

The clients connect to the Virtual IP (VIP) that represents the service. The Layer 4 switch thendistributes the connection on behalf of the client to one of the servers within the server farms servicing

the particular service requested by the client. The Layer 4 switch performs many functions with respect

to this, including client connection management, real server (the servers that make up the server farm)

health monitoring, security, and even Layer 2 services just to name a few. The configuration can getquite complex, depending on what services are being provided by the Layer 4 switch. Its important to

note here that the physical topology is essentially the same as if only a Layer 2 switch were

deployed.

SLB is classified as switching at the OSI Layers 4 through 7. In layer 4 SLB, the switch is

distributing client connections based on either TCP or UDP information gleaned from the clientrequest. With the upper layers, the switch is more application aware; the device will examine the

client request in more detail in an attempt to glean information from which a switching decision can be

made. Layer 7 switching involves more processing overhead, but with some applications, can offer a

more robust connectivity model for application switching decisions.Advanced SLB Concepts

Layer 7 Switching

Layer 7 switching involves the switch making SLB decisions based on information gleaned from theclient request, at the application level. This includes the actual URL, cookies, and even SSL Ids.

URL switching

The ServerIron has the ability to identify portions of (or match the entire) URL request from the

client and direct that request to a group of (or a specific server) based on what was gleaned from therequest.

Cookie Switching


2/36

Similar to URL switching, the ServerIron is looking into the URL data to identify a cookie

within the request. Cookies take the form of a Name=Value pair, and the request is directed to the

group or specific real server based on that match.

SSL switching

SSL switching allows the ServerIron to direct a request from a client to the real servers where

an SSL session was initially established.

Caching

Proxy caching and transparent caching is the ability of the switch to identify based on policies and

other configuration parameters client requests for services, and direct them to a cache server farm.The Foundry ServerIron SLB Products

The Foundry ServerIron comes in 2 flavors, the stackable and chassis-based device.ServerIron XL Stackable

The ServerIron XL is a stackable device with 24 10/100 ports combined with 2 dedicated uplink ports;

the uplinks can be either 10/100 or Gigabit Ethernet. The device contains a 400 Mhz Power PC and 32Mbs of RAM. This device will be refered to as the XL in the remainder of the document.

Big ServerIron ChassisThe Big ServerIron is an upgraded BigIron chassis device, with either an M3 or M4 management

board. The upgrade consists of an EEPROM installation coupled with a code upgrade. This

configuration will be refered to as the BSI for the remainder of the document.

ServerIron 400The ServerIron 400/800 is a 4 slot and 8 slot chassis device with an M5 / WSM4 management module.

The M5 has four processors, a management processor (MP), and three barrel processors (BP). The

management processor handles basic system functions such as bootup / initialization, MAC / ARP tablemaintenance, and other system-wide functions. The barrel processors perform the actual SLB function

for the slots that are mapped to it (slot mapping is discussed in section #).. This device will be referred

to as the SI400 for the remainder of the document.

Configuring the SI products consist of setting general global system parameters, advanced global

parameters, and the SLB specific parameters required for SLB functions, as well as interface

specific configuration information. They are consistent between the products, with a few notedexceptions. The configuration format can be illustrated as:

!general and advanced global configuration

!

SLB-specific configuration (Real servers and Virtual Servers)!

Interface configuration (Link parameters, Caching parameters)

!

Sections 3 through 6 provide detailed information concerning general configuration recommendations

and practices.


3/36

General Global System Configuration Parameters

General system parameters consist of assigning the Layer 4 switch an IP address, IP default gateway,

device hostname, DNS zone, SNMP parameters, syslog server details, and passwords. These aregeneric, and will be determined prior to or at system installation. Example configuration follows:

console timeout 10 (timeout for console connections, set to 10 minutes)enable password-display (enables non-encrypted password display via sh run, sh config)

enable super-user-password 0 xxxxxx (sets enable password)

hostname serveriron (sets hostname)ip address 1.1.1.6 255.255.255.0 (sets IP address of the switch)

ip default-gateway 1.1.1.254 (set default gateway)

ip dns domain-name AOL.COM (sets domain name)

ip dns server-address 205.188.152.6 205.188.152.37 (sets DNS servers)logging 205.188.151.2 (sets syslog hosts)

logging 172.31.15.13

logging 172.20.20.33logging facility local5 (sets the syslog logging facility on log host)

snmp-server community 0 xxxxro 8 (sets the SNMP community string, along with the ACL required for

snmp queries)snmp-server contact NetOps (sets the snmp contact information)

snmp-server location cr5 22aa (sets the location)

It is recommended that the SNMP variable location be included for quick reference.

Advanced Global System Configuration Parameters

The advanced global parameters consists of configuration statements that affect the overall operation of

the device. These settings include layer 2 and layer 4 parameters, as well as system security settings.

Mac Aging

Background

There is no direct advantage to not having mac-aging turned on except under high traffic loads coupledwith high density server farms; the mac-aging process will disrupt traffic flow, a process obviously

undesirable under these circumstances.

MAC Aging Recommendations

It is recommended that we disable MAC aging where it is determined the environment is static; i.e. thehosts are not moving from one physical port to another. To configure MAC aging, enter the following

CLI command:

Serveriron(config)# mac-age 0

This is set on all platforms.


4/36

Hardware Aging

Background

A Foundry chassis device ages out Layer 2 CAM (Content Addressable Memory) entries once every 65

seconds after the flow has been established through the device. As is the case with MAC aging, this

process is also disruptive to the flow (subsequent packets sent during the CAM relearning process aredropped). In the past this had a dramatic impact in areas of the network that experienced very high

traffic rates.

Hardware Aging Recommendations

It is recommended we disable Layer 2 hardware aging across the all chassis products, not just Layer 4

devices. To disable hardware aging, enter the following CLI command:

Serveriron(config)# l2-hw-age-dis

There will be a complete description of Foundry CAM operation forthcoming in future tech notes.

SpanningTree

Unless implicitly required by the Layer 2 design / topology, spanning tree should be disabled. To

disable spanning tree, enter the following command via CLI:

Serveriron(config)# no spanning

IP Policy

To enable the SI400 management module barrel processors for SLB processing, the following global

command must be set:

ServerIron(config)# ip l4-policy 1 cache tcp 0 global

This command initializes the barrel processors at startup. Note: This command also has the effect of

forcing ALL of the traffic for a slot through the mapped barrel processor.

Server Router Ports

Server Router Ports are required on the uplink ports to the routers. This parameter indicates to the SI on

which ports traffic flows between the client and real servers. To configure the server router ports, use

the following CLI command:

On the XL and BSI:

Serveriron(config)# server router-ports 25 26

On the SI400:

serveriron(config)# server router-ports ethernet 2/1serveriron(config)# server router-ports ethernet 2/2

As indicated above, the configuration differs between the XL/BSI and SI400 device; on the XL and


5/36

BSI, you can enter all the router ports on one line; a separate line entry is required for each port on

the SI400. As above concerning the ip policy statement on the SI400,

Router Port Recommendations

The server-router port definitions are required for proper SLB operation.

TCP/UDP Age

The TCP/UDP-AGE parameter specifies how long an idle TCP or UDP connection remains in the

session table. The default for TCP is 30 minutes, while UDP is 5 minutes. For TCP, the SI will deleteimmediately a TCP session entry once the connection is closed. The connection is considered closed

if the SI observes either the TCP FIN / FIN-ACK sequence or a RESET from either the client or the

real server. For UDP, the session remains in the table for the duration of the configured UDP age timer.

Age Timer Recommendations

This setting is dependent on the application being serviced. For TCP, the default timer should be set at10 minutes, and adjusted as necessary depending on the application operational requirements dictate.

Ten minutes is suggested as a means of resource preservation on the ServerIron, and falls within the

range of SST based applications (SST is the proprietary AOL TCP/IP stack in use on many serverswithin AOL). A detailed discussion of SST is beyond the scope of this document. The UDP-AGE timer

should remain at the default setting. To set the TCP or UDP age, use the following CLI commands:

Serveriron(config)# tcp-age [time] (default 30 minutes)

Or

Serveriron(config)# udp-age [time] (default 5 minutes*)

Age Timer Caveats

TCP age is advertised as an on the fly setting; in code revisions 7.1.18 and possibly others with the7.1 train, it has been determined that a reboot of the device is required. In 7.3.x and SI400 device code

revisions, no reboot is required for this setting to take effect.

*The UDP-AGE timer applies to all UDP-based applications with the exception of DNS and RADIUS;

the SI will immediately age out the session once it determines there has been a reply to a DNS or

RADIUS query. This default can be overridden, but it is highly recommended it remain on the default

setting. If a change to the default behavior for DNS and RADIUS is deemed necessary, then carefulconsideration of system resources should be given. DNS and RADIUS queries and responses (in

respect to the nature of the application) can be very high rate- if the sessions on the SI are not aged out

quickly enough, then resource starvation can occur, and performance can suffer dramatically.Load Balancing Predictor

The SI provides several methods for the SLB predictor (the mechanism which determines how thetraffic is distributed amongst the configured real servers) including least connections, least sessions,

round-robin, weighted, server response time, least local connections*, and least local sessions*. The

predictor can be set globally or locally in the VIP configuration. Setting it locally on the VIP overridesthe global setting.

Least Connections


6/36

The ServerIron sends the request to the real server that currently has the fewest active connections with

clients.

Least SessionsThe ServerIron sends the request to the real server that currently has the fewest session table entries.

Round RobinThe ServerIron sends the request to each server in rotation, regardless of how many connections or

sessions each server has.

Weighted

The ServerIron uses the weights you assign to the real servers to select a real server. The weights are

based on the number of session table entries the ServerIron has for each server.

Response Time

The ServerIron selects the real server with the fastest response time.

Predictor Recomendations

In general, the load balancer predictor configured should be Round-Robin globally. There are times

when this predictor will not provide the desired results; configuring the predictor on the Virtual Serverwill override the global configuration. Round-robin is also required when port translation is configured.

This can be configured both globally and local to the VIP; configuring under the VIP definitionoverrides the global command. To configure the predictor globally, use the following commands:

Serveriron(config)# server predictor round-robin

Session Limits

The server session limit setting indicates to the SI how much memory resource to set aside for thesession table. This should be set to 1 million on the XL and BSI, and 2 million on the SI400 device. If

this setting, particularly on the stackable devices, is not configured appropriate to the load on the

switch, then high CPU and ultimately poor performance can result. To avoid this potential issue, use thefollowing commands:

XL / BSI / SI400

Serveriron(config)# server session-limit 1000000

SI400 specific

The above command as well as the following should be configured:

Serveriron(config)# server session-wsm-limit 2000000

Stateless Sessions

The term stateless refers to the SI not maintaining an active session entry in memory for the client

transaction. The obvious advantage here is resource usage- no memory is dedicated to maintaining

information about the client to server transaction.


7/36

Stateless Recommendations

The use of stateless settings is currently being evaluated. At this time, it is recommended stateless notbe used.

Security

Background

Foundry Layer 4 devices have network security mechanisms built into the code. These include SYN

defense (SYN-DEF, SYN-Guard), TCP rate limiting, and many others. At this point, it is recommendedonly SYN-DEF be utilized until further testing/verification can be performed on other techniques.

SYN-DEF

SYN-Def is a security mechanism that allows the ServerIron to provide syn-defense for the servers andsegment it is deployed on. The operation of SYN-DEF differs depending on the product in use. On the

XL and BSI devices, the mechanism monitors the three-way handshake between the client and the real

server. Once the client has sent the initial SYN, it waits for the configured time specified and expects tosee the final ACK from the client within this time frame. If it does not receive the ACK, it drops

subsequent SYN requests from the client and clears the session. On the SI4000 device, as well as the

stackable code revision 7.3 and above, enhanced SYN-DEF is used. The mechanism will not onlymonitor the connection attempt, but it will complete the three-way handshake with the server on behalf

of the client during the connection setup. This allows the connection on the server to be moved out of

the connection queues and into the established queues. Again, the timer starts upon receipt of an initialSYN from a client. If the client fails to send the final ACK within the configured time frame, then the

SI will send a TCP RESET to the real server (clearing the session on the server), and clear the session

from its session table. No RESET is sent to the client.

Enhanced SYN-DEF also has the ability to monitor pass-through TCP traffic (traffic passing through

the SI, not directed to the SI itself).

SYN-DEF Recommendations

SYN-DEF, as a feature, should be enabled. The recommended configured tolerances warrant further

testing, but are directly tied to the traffic patterns through the device it is configured on. Watching the

syn-def counters via show server debug CLI command provides insight into how SYN-DEF is

operating, as well as any potential SYN attack that may be in progress. If the counter is trippingconstantly, it may prove useful to apply a sniffer to the router interfaces to determine if legitimate

traffic is being affected, and what adjustments if any are required. Its recommended that a setting of 6

seconds be applied initially, and then adjusted (tuned per application requirements) from this point ifnecessary.

To enable syn-def, use the following CLI commands:

Serveriron(config)# server syn-def 6

On the SI400 device, you must also configure syn-def at the interface level:Serveriron(config-if-3/1)# syn-def

Access Control Lists


8/36

Weve had persistent challenges in the past getting these to work properly, resulting in many different

revisions of code that are application specific. To reduce the amount of potential issues and increase

code revision manageability, its highly recommended Foundry Layer 4 devices not run ACLS.

Source NAT

Source-NAT may be required in certain circumstances where the return traffic to the client does not

pass back through the ServerIron. One example is when using remote (or backup) servers on a differentIP segment other than the one the SI is configured for (i.e. communication between the ServerIron and

real server must pass through a router). These instances are rarely used- this command is best used as a

troubleshooting tool, and is described in section ().Force Delete

The command server force-delete allows the SI to force connection closure when a server or port on

that real server is administratively taken out of rotation. This should be set on all SI platforms. To setforce-delete, use the following CLI command:

Serveriron(config)# server force-delete

Health Checks

Layer 4 through 7 switching inherently implies and requires health check mechanisms to ensure client

perceived quality of service, server resource availability, and to some degree, maintenance. Various

aspects of the AOL client, as well as many non-client* related services, depend very heavily onservices provided by server farms front-ended by layer 4 through 7 switching products- and the

ServerIron healthchecks help ensure these services remain available.

What are health checks?

In essence, health checks refer to the ability of the Layer 4/7 application switch to determine server and

resource availability. The application switch has to perform two tasks: first, it has to ensure that theconfigured servers are initially available for rotation into the server pool for service, and second,

continued monitoring of server and application status and availability must be performed. This is

accomplished in various ways depending on the SI product in use.

ServerIron Health Check Operation

The Foundry Networks ServerIron products offer a robust scope of health check options, though some

aspects of healthcheck operation differs between the stackable and chassis based products. The SIhealthcheck mechanisms can be considered to run in two modes: initialization and operational

monitoring.

ServerIron Healthcheck Methods

ServerIron Healthchecks take place at layers 2 (Link), 3 (IP), 4 (TCP/UDP) and the application layer

(7).Layer 2/3

Layer 2/3 healthchecks consist of the ServerIron ARPing for the real server IP address and IP ICMP


9/36

echo requests (ping). When you add a real server to the configuration, the ServerIron will ARP for the

MAC address of the real server. A successful ARP can be verified via either the cli commands sh arp

or sh server debug. The latter command provides a great deal of information about the real server and

its attributes, which includes the real server state, real server port state, number of active connections,just to name a few. It is recommended the reader review the Foundry manuals online for more detail- it

is an invaluable tool for troubleshooting purposes.

For this discussion, the server state is of interest. The state can be one of 6 values: Enabled, Failed,Test, Suspect, GRACE_DN, and Active. The state is represented numerically; each state is listed

below:

(1) ENABLED

The real server is configured, but the server has not responded to an ARP or ping (echo) request.

Physical connections, IP address configuration (on the ServerIron and real server) should be verified.

(2) FAILED

The real server has failed to respond to repeated pings.

(3) TEST

The real server is still reachable at Layer 3, but one or more of the application ports configured on the

real server has failed to respond to health checks (either during initialization or configured healthchecks). The SI will continue to try to the reach the application indefinitely. For example, if the server

continues to be reachable at Layer 3 but the application check does not pass, the state will remain TEST

so long as the SI cannot reach the application that is failing its health check.

(4) SUSPECT

The ServerIron associates a time stamp with each packet sent to and received from the real servers. If

the time gap between the last packet received from the server and the last packet sent to the servergrows to 3 or 4 seconds, the ServerIron sends a ping (Layer 3 health check) to the server. If the server

doesnt respond within the ping interval (a configurable parameter), the ServerIron changes the state to

SUSPECT and resends the ping, up to the number of retries specified by the ping retries parameter(also configurable). If the server still doesnt respond after all the retries, the state changes to FAILED.

If the server does respond, the state changes to ACTIVE.

(5) GRACE_DN

The forced-shutdown(delete) option has been used to gracefully shut the server down.

(6) ACTIVEThe server has responded to the Layer 3 health check (IP ping). Also, all the services on the real server

have passed their Layer 4 (and if applicable, Layer 7) health checks.

Layer 4Layer four consists of TCP or UDP checks to the configured real server ports. For TCP, the SI will send

the real server a TCP SYN on its configured ports, expecting a SYN-ACK in response. If the SI sees

the SYN-ACK from the real server, then it will respond with a TCP RESET to the real server, andconsider the port active. For UDP, the ServerIron sends to the real server bogus data on the

configured UDP ports. Since UDP is a connectionless protocol, the ServerIron does not expect to see

any response from the real server. An ICMP port unreachable message from the real server indicatesto the SI the configured port is not active and should be brought out of rotation.

Layer 7

Layer 7 healthchecks consist of the SI interacting with the actual application on the real server to


10/36

determine operational status. Layer 7 healthchecks are for well known application ports; to the SI

these include, but are not limited to, HTTP, SSL, FTP, DNS, and SMTP. By default, there are no Layer

7 healthchecks performed by the SI. If required, they should be enabled via port profiles, or within the

real server definition itself.Real Server Initialization

Initialization takes place at Layer 2 (link), Layer 3 (IP), and Layer 4 (TCP/UDP). Upon adding the real

server to the SI configuration, the device will first ARP for the IP address of the real server. Aftersuccessfully completing the ARP process, it will then ping the real server to verify Layer 3

connectivity. Provided the real server has passed both the layer 2 and layer 3 checks, layer 4 testing

takes place after configuring the logical ports (under the real server configuration) that server will beservicing. No layer 7 health checks are performed at this time.

Real Server Operational Monitoring

Operational monitoring, depending on the device (XL/BSI vs SI400) and CLI configuration, consists oflayer 2/3, 4 and layer 7 checks. Both products will continuously send ARP requests every twenty

seconds for the real server to verify layer 2/3 connectivity. Additionally, the SI400 will send ICMP

pings to the real server every two seconds, whereas the stackable device does not.By default, the SI does not perform active periodic monitoring of real server application status.

Active health checks refer to the SI initiating and sending periodic checks to the configured server

services to ensure service availability. Instead, the SI passively monitors the session between itself,the client and the real server. This process is referred to as the reassign monitor, and its operation

differs between the XL/BSI and the SI400 based products.

The Reassign Monitor

As stated above, depending on the SI product in use, the reassign monitor operation and the indications

provided by the CLI differ greatly. The reassign monitor and its operation is discussed below.

XL/BSI Reassign Monitor Operation

On the XL and BSI, the device monitors the connection attempts between itself and the real server.

When the SI attempts to pass a new client connection onto a real server, the device will wait for twoconsecutive SYNs from the client before the connection attempt is passed to the next real server in the

rotation for that service. The reassigned servers reassign counter is increased by 3. If the reassign

counter increments past the reassign threshold (default 20), then the real server port is marked failed,taken out of rotation, and the initialization process is repeated until that real server port passes the

initialization process again. The reassign counter is decremented to zero after a TCP SYN-ACK is

received from the real server.

SI400 Reassign Monitor Operation

The reassign monitor on the SI400 operates quite differently than the stackable (prior to software

revision 7.2.23, the reassign monitor was disabled by default). Instead of monitoring the connectionstate between itself and the real server, the SI400 monitors the connection between itself and the client

device. This is accomplished by monitoring for the final ACK from the client in the TCP three way

handshake. Another significant difference is that there is no actual reassign (moving the connectionattempt to another available real server) should a real server fail to accept the connection. Based on the

application in use, the client will either times out, or the third consecutive connection attempt will be

passed to the next active server in rotation.Periodic Healthchecks (keepalives)

Periodic keepalives, or active monitoring of server port status is disabled by default. In order to


11/36

enable periodic keepalives, you must configure a port profile for the port in use on the real server.

Port Profiles

Port profiles allow the user to specify to the ServerIron various attributes of the application port in use.These attributes include what type of port (TCP or UDP for non well-known ports), keepalive interval

and retries, and many other parameters. The ServerIron assumes that non well-known ports are UDP by

default. The keepalive parameters interval and retries are analogous to OSPF HELLOs and theDEAD timers. One item to keep in mind is that when modifying keepalive parameters for the ports in

use, the parameters modified apply to the Layer 7 checks for well-known ports unless Layer 7 checks

are disabled. For non well-known ports, the parameters apply to only to Layer 4 checks.Health Checking Recommendations

These recommendations should provide a stable and predictable environment which facilitates the

highest amount of uptime and maintainability, as well as clear indications when things do go wrong

Layer 2/3 HealtchecksThe ARP process described above in section () cannot be modified at this time. However, it doesnt

appear there are any suitable reasons why constant pings every 2 seconds from the chassis device is

required, and may in fact contribute to unnecessary traffic load on the segment and real servers.

Layer 3 Health Check Recommendations

It is recommended the layer 3 healthchecks be set to 1 every 15 seconds, with 1 retry. To do this, usethe CLI commands below:

Serveriron(config)# server ping-interval 15

Serveriron(config)# server ping-retry 1

These commands are available in software releases 7.3.x for the XL, and 7.2.23 and above for the

chassis SI400.Reassign Monitor

The reassign monitor is an excellent tool for troubleshooting and early warning when a particular host

or segment is in trouble. Some reassigns are expected during normal operation, but shouldnt exceed 1or 2 percent of the total client to server (Total C->S Conn) connections counter in the CLI command

sh server session. Unfortunately, these counters are cumulative, so the observation of reassigns

should be via a snapshot of the device when trouble-shooting. It is recommended that the reassign

counter threshold be set to 100; this provides ample room for reassign activity and the reassign counterto grow without unnecessarily tripping the threshold, which would result in erroneous syslog messages

and the associated healthcheck mechanisms being instigated. To configure the reassign threshold, use

the following commands:

Serveriron(config)# server reassign-threshold 100

It may be desirable when trouble-shooting reassign issues to temporarily disable the reassign monitor.

To do this, use the following CLI command:

Serveriron(config)# server no-reassign-count

NOTE: When using DSR mode, the reassign monitor is disabled.


12/36

Periodic Healthchecks (keepalives)

It is recommended that periodic keepalives are enabled at layer 4 only; layer 7 healthchecks should

only be enabled as the application requires. Please consult the Foundry documentation for configuringlayer 7 healthchecks. Keepalives should be enabled via port profiles. The settings of one keepalive

every 30 seconds, with a retry of value 2 should be sufficient in most cases- this will provide a time of

one minute before the SI recognizes the port is down. The reassign monitor will compensate for clientconnectivity otherwise. To set periodic keepalives at layer 4 only, configure a port profile such as the

following:

Serveriron(config)# server port 8888

Serveriron(config)# tcp

Serveriron(config)# tcp keepalive 30 1

Serveriron(config)# tcp l4-check-only

This identifies to the ServerIron port 8888 is TCP (non well-known ports are UDP by default), sets the

keepalive interval to 30 seconds with 2 retries, and enables layer 4 only. The keepalive interval andretry settings may need to be adjusted based on application expectations.

Caveats

When you configure a port profile for a well known port, the healthcheck settings refer to Layer 7

healthchecks by default. Set the tcp l4-check-only for ALL ports unless Layer 7 healthchecks aredesired.

The following diagram illustrates the configured healthcheck operation:

?

The black outline indicates the reassign monitor. The diagram depicts the healthcheck mechanisms and

the resultant actions based on configuration.Server Load Balancing Configuration

By default, the ServerIron operates as a Layer 2 switch. To enable Layer4/7 switching, variousparameters must be set. These include defining the real and virtual servers, as well as any pertinent

parameters for the functions the device will be performing.

The IP Policy Command

The IP Policy command is required on the SI400; this enables the 3 barrel processors on the WSM4 /

M5 management board for operation. If this is not enabled, then the management processor will be

performing the SLB functions- resulting in less than desirable performance.

To enable the barrel processors on the WSM4 / M5, use the following global command:

ServerIron(config)# ip policy 1 tcp 0 global

Basic Server Load Balancing

Real Server Configuration Parameters

Below is a standard real server configuration:


13/36

server real real1 1.1.1.1

port http

port http url "HEAD /"

The first line, server real real1 1.1.1.1, identifies the server as a real server with the name real1 and

IP address 1.1.1.1. The second line tells the ServerIron the layer 4 port this real server will be accepting

connections on. This value can be between the ranges of Because HTTP is a well known service tothe ServerIron, the third line is added automatically for Layer 7 healthcheck purposes. Other real server

options that are not standard are discussed in the following sections.

Virtual Server Configuration Parameters

The virtual server, or VIP, is what the client sees as the host providing the service(s) being requested.

The following depicts a basic Virtual server configuration:

server virtual VIP 1.1.1.1 (Virtual Server Name and IP address)

port http (Port serviced by the VIP)

bind http real1 http real2 http real3 http (VIP port to real server port binding)

A more complex configuration may look as follows:

server virtual VIP 1.1.1.1 (Virtual Server Name and IP address)

port ftp (Port serviced by the VIP)

port ftp dsr (sets FTP to DSR mode)port ftp sticky (sets stickyness)

port ftp concurrent (sets concurrent mode)

Load Balancing Predictor

The load balancing predictor was previously discussed in Section (). As stated, this can also be

configured on a per-VIP basis. Configuring the predictor on the VIP will override the globallyconfigured predictor settings.

Load Balancing Predictor RecommendationsIt is recommended that the initial predictor be set to Round-Robin. To set the predictor under the

vip, use the following commands:

serveriron(config)# server virtual VIP 1.1.1.1ServerIron(config-vir-VIP)# predictor round-robin

Sticky Ports

Sticky refers to the ServerIron sending multiple requests for the same resource from a client to the

same real server chosen during the real server selection process. A good example of when to use sticky

ports would be shopping cart applications (online shopping), where web pages are created dynamicallyby the real server based on user input. Streaming media applications, SSL, are others that may require

sticky ports. To set a real server application port as sticky, use the following CLI commands:

serveriron(config)# server virtual VIP 1.1.1.1

serveriron(config-virtual-VIP)# port rtsp sticky

Concurrent Ports


14/36

Concurrent allows a client to have multiple connections to the same real server on multiple application

ports, as is the case with non-passive FTP. To configure concurrent ports, use the following CLI

commands:

serveriron(config)# server virtual VIP 1.1.1.1

serveriron(config-virtual-VIP)# port rtsp concurrentDirect Server Return (DSR)

Direct Server Return, or switchback, refers to the real server communicating directly with the clientwithout the need of the packet flow traversing back through the switch.

Non-DSR Review

In non-DSR, the ServerIron operates in half-NAT mode. This means the ServerIron will NAT the

destination IP address from the VIP to the real server IP address. Two session entries are created in thisinstance- one for the client to VIP connection, and one for the ServerIron to real server

connection. Each session consumes approximately 32 bytes of memory.

DSR OperationIn DSR mode, no NAT function is performed. Instead, a loopback interface is configured on the real

server with the same IP address of the ServerIron VIP. Client traffic is passed onto the real server

without changing the destination IP address. Instead, the SI will set the destination MAC address to thatof one of the real servers; the real server will accept the frame at the link level, and pass the datagram

up to the IP layer. Since the destination IP address is locally configured, the real server will accept the

datagram and pass the segment to the appropriate service. This allows the ServerIron to only create onesession entry, the client to VIP session. The ServerIron basically ignores the return traffic from the

real server, increasing overall throughput. The are two distinct benefits here- since only one session

entry is created, twice the amount of client sessions are now possible and overall throughput increases

due to decreased CPU load.Another possible benefit is that using DSR with the SI400, we reduce the possibility of having

recurring session synchronization issues between the barrel processors.

DSR / Non-DSR RecommendationsIt is recommended that where possible, DSR mode is used. Instances where DSR may not be suitable is

when providing load balancing for protocols that require a back channel; for example, some

streaming media applications. DSR mode also depends on the capability of configuring a loopbackaddress on the real server operating system, which may not always be possible (i.e. DEC). The benefits

of using DSR are twofold, literally. DSR provides for twice the session capacity on the switch and

twice the overall performance.

DSR is enabled on a per port basis under the virtual server definition. To configure DSR, use thefollowing cli commands:

Serveriron(config)# server virtual VIP 1.1.1.1

Serveriron(config-virtual-VIP)# port http dsr*For FTP, particularly non-passive mode FTP, sticky and concurrent must also be configured for

the port when using DSR.

Port TranslationOne-to-Many

Port translation is used when the actual real server port does not match the port accepting connections

on the VIP. The ServerIron translates the port numbers based on the binding from the VIP portnumber to the port on the real server. This is the default behavior, and is referred to sometimes as a

one to many configuration.

Many-to-One


15/36

By default, the ServerIron will not allow many different ports on the VIP(s) to be bound to the same

port on a real server. Disabling the port translation on the VIPs, as well as healthchecks on the fake

ports provides a work-around. The following illustrates this method:

server real1 1.1.1.10

port 567

port 568port http

!

server real2 1.1.1.11port 567

port 568

port http

!server real3 1.1.1.12

port 567

port 568port http

!

server real4 1.1.1.13port 567

port 568

port http!

server virtual VIP1 1.1.1.1

port http

no port http translatebind http real1 568 real2 568 real3 568 real4 568

!

server virtual VIP2 1.1.1.2port http

no port http translate

bind http real1 567 real2 567 real3 567 real4 567

The device will accept HTTP requests on both VIPs, and distribute the connection untranslated to all

three real servers on port 80.

*Note: Round Robin must be the predictor set.

DO NOT CONFIGURE HEALTHCHECKS ON THE FAKE REAL SERVER PORTS!

DSR mode cannot be used in conjunction with port translation.

Advanced Server Load BalancingLayer 7 Switching

To be added in later revisions.

Transparent CachingTo be added in later revisions.

Code Matrix

The following is the recommended code revisions for Foundry Layer 4 switchs:


16/36

Product

Code Revision

PatchSI

7.1.06

SI/XL

7.3

05d*BSI

7.1

21b**

SI4007.2

26f***

*Patch revision 05d: Fixes x.x.x.x

**Patch revision 21b has dm flash fix.***Patch revision 26f fixes an ACL range issue.

Foundry release notes are in the customer technical support area on the Foundry website. Patch revisionrelease notes can be obtained the Foundry onsite support engineers.

System Maintenance

Software Maintenance

Upgrading Code

To upgrade code on the XL and BSI devices, the new code is loaded via TFTP into flash and

subsequently rebooted. To upgrade code on the XL and BSI device, use the following command:

Serveriron# copy tftp flash (tftp server IP) (image name) primary / secondary

For the SI400, there are two code releases required- one for the management Processor, and one for the

barrel processors. IT IS REQUIRED THAT BOTH RUN THE SAME REVISION / PATCH LEVEL.

The barrel processors must be loaded first. To upgrade the code on the barrel processor, use thefollowing command:

Serveriron# wsm copy tftp flash (tftp server IP) (image name) primary / secondary

To load the management processor, use the following command:

Serveriron# copy tftp flash (tftp server IP) (image name) primary / secondary

*NOTE If you are upgrading code, please refer to the release notes for that revision to check for bootcode dependencies.

Hardware Maintenance / Caveats


17/36

WSM4 / M5 Chassis Requirements

The SI400 requires -A lincards. These include both the B8G-A (8 port Gigabit Ethernet Module) and

the B24E-A (24 port 10/100 Ethernet Module). There is no restriction on placement of the various

modules within the device; however, it is recommended for consistency the Management board beplaced into Slot 1.

A BigIron Chassis can be upgraded to an SI400 by replacing the Management board, but as stated

above, the appropriate linecards are required.Upgrading a BigIron L2/L3 device for SLB/BSI Requirements

An M3 or M4 management board with an EEPROM upgrade is required in order to run the BSI code.

Operational MaintenanceRemoving A Real Server (or port) from rotation

Temporarily removing a real server from rotation

Disable the port on the real (the force-delete setting forces session clearing within two minutes)

Permanently removing a real server

To permanently remove a real server (or port) from the configuration, perform the following steps:

Under the real server configuration, enter the following:

Serveriron(config-rs-1)# disable port [port #]

Under the virtual server configuration, enter the following:

Serveriron(config-vir-1)# no bind [port #] [real-server] [port #]

SummaryThis document hopes to take some of the cloudiness out of configuring a Foundry device for SLB. If

there are any suggestions, recommendations please contact Tony Cooper ([email protected])

or

Appendix A Case Studys: Example Configurations and Caveats

Basic SLB Configuration Examples

Basic SLB (with Layer 7 healthchecks enabled)ver 07.2.25T22

module 1 bi-0-port-wsm-management-module

module 2 bi-8-port-gig-modulemodule 3 bi-24-port-copper-module

module 4 bi-24-port-copper-module

!

no global-stp!

!

server force-deleteserver reassign-threshold 250

server tcp-age 6

server syn-def 6server router-ports ethernet 2/1

server router-ports ethernet 2/2

!server real wads-d01b 205.188.165.65

port http

port http keepalive


18/36

port http url "GET /admin/hello"

!

server real wads-d02b 205.188.165.66

port httpport http keepalive



port http

port http keepaliveport http url "GET /admin/hello"

!





port http


!

server real wads-d06b 205.188.165.70port http

port http keepalive



port http


!


port http keepalive



port http


!


port http keepalive

port http url "GET /admin/hello"!


port http


19/36

port http keepalive


!


port http keepalive





!


port http keepalive






port http

port http keepalive






port http

port http keepalive






port http


!



20/36

port http

port http keepalive



port http


!


port http keepalive



port http


!


port http keepalive



port http


!


port http keepalive



port http


!


port http keepalive





!


21/36


port http

port http keepalive






port http

port http keepalive






port http


!





port http


!





port http


!


port http keepalive



22/36

!

!

server virtual ads.web.aol.com 205.188.165.121

predictor round-robinport http

bind http wads-d40b http wads-d39b http wads-d38b http wads-d37b http

bind http wads-d36b http wads-d35b http wads-d34b http wads-d33b httpbind http wads-d32b http wads-d31b http wads-d30b http wads-d29b http






!

!no spanning-tree

qd-100 160 31 31 31

qd-1000 160 31 31 31l2-hw-age-dis

system-max tcp-buffer 512

!console timeout 10

enable password-display

enable super-user-password 0 articul8

hostname wadse2-dr5-sw0ip address 205.188.165.122 255.255.255.192

ip default-gateway 205.188.165.125

ip dns domain-name AOL.COMip dns server-address 205.188.152.6 205.188.152.37

ip policy 1 cache tcp 0 global

logging 205.188.151.2logging 172.31.15.13

logging 172.20.20.33

logging facility local5

mac-age 0no flow-control

telnet client 152.163.136.1

telnet client 152.163.136.2telnet client 205.188.151.9




telnet timeout 10snmp-server community 0 Just1SNmPcomuntyRO2bchangd2k+ ro 8

snmp-server contact NetOps

snmp-server location dr5 cz05


23/36

clock summer-time

clock timezone us Eastern

sntp server 205.188.152.1

sntp server 205.188.152.2no web-management

interface e 2/1

no flow-control!

interface e 2/2

no. (concantenated)!

access-list 8 permit 64.12.64.32 0.0.0.15


access-list 8 permit 152.163.129.0 0.0.0.255access-list 8 permit 152.163.136.0 0.0.0.255






access-list 8 permit host 206.222.240.77!

!

end

Basic SLB with DSR Enabled

ver 07.1.18T12

no global-stp!

!

server force-deleteserver reassign-threshold 250

server syn-def 6

server tcp-msl 6

server sticky-age 30!

server port 554

tcp!

!

server real ondmdms1-r01 64.12.38.33port rtsp

port http

port http url "HEAD /"!

server real ondmdms1-r02 64.12.38.34

port rtsp


24/36

port http


!

server real qtdownload1-sd01 64.12.38.35port rtsp

port http


server real qtdownload1-sd02 64.12.38.36

port rtspport http


!

server real qtdownload1-sd03 64.12.38.37port rtsp

port http


!

server virtual ondmdms1-VIP 64.12.38.57predictor weighted

port rtsp

port rtsp dsrport http sticky

port http dsr

bind rtsp ondmdms1-r01 rtsp ondmdms1-r02 rtsp

bind http ondmdms1-r01 http ondmdms1-r02 http!

server virtual vip1-qt 64.12.38.56


port http dsr

!server virtual darwin-vip1 64.12.38.55

port rtsp sticky

port rtsp dsr

port http stickyport http dsr

bind rtsp qtdownload1-sd01 rtsp qtdownload1-sd02 rtsp qtdownload1-sd03 rtsp

bind http qtdownload1-sd01 http qtdownload1-sd03 http qtdownload1-sd02 http!

vlan 1 name DEFAULT-VLAN by port

no spanning-tree!

console timeout 10

enable password-displayenable super-user-password 0 xxxxx

hostname abasketms1-mr1-sw0

ip address 64.12.38.58 255.255.255.224


25/36


ip dns domain-name AOL.COM

ip dns server-address 64.12.66.5 64.12.66.36

logging 172.20.20.33logging 172.20.20.33

logging 152.163.177.103

logging 205.188.151.2logging facility local5

mac-age 0

system-max tcp-buffer 512!






telnet timeout 10

snmp-server community 0 Just1SNmPcomuntyRO2bchangd2k+ ro 8snmp-server contact NetOps

snmp-server location MR1 CH-138

clock timezone us Easternsntp server 64.12.66.33


no web-management

interface e 1speed-duplex 100-full

no flow-control

!interface e 2

speed-duplex 100-full

no flow-control!

interface e 3

speed. (concatenated)

!access-list 8 permit 64.12.64.32 0.0.0.15








access-list 8 permit host 206.222.240.77


26/36

!

!

end

Advanced SLB Configuration ExamplesSLB with Cookie Switching Enabled

server force-delete

server session-limit 1000000server reassign-threshold 100

server syn-def 6

server port 80

tcp

server port 1030tcp

server port 1031tcp

server port 1032tcp

server port 1033tcp

server port 8001

tcpserver router-ports 25 26

!

!server real plogic-sd03 205.188.135.99

port 1032

port 1031port 1031 server-id 1027

port 1030

port 1030 server-id 1027

port httpport http url "HEAD /"

port http l4-check-only

port 1033port 8001


port 1040!

server real plogic-sd04 205.188.135.100

port 1032port 1031


port 1030


27/36


port http


port http l4-check-onlyport http server-id 1028

port 1033


port 1040

!server real plogic-sd05 205.188.135.101

port http


port http l4-check-onlyport http server-id 1029

port 1030

port 1030 server-id 1029port 1031


port 1032port 1033

port 8001


!


port 1032port 1031

port 1030



port http server-id 1025port 1033

port 1041

port 1043

port 1044!


port 1032port 1031

port 1030



port http server-id 1026port 1033

port 1041

port 1043


28/36

port 1044

!




port http server-id 1030!




port http server-id 1031


port 1031




port 8001

port 8001 server-id 1031!


port http

port http url "HEAD /"port http l4-check-only

port http server-id 1032


port 1031



port 1033



!server virtual aolsvc.decisionguides 205.188.135.118

predictor round-robin

port http stickyport http cookie-name "pl_serverid"

port http cookie-switching

bind http plogic-sd03 http plogic-sd04 http plogic-sd05 http!

server virtual cssvc.decisionguides 205.188.135.119



29/36

port http sticky

port http cookie-name "pl_serverid"


bind http plogic-sd03 1030 plogic-sd04 1030 plogic-sd05 1030!

server virtual webcenter.decisionguides 205.188.135.120

predictor round-robinport http sticky

port http cookie-name "pl_serverid"


server virtual webcenter.netscape 205.188.135.121





server virtual aolsvc.recipes.aol.com 205.188.135.114


bind http plogic-sd01 http plogic-sd02 http

!server virtual cssvc.recipes.compuserve.com 205.188.135.115


port http

bind http plogic-sd01 1030 plogic-sd02 1030!

server virtual webcenter.recipes.aol.com 205.188.135.116


bind http plogic-sd01 1031 plogic-sd02 1031

!server virtual webcenter.recipes.netscape.com 205.188.135.117


port http


server virtual cookinglight.recipes.aol.com 205.188.135.113


bind http plogic-sd01 1033 plogic-sd02 1033

!server virtual autotrader.decisionguides 205.188.135.112




bind http plogic-sd03 1033 plogic-sd04 1033 plogic-sd05 1033


30/36

!

server virtual aolsvc.illnesses 205.188.135.109


port httpbind http plogic-sd01 1041 plogic-sd02 1041

!

server virtual webcenter.illnesses 205.188.135.110predictor round-robin

port http


server virtual cssvc.illnesses 205.188.135.111


port httpbind http plogic-sd01 1044 plogic-sd02 1044

!

server virtual decisionguides.websys.aol.com 205.188.135.106predictor round-robin

port http sticky

port http cookie-name "pl_serverid"port http cookie-switching

bind http plogic-sd03 8001 plogic-sd04 8001 plogic-sd05 8001

!server virtual staging.websys.aol.com 205.188.135.107


port http sticky

port http cookie-name "pl_serverid"port http cookie-switching

port 1030

port 1030 cookie-name "pl_serverid"port 1030 cookie-switching

port 1032

port 1032 cookie-name "pl_serverid"port 1032 cookie-switching

port 1033

port 1033 cookie-name "pl_serverid"

port 1033 cookie-switchingport 8001


port 8001 cookie-switchingport 1031


port 1031 cookie-switchingbind http plogic-sd07 http plogic-sd08 http

bind 1030 plogic-sd07 1030 plogic-sd08 1030

bind 1032 plogic-sd07 1032 plogic-sd08 1032bind 1033 plogic-sd07 1033 plogic-sd08 1033




31/36

!

server virtual careerfinder.web.aol.com 205.188.135.108





vlan 1 name DEFAULT-VLAN by port

no spanning-tree!

console timeout 10

enable password-display

enable super-user-password 0 articul8hostname plogic1-dr1-sw0

ip address 205.188.135.122 255.255.255.224

ip default-gateway 205.188.135.125ip dns domain-name AOL.COM

ip dns server-address 205.188.152.6 205.188.152.37

logging 205.188.151.2logging 172.31.15.13

logging 172.20.20.33

logging facility local5mac-age 0

no flow-control






telnet timeout 10snmp-server community 0 Just1SNmPcomuntyRO2bchangd2k+ ro 8

snmp-server contact NetOps

snmp-server location AU36

clock summer-timeclock timezone us Eastern


sntp server 205.188.152.2no web-management

interface e 1

speed-duplex 100-fullno flow-control

!

interface e 2speed (concatenated)

!



32/36








access-list 8 permit 207.200.78.0 0.0.0.255access-list 8 permit host 206.222.240.77

!

!

endSLB with URL Switching Enabled

ver 07.1.18T12

no global-stp!

!

server force-deleteserver syn-def 6

server tcp-msl 6

server router-ports 25 26!

url-map "policya"

method pattern

default 3match "MC_RSC=7" 2

match "MC_RSC=5" 1

!!

server real mc-s01 64.12.144.1

max-conn 10000port 12443

port http


port sslport 1080

port 1443

port 2080port 2443

port 3080

port 3443port 4080

port 4443

port 5080port 5443

port 6080

port 6443


33/36

port 7080

port 7443

port 8080

port 8443port 9080

port 9443

port 10080port 10443

!

server real mc-s03 64.12.144.2max-conn 10000

port 12443

port http

port http url "HEAD /"port ssl

port 1080

port 1443port 6443

port 6080

port 4443port 4080

port 3443

port 3080port 2080

port 2443

port 5080

port 5443port 7080

port 7443

port 8080port 8443

port 9080

port 9443port 10080

port 10443

!


port 13443


port 14080


port 12543


port 10050 group-id 1 1 3 3

port 12443


34/36

port http


port ssl


port 11443

port 12080!

server real screen2-s03 64.12.144.4

port http disableport http url "HEAD /"

port ssl

!


port 13443


port 14080


port 12543


port 10050 group-id 2 2 3 3

port 12443


port ssl


port 12080

port 11443!

server real screenname3 64.12.144.6

port http

port http url "HEAD /"port ssl

!

server real screenname-03-br 64.12.144.16port http


port ssl!

server virtual sandbox.screenname.aol.com 64.12.144.22


port http url-map "policya"

port http cookie-name "MC_RSC"


35/36

port http url-cookie-switching

port ssl sticky

bind http mc-s07 10050 mc-s05 10050

bind ssl mc-s05 ssl mc-s07 ssl!

server virtual my.screenname.aol.com 64.12.144.3

predictor round-robinport ssl sticky

port http

bind ssl mc-s01 12443 mc-s03 12443bind http mc-s01 10080 mc-s03 10080

!

server virtual sandboxt.screenname.aol.com 64.12.144.19


port ssl sticky

bind http mc-s05 10010 mc-s07 10010bind ssl mc-s05 10020 mc-s07 10020

!

server virtual aolcom.screenname.aol.com 64.12.144.23predictor round-robin

port http

port ssl stickybind http mc-s05 14080 mc-s07 14080

bind ssl mc-s05 14443 mc-s07 14443

!

server virtual login.screenname.aol.com 64.12.144.24predictor round-robin

port http

port ssl stickybind http mc-s05 13080 mc-s07 13080

bind ssl mc-s05 13443 mc-s07 13443

!vlan 1 name DEFAULT-VLAN by port

no spanning-tree

!

console timeout 10enable password-display

enable super-user-password 0 articul8

hostname snagreg-mr6-sw0ip address 64.12.144.26 255.255.255.224


ip dns domain-name AOL.COMip dns server-address 64.12.66.5 64.12.66.36

logging 172.20.20.33

logging 152.163.177.103logging 205.188.151.2

logging 172.31.15.13

logging 172.20.20.33


36/36

logging facility local5

mac-age 0

no flow-control

system-max tcp-buffer 512!






telnet timeout 10

snmp-server community 0 Just1SNmPcomuntyRO2bchangd2k+ ro 8snmp-server contact NetOps

snmp-server location MR6 CQ-605

clock summer-timeclock timezone us Eastern


no web-managementinterface e 1

speed-duplex 100-full

no flow-control!

interface e 2

speed(concatenated)

!access-list 8 permit 64.12.64.32 0.0.0.15








access-list 8 permit host 206.222.240.77

!!

end

Aol Internal Networking Manual

Documents

Transcript of Aol Internal Networking Manual