COLLABORATE 12 1

Oracle Database 11g SCAN: Sharing a Successful Implementation Experience

Yury Velikanov, Sr. Oracle DBA

The Pythian Group,

Todd Carlson, Sr. Manager, DBA Team

World Wide Technology

Introduction

The Single Client Access Name (SCAN) is a concept that makes a database deployment easy on Oracle Database 11g R2 grids and completes the level of abstraction from the application perspective. While it makes an Oracle network configuration simpler on a client side, however, an additional layer of complexity is added on the server side to support that simplicity. This paper explains the SCAN concepts, walks you through the main steps of the process of establishing a connection, and gives troubleshooting hints you may use to investigate SCAN-related issues.

One of the main challenges for an Oracle DBA who starts to manage a SCAN infrastructure is the fact that it consists of traditional Oracle Net components put together in a new way. Some usual ways of managing these components can't be used in a SCAN configuration. Oracle DBAs should adjust their "habits" to ensure successful SCAN infrastructure operations. This paper helps the Oracle DBA to start working with the SCAN configuration and to use basic troubleshooting techniques.

Oracle Client Connection Process Using SCAN Infrastructure

This section introduces an Oracle client's connection flow in the SCAN configuration. The section's goal is to give you a good overview of the process of establishing a connection. It should prepare you for your first SCAN infrastructure implementation and troubleshooting efforts.

Single Client Access Name (SCAN)

`

7.

5.

6.

nameserver

10.10.192.12 (prim)

10.10.192.25 (sec)

Storage ASM

DB

eth0

Public IP (OS/Fixed)

Virtual IP (Cluster/DHCP)

SCAN1 IP (Cluster/DHCP)

eth1

Private IP (OS/ Fixed or DHCP)

SCAN1 Listener

DB Listener

SSH

DNS

DHCP

clustgrid-prod.yourdomain.com NS gns.clustgrid-prod.yourdomain.com gns.clustgrid-prod.yourdomain.com 172.30.193.201

GNS virtual IP (Cluster/Fixed) Grid Domain Service gns.clustgrid-prod.yourdomain.com 10.10.1.201 (fixed)

scan.clustgrid-prod.yourdomain.com

RAC host 1

at least 5 free IPs 3 for SCAN IPs, 2 for VIPs, optional 2 Private IPs

nameserver

10.10.192.12 (prim)

10.10.192.25 (sec)

eth0

Public IP (OS/Fixed)

Virtual IP (Cluster/DHCP)

eth1

Private IP (OS/ Fixed or DHCP)

DB Listener

SSH

RAC host 2

SCAN3 IP (Cluster/DHCP) SCAN3 Listener

SCAN2 IP (Cluster/DHCP) SCAN2 Listener

Interconnect Traffic

2.

4.

3.

1.

COLLABORATE 12 2

Step 1: To connect to a service (Oracle Instance or Database), the Oracle Client should know the cluster's SCAN domain name. We use scan.clustgrid-prod.yourdomain.com as the SCAN name as an example in this paper. The SCAN name is configured at the cluster initial setup time. Step 2: The Oracle Client sends a request to a DNS service providing the SCAN name and receives 3 IP addresses associated to the SCAN name. On the DNS level, the IPs could be configured statically or dynamically (GNS). Step 3: If there is a configured GNS (Oracle Grid Naming Service) then the DNS delegates the request processing to the GNS service. I would like to mention a few points here. First of all, if you are wondering whether to use the GNS configuration, then the most probable answer is, "No! You don't need it." The GNS service ensures the ability to add nodes to your cluster without specifying IP addresses for such components like VIPs and Cluster Interconnect IPs. Those are obtained dynamically from the DHCP service during the node startup process. The GNS is used to inform the DNS about new components and associated IP addresses. The second point is that DNS' and GNS' integration and communications are a bit more complex than just described. I will not go into the details, to keep the description simple. Step 4: DNS returns the 3 IP addresses associated with the SCAN domain name in a round robin fashion. This way, each subsequent client request gets the same 3 IP addresses but in a different order. Step 5: The Oracle Client sends a TNS request (IP/PORT/Service) to one of the SCAN listeners. It starts with the first IP address in the list returned by DNS service. Note that clients older than Oracle 11G don't know how to work with 3 IPs returned by DNS, and use only the first IP returned. If this IP for one reason or another isn't available, the connection attempt fails. Starting from version 11G, the Oracle Client uses the second and the third IP addresses if the first connection attempt fails. Step 6: The SCAN listener knows about all Oracle services served by instances running on each cluster's nodes. If there are multiple instances that could serve the incoming connection, the SCAN listeners could be configured to balance incoming connections' load. The SCAN listener forwards the Oracle Client's connection (providing IP and PORT) to a local DB listener running on the less loaded node. Step 7: The Oracle Client connects to the local DB Listener and requests a connection to the Oracle Instance. The local DB listener uses a traditional process to fork a database foreground process, and connects the Oracle Client with that process.

Oracle Instance Registration Process in the SCAN Infrastructure

Now that we have introduced the client's connectivity flow, let's have a look at how an Oracle Instance is registered in the SCAN infrastructure from the perspective of the backend. Let's discuss the main components and configuration items involved in the process. An Oracle database instance should get registered within the SCAN infrastructure before it can start accepting incoming clients' connections.

COLLABORATE 12 3

Node 1

LISTENER (default)

remote_listener

local_listener

LISTENER_ERP (custom)

Node 2

DNS resolves scan-cluster_a.mycomany.com to 3 SCAN IP addresses

SCAN (remote) listeners register and distribute information about local listeners serving a SERVICE

Oracle Instance: 1. registers SERVICES it running in LOCAL listeners 2. registers LOCAL listeners serving a service in SCAN listeners

LISTENER LISTENER LISTENER

1.

2.

service_name

First of all, it is very important to understand the difference between the remote_listener and local_listener init.ora parameters. These parameters play important and slightly different roles in the instance SCAN registration process.

- The Oracle Instance (in particular PMON - process monitor) uses the list of listeners' addresses specified by the local_listener parameter to contact each listener from the list and "says": "Hey Listener! I am instance X. I run the following services. I am the local instance and run on the same host that you". From now on, if the local listener receives a new session request to connect to one of the services the instance provides it spawns an Oracle foreground process, connects it to the instance X and passes a TCP/IP socket to the client session. Please note that the listener doesn't check if the instance is local or not by the time the PMON process provides the information. It just blindly believes what was told. Therefore the local_listener parameter must point only to the local listener. Otherwise, some of the incoming connections may fail.

- During the next step, the PMON process reads the remote_listener parameter and connects to each of the listeners specified by the parameter. It informs the remote listeners about any services and the corresponding local listeners the instance runs. It is important to mention that the SCAN listeners expect an Oracle Instance only to report local listeners. SCAN listeners don’t check the information sent by the PMON at the time of registration. SCAN listeners rely on the information provided and start forwarding incoming connections to the local listeners immediately. If the listener isn't the local listener, connection attempts fails.

- To reiterate what was said: the SCAN and local listeners don’t check the information provided by the Oracle Instance (PMON). The listeners rely on the information provided and "blindly" send sessions to other (local) listeners, or try to spawn a new Oracle process to serve the incoming connection. Therefore it is important to make sure the right values are set for both remote_listener and local_listener parameters.

The Oracle Instance (PMON) updates remote and local listeners on services it is running during the following events:

- at start up and shutdown at regular intervals (up to 60 seconds)

COLLABORATE 12 4

- at the time the "ALTER SYSTEM REGISTER" command is issued

The following is Oracle Instance's short registration flow:

Step 1: Straight after a startup and on a regular basis thereafter, the Oracle Instance reads the service_name init.ora parameter to determine what services it should provide to the outside world. We should mention that in the 11G infrastructure, information about Oracle services is managed by Oracle Clusterware processes. Therefore, don’t try to change this information using the "ALTER SYSTEM..." command. Use the srvctl utility instead.

Step 2: The Oracle Instance reads the local_listener parameter and registers the list of services obtained during the first step within the local listener.

Step3. The Oracle Instance reads the remote_listener parameter and registers the local listener along with services it is serving within remote (SCAN) listeners.

Oracle Net: Easy Connect

Before you start implementing or troubleshooting the SCAN infrastructure, we strongly suggest that you get comfortable with the Oracle Net Easy Connect naming method. This is one of the many Oracle Net naming methods. A naming method is nothing more than a way to describe a network configuration that an Oracle client should know to get connected to an Oracle Instance. A more commonly used method is the local naming method, where you edit the tnsnames.ora configuration file to specify the listener's address and services to connect to. Easy Connect is a less popular method. In the case of Easy Connect, the network configuration isn't stored in a separate configuration file, but provided in one relatively short connect string. This method is used extensively in the SCAN configuration; therefore it is important to understand it well.

We suggest that you read through the "Using the Easy Connect Naming Method" section of the "8 Configuring Naming Methods" chapter of the "Oracle® Database Net Services Administrator's Guide" book from the standard Oracle 11GR2 Documentation Set.

As an example, you may get confused by looking at the initial value of the remote_listener init.ora parameter that is set to "scan.clustgrid-prod.yourdomain.com" (or your cluster SCAN name). This is nothing but the following tnsnames.ora configuration:

LISTNER=(ADDRESS = (PROTOCOL = TCP)(HOST = scan.clustgrid-prod.yourdomain.com)(PORT = 1521))

SCAN Troubleshooting Hints

If you have read the previous sections carefully, you may find this section to be redundant to some extent, as most of the information provided below is related to the components discussed already.

service names - check the service_name init.ora parameter to start with and make sure that it corresponds to the configuration you expect to see. You can use the following traditional command to retrieve the instance'-side information for the troubleshooting purposes. Compare it with the next command output and your oracle client-side configuration. Please notice that you should check each node in your cluster.

SQL> show parameter service_name

NAME TYPE VALUE

-------------------- ----------- --------------------------------------------------

service_names string DEVERP_CDC.GGT.COM, SYS$APPLSYS.WF_CONTROL.DEVERP.

WORLD, SYS$STREAMS_ADMIN.CDC$Q_ERP.DEVERP.WORLD, D

EVERP_WEBM.GGT.COM, DEVERP_W2T_B2B.GGT.COM, DEVERP

_RFUI.GGT.COM, DEVERP_IBI.GGT.COM, DEVERP_GENERAL.

W2T.COM, DEVERP_BI.GGT.COM, DEVERP_APEX.GGT.COM, D

EVERP_10g, DEVERP1, DEVERP

SQL>

COLLABORATE 12 5

Do not modify the service_name parameter using the "ALTER SYSTEM SET ..." command. Use the srvctl utility to retrieve and modify service-related configuration instead. The cluster sets the service_names parameter dynamically at the instance's start up time and each time it is modified. The following example demonstrates how you can retrieve the services' configuration using the srvctl utility.

srvctl config service -d <DB Name>

…

Service name: DEVERP_APEX.GGT.COM

Service is enabled

Failover type: NONE

Preferred instances: DEVERP1

Available instances: DEVERP1,DEVERP2,DEVERP3,DEVERP4,DEVERP5,DEVERP6

…

local_listener and remote_listener parameters - We can't stress enough: the local_listener parameter must only point to the local (running on the same nodes as the instance it is related to) listener address, and the remote_listener parameter should only point to the SCAN listeners. The other important point to mention is that the instance won't start if there is a syntax or misconfiguration issue in the parameters' settings. For example, typically you will find that the remote_listener parameter points to the SCAN listeners using the Oracle Net: Easy Connect syntax (remote_listener=scan.clustgrid-prod.yourdomain.com). If you forget to include an (or at some point in time remove) NAMES.DIRECTORY_PATH=(..., EZCONNECT,...) line in the sqlnet.ora file, the instance will fail to start next time you try to restart it.

Oracle Listeners - Make sure that listeners in your SCAN infrastructure are running under the grid OS user. One of the typical mistakes a traditional Oracle DBA makes when connecting to the 11GR2 cluster for the first time is starting the local listener under the oracle OS user. The listener starts listening on all IP addresses available on the server, including the entire SCAN and VIP IP addresses, on the default 1521 TCP port. Unfortunately, there is nothing preventing the Oracle DBA from doing this. However, it introduces a lot of different issues in the SCAN configuration. As with the services, listeners should be managed using the srvctl cluster management utility. Starting from the 11GR2 version, the listener.ora file is managed by the cluster itself. Therefore, try to avoid editing the listener.ora file manually unless it is absolutely necessary. For troubleshooting purposes, you still can use the lsnrctl status <name e.g. LISTENER_SCAN1 or LISTENER > or lsnrctl status <name e.g. LISTENER_SCAN1 or LISTENER > commands to make sure that the expected services are registered under each listener.

DNS - as Oracle clients become dependent on the SCAN domain name resolution, the DNS service becomes a critical component of the connectivity process. You should make sure that the DNS returns the expected list of the SCAN IP addresses each time a SCAN resolution request is coming through. Some old Windows versions may cache in local memory domain names that have been resolved before. You should keep this in mind while troubleshooting the SCAN connectivity issues. The other important verification you should run through is to make sure that the secondary DNS service is configured in the same way in the SCAN context. Sometimes you may find that some of a cluster's users can't connect to its services. This is a good indicator that there may be a configuration issue with the secondary DNS.

Conclusion

The information provided in this paper should be sufficient to get comfortable with a typical SCAN configuration and to start troubleshooting related issues successfully. While it isn't a comprehensive technology description, it is based on several real life implementation projects and covers typical problems Oracle DBAs face starting to work with the SCAN configuration for the first time.