How to Configure LUNs for ASM Disks using WWID, DM-Multipathing, and ASMLIB on RHEL 5/OEL 5 [ID 1365511.1]

To Bottom

In this Document  Goal  Solution     1.   Configure SCSI_ID to Return Unique Device Identifiers:      2.   Configure LUNs for ASM :      4.   Create ASM diskgroups:      5.   To make the disk available enter the following commands:                  7.   Ensure that the allocated devices can be seen in /dev/mpath:      8.   Ensure that the devices can be seen in /dev/mapper:      9.   Check the device type:      10.   Setup the ASM parameter (ORACLEASM_SCANORDER), in the file for ASMLIB configuration, /etc/sysconfig/oracleasm, for forcing ASM to bind with the multipath devices  References

Applies to: Oracle Server - Enterprise Edition - Version: 10.2.0.4 to 11.2.0.2 - Release: 10.2 to 11.2Linux x86-64Oracle Server Enterprise Edition - Version: 10.2.0.4 to 11.2.0.2

GoalThis document details "how to" steps by using an example that creates devices for Automatic Storage Management (ASM) using World Wide Identifier (WWID), DM-Multipathing, and ASMLIB, utilizing a Hitachi Storage Sub-system.  The simplified "how to" steps are for Red Hat Enterprise Linux version 5 (RHEL 5) and Oracle Enterprise Linux version 5 (OEL 5) on Linux x86-64 for preparing storage to use ASM.

Each multipath device has a World Wide Identifier (WWID), which is guaranteed to be globally unique and unchanging. By default, the name of a multipath device is set to its WWID. Alternately, you can set the user_friendly_names option in the multipath configuration file, which sets the alias to a node-unique name of the form mpathn.  When the user_friendly_names configuration option is set to yes, the name of the multipath device is set to /dev/mpath/mpathn. Configuring multipathing by modifying the multipath configuration file, /etc/multipath.conf, will

not be addressed in this document.

The WWID is a persistent, system-independent ID that the Small Computer Storage Interface (SCSI) Standard requires from all SCSI devices.  Each disk attached to a Linux-based server has a unique SCSI ID. The WWID identifier is guaranteed to be unique for every storage device, and independent of the path that is used to access the device.  This identifier can be obtained by issuing a SCSI Inquiry to retrieve the Device Identification Vital Product Data (page 0x83) or Unit Serial Number (page 0x80). The mappings from these WWIDs to the current /dev/sd names can be seen in the symlinks maintained in the /dev/disk/by-id/ directory.    

In this document, the accessible disks are connected via a Host Bus Adapter Card (HBA) to a Storage Area Network (SAN) or switch. If the disks are attached via Hitachi SAN, the path and port information is also extracted.  Disk arrays that are grouped together as Logical Unit Number (LUN) storage in SANs can also present themselves as SCSI devices on Linux servers. The command, "fdisk -l", lists attached SCSI disk devices, including those from a SAN.   Multiple devices share common SCSI identifiers.

Automatic Storage Management Library (ASMLIB) driver is a support library for the Automatic Storage Management (ASM) feature of the Oracle Database and is available for the Linux operating system.  This document discusses usage with Red Hat Enterprise Linux version 5 and Oracle Enterprise Linux version 5.

Solution

1.  Configure SCSI_ID to Return Unique Device Identifiers:

1a.  Whitelist SCSI devices

(System Administrator's Task)

Before being able to configure udev to explicitly name devices, SCSI_ID (scsi_id(8)) should first be configured to return their device identifiers. SCSI commands are sent directly to the device via the SG_IO ioctl interface.  Modify the /etc/scsi_id.config file - add or replace the 'option=-b' parameter/value pair (if exists) with 'option=-g', for example:

# cat /etc/scsi_id.configvendor="ATA",options=-p 0x80options=-g

1b.  List all SCSI devices

Using the command SCSI_ID for each /block/a-h (for example, for /dev/sda we type scsi_id -g -s /block/sdb) generates the output:SATA WD2502ABYS-23B7 WD-WCAT1H504691 SATA HUA721075KLA330 GTA260P8H8893E360060e80045b2b0000005b2b000006c4360060e80045b2b0000005b2b000006d8

360060e80045b2b0000005b2b00001007360060e80045b2b0000005b2b00001679360060e80045b2b0000005b2b0000163c

The two initial SCSI ids represent the local disks (/dev/sda and /dev/sdb).  The remaining five represent the SCSI ids of the fibre channel attached LUNs.  As a subset of the output string, scsi_id generates for the fibre LUNs matches the World Wide Identifier (WWID).  A simple example would be a disk connected to two Fibre Channel ports. Should one controller, port or switch fail, the operating system can route I/O through the remaining controller transparently to the application, with no changes visible to the applications, other than perhaps incremental latency.

Important:  If Real Application Clusters (RAC), Clusterware devices must be visible and accessible to all cluster nodes. Typically, cluster node operating systems need to be updated in order to see newly provisioned (or modified) devices on shared storage i.e. use '/sbin/partprobe <device>' or '/sbin/sfdisk -r <device>', etc., or simply reboot. Resolve any issues preventing cluster nodes from correctly seeing or accessing Clusterware devices before proceeding.

1c. Obtain Clusterware device unique SCSI identifiers:

Run the scsi_id(8) command against Clusterware devices from one cluster node to obtain their unique device identifiers. When running the scsi_id(8) command with the -s argument, the device path and name passed should be that relative to sysfs directory /sys/ i.e. /block/<device> when referring to /sys/block/<device>. Record the unique SCSI identifiers of Clusterware devices - these are required later configuring multipathing, for example:

# for i in `cat /proc/partitions | awk {'print $4'} |grep sd`; do echo "### $i: `scsi_id -g -u -s /block/$i`"; done...### sdh: 360060e80045b2b0000005b2b0000163c### sdh1:### sdi: 360060e80045b2b0000005b2b0000163c### sdi1:...### sdk: 360060e80045b2b0000005b2b00001679### sdk1:...### sdm: 360060e80045b2b0000005b2b000006c4### sdm1:### sdn: 360060e80045b2b0000005b2b000006d8### sdn1:### sdo: 360060e80045b2b0000005b2b00001007### sd01:...### sdz: 360060e80045b2b0000005b2b00001679### sdz1:From the output above, note that multiple devices share common SCSI identifiers. It should now be evident that devices such as /dev/sdh and /dev/sdi refer to the same shared storage device (LUN).

Another command can be used for listing the SCSI identifiers.# ll /dev/disk/by-id/lrwxrwxrwx 1 root root 9 Jun 27 07:17 scsi-3600508e000000000158d6d2169801c0e -> ../../sdalrwxrwxrwx 1 root root 10 Jun 27 07:17 scsi-3600508e000000000158d6d2169801c0e-part1 -> ../../sda1lrwxrwxrwx 1 root root 10 Jun 27 07:17 scsi-3600508e000000000158d6d2169801c0e-part2 -> ../../sda2lrwxrwxrwx 1 root root 10 Jun 27 07:17 scsi-3600508e000000000158d6d2169801c0e-part3 -> ../../sda3lrwxrwxrwx 1 root root 10 Jun 27 07:17 scsi-3600508e000000000158d6d2169801c0e-part4 -> ../../sda4lrwxrwxrwx 1 root root 10 Jun 27 07:17 scsi-3600508e000000000158d6d2169801c0e-part5 -> ../../sda5lrwxrwxrwx 1 root root 10 Jun 27 07:17 scsi-3600508e000000000158d6d2169801c0e-part6 -> ../../sda6lrwxrwxrwx 1 root root 10 Jun 27 07:17 scsi-3600508e000000000158d6d2169801c0e-part7 -> ../../sda7lrwxrwxrwx 1 root root 10 Jun 27 07:17 scsi-360060e80045b2b0000005b2b000006b0 -> ../../sdaalrwxrwxrwx 1 root root 10 Jun 27 07:17 scsi-360060e80045b2b0000005b2b000006b0-part1 -> ../../sdl1lrwxrwxrwx 1 root root 10 Jun 27 07:17 scsi-360060e80045b2b0000005b2b000006c4 -> ../../sdablrwxrwxrwx 1 root root 11 Jun 27 07:17 scsi-360060e80045b2b0000005b2b000006c4-part1 -> ../../sdab1lrwxrwxrwx 1 root root 9 Jun 27 07:17 scsi-360060e80045b2b0000005b2b000006d8 -> ../../sdnlrwxrwxrwx 1 root root 10 Jun 27 07:17 scsi-360060e80045b2b0000005b2b000006d8-part1 -> ../../sdn1

1d.  Run fdisk to create partitions for ASM disks:

(System Administrator's Task)

# fdisk -l

Disk /dev/sdi: 590.5 GB, 590565212160 bytes255 heads, 63 sectors/track, 71798 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System/dev/sdi1 1 130543 1048586616 83 Linux

MUST run fdisk on respective devicesfdisk /dev/?Example:

# fdisk /dev/sdhCommand (m for help): n

Command actione extendedp primary partition (1-4)p

Partition number (1-4): 1First cylinder (1-1011, default 1): [use default]Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-1011, default 1011): [use default]Using default value 1011

Command (m for help): w

The partition table has been altered!Calling ioctl() to re-read partition table.Syncing disks.

1e.  Run the fdisk(8) and/or 'cat /proc/partitions' commands to ensure devices are visible. (If Real Application Clusters (RAC), Clusterware devices are visible on each node.)  For example:

# cat /proc/partitionsmajor minor #blocks name8 0 142577664 sda8 1 104391 sda18 2 52428127 sda28 3 33551752 sda38 4 1 sda48 5 26218048 sda58 6 10482381 sda68 7 8385898 sda78 16 263040 sdb8 17 262305 sdb18 32 263040 sdc8 33 262305 sdc18 48 263040 sdd8 49 262305 sdd18 64 263040 sde8 65 262305 sde18 80 263040 sdf8 81 262305 sdf18 96 263040 sdg8 97 262305 sdg18 112 576723840 sdh8 113 576709402 sdh18 128 576723840 sdi8 129 576709402 sdi18 144 576723840 sdj8 145 576709402 sdj18 160 52429440 sdk8 161 52428096 sdk1

8 176 524294400 sdl8 177 524281275 sdl18 192 524294400 sdm8 193 524281275 sdm1...65 208 262147200 sdad65 209 262132605 sdad165 224 262147200 sdae65 225 262132605 sdae1253 0 524294400 dm-0253 1 524294400 dm-1253 2 524294400 dm-2253 3 262147200 dm-3253 4 262147200 dm-4253 5 263040 dm-5253 6 263040 dm-6253 7 263040 dm-7253 8 263040 dm-8253 9 263040 dm-9253 10 263040 dm-10253 11 576723840 dm-11253 12 576723840 dm-12253 13 576723840 dm-13253 14 52429440 dm-14253 15 524281275 dm-15253 16 262305 dm-16253 17 524281275 dm-17253 19 262132605 dm-19253 20 524281275 dm-20

Note:  At this point, if  Real Application Clusters (RAC), each node may refer to the would-be Clusterware devices by different device file names.  This is expected.  Irrespective of which node the scsi_id command is run from, the value returned for a given device (LUN) should always be the same.

2.  Configure LUNs for ASM:

(System Administrator's Task)

2a. Verify Multipath Devices:

Once multipathing has been configured and the multipathd service started, the multipathed devices should now be available.

For detailed multipathing commands, please refer to http://magazine.redhat.com/2008/07/17/tips-and-tricks-how-do-i-setup-device-mapper-multipathing-in-red-hat-enterprise-linux-4/

Update the kernel partition table with the new partition as follow (If Real Application Clusters (RAC), do on each node.):

# /sbin/partprobe

Then verify that all multipaths are active by executing:

# multipath -ll

360060e80045b2b0000005b2b000006c4dm-1 HITACHI,OPEN-V*20[size=500G][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active]\_ 3:0:0:17 sdab 65:176 [active][ready]\_ 1:0:0:17 sdm 8:192 [active][ready]360060e80045b2b0000005b2b000006d8dm-2 HITACHI,OPEN-V*20[size=500G][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active]\_ 3:0:0:18 sdac 65:192 [active][ready]\_ 1:0:0:18 sdn 8:208 [active][ready]360060e80045b2b0000005b2b00001679dm-14 HITACHI,OPEN-V[size=50G][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active]\_ 1:0:0:9 sdk 8:160 [active][ready]\_ 3:0:0:9 sdz 65:144 [active][ready]360060e80045b2b0000005b2b0000312edm-9 HITACHI,OPEN-V[size=257M][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active]\_ 1:0:0:4 sdf 8:80 [active][ready]\_ 3:0:0:4 sdu 65:64 [active][ready]360060e80045b2b0000005b2b00001007dm-3 HITACHI,OPEN-V*5[size=250G][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active]\_ 3:0:0:19 sdad 65:208 [active][ready]\_ 1:0:0:19 sdo 8:224 [active][ready]360060e80045b2b0000005b2b0000163cdm-20 HITACHI,OPEN-V*11 ---> multipathed[size=550G][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active]\_ 1:0:0:7 sdh 8:128 [active][ready]   ---> Required to be [active][ready]\_ 3:0:0:7 sdi  8:112 [active][ready]   ---> Required to be [active][ready]

Note:   DM-Multipath provides a way of organizing the I/O paths logically, by creating a single multipath device on top of the underlying devices.  Each device that comes from Hitachi Storage Sub-system (360060e80045b2b0000005b2b0000163c - WWID) requires two underlying physical devices. These two physical devices have to be partitioned (e.g. sdh1 and sdi1).  Also, the WWID device has to be partitioned (e.g. 3360060e80045b2b0000005b2b0000163cp1). Also, the two physical devices are required to be the same size within that group, both non-partitioned and partitioned, and across nodes in Real Application Clusters (RAC).

In fact, various device names are created and used to refer to multipathed devices, for example:# dmsetup ls | sort360060e80045b2b0000005b2b000006b0 (253, 0)360060e80045b2b0000005b2b000006b0p1 (253, 15)360060e80045b2b0000005b2b000006c4 (253, 1)360060e80045b2b0000005b2b000006c4p1 (253, 17)

360060e80045b2b0000005b2b000006d8 (253, 2)360060e80045b2b0000005b2b000006d8p1 (253, 26)360060e80045b2b0000005b2b0000163c (253,11)360060e80045b2b0000005b2b0000163cp1 (253,20)

# ll /dev/mpath/lrwxrwxrwx 1 root root 7 Jun 27 07:17 360060e80045b2b0000005b2b000006b0 -> ../dm-0lrwxrwxrwx 1 root root 8 Jun 27 07:17 360060e80045b2b0000005b2b000006b0p1 -> ../dm-15lrwxrwxrwx 1 root root 7 Jun 27 07:17 360060e80045b2b0000005b2b000006c4 -> ../dm-1lrwxrwxrwx 1 root root 8 Jun 27 07:17 360060e80045b2b0000005b2b000006c4p1 -> ../dm-17lrwxrwxrwx 1 root root 7 Jun 27 07:17 360060e80045b2b0000005b2b000006d8 -> ../dm-2lrwxrwxrwx 1 root root 8 Jun 27 07:17 360060e80045b2b0000005b2b000006d8p1 -> ../dm-26lrwxrwxrwx 1 root root 7 Jun 27 07:17 360060e80045b2b0000005b2b0000163c -> ../dm-11lrwxrwxrwx 1 root root 8 Jun 27 07:17 360060e80045b2b0000005b2b0000163cp1 -> ../dm-20

# ll /dev/mapper/brw-rw---- 1 root disk 253, 0 Jun 27 07:17 360060e80045b2b0000005b2b000006b0brw-rw---- 1 root disk 253, 15 Jun 27 07:17 360060e80045b2b0000005b2b000006b0p1brw-rw---- 1 root disk 253, 1 Jun 27 07:17 360060e80045b2b0000005b2b000006c4brw-rw---- 1 root disk 253, 17 Jun 27 07:17 360060e80045b2b0000005b2b000006c4p1brw-rw---- 1 root disk 253, 11 Jun 27 07:17 360060e80045b2b0000005b2b0000163cbrw-rw---- 1 root disk 253, 20 Jun 27 07:17 360060e80045b2b0000005b2b0000163cp1

# ls -lR /dev|more/dev:drwxr-xr-x 3 root root 60 Jun 27 07:17 buslrwxrwxrwx 1 root root 4 Jun 27 07:17 cdrom -> scd0lrwxrwxrwx 1 root root 3 Jun 27 07:17 cdrom-hda -> hdalrwxrwxrwx 1 root root 4 Jun 27 07:17 cdrom-sr0 -> scd0lrwxrwxrwx 1 root root 3 Jun 27 07:17 cdrw -> hdalrwxrwxrwx 1 root root 3 Jun 27 07:17 cdrw-hda -> hdalrwxrwxrwx 1 root root 3 Jun 27 07:17 cdwriter -> hdalrwxrwxrwx 1 root root 3 Jun 27 07:17 cdwriter-hda -> hdacrw------- 1 root root 5, 1 Jun 27 07:18 consolelrwxrwxrwx 1 root root 11 Jun 27 07:17 core -> /proc/kcoredrwxr-xr-x 10 root root 200 Jun 27 07:17 cpudrwxr-xr-x 6 root root 120 Jun 27 07:17 diskbrw-rw---- 1 root root 253, 0 Jun 27 07:17 dm-0brw-rw---- 1 root root 253, 1 Jun 27 07:17 dm-1brw-rw---- 1 root root 253, 10 Jun 27 07:17 dm-10brw-rw---- 1 root root 253, 11 Jun 27 07:17 dm-11brw-rw---- 1 root root 253, 12 Jun 27 07:17 dm-12brw-rw---- 1 root root 253, 13 Jun 27 07:17 dm-13brw-rw---- 1 root root 253, 14 Jun 27 07:17 dm-14brw-rw---- 1 root root 253, 15 Jun 27 07:17 dm-15

brw-rw---- 1 root root 253, 16 Jun 27 07:17 dm-16brw-rw---- 1 root root 253, 17 Jun 27 07:17 dm-17brw-rw---- 1 root root 253, 18 Jun 27 07:17 dm-18brw-rw---- 1 root root 253, 19 Jun 27 07:17 dm-19brw-rw---- 1 root root 253, 2 Jun 27 07:17 dm-2brw-rw---- 1 root root 253, 20 Jun 27 07:17 dm-20brw-rw---- 1 root root 253, 21 Jun 27 07:17 dm-21brw-rw---- 1 root root 253, 22 Jun 27 07:17 dm-22brw-rw---- 1 root root 253, 23 Jun 27 07:17 dm-23brw-rw---- 1 root root 253, 24 Jun 27 07:17 dm-24brw-rw---- 1 root root 253, 25 Jun 27 07:17 dm-25brw-rw---- 1 root root 253, 26 Jun 27 07:17 dm-26.../dev/disk/by-label:lrwxrwxrwx 1 root root 10 Jun 27 07:17 1 -> ../../sda5lrwxrwxrwx 1 root root 10 Jun 27 07:17 boot1 -> ../../sda1lrwxrwxrwx 1 root root 10 Jun 27 07:17 optapporacle1 -> ../../sda2lrwxrwxrwx 1 root root 10 Jun 27 07:17 SWAP-sda3 -> ../../sda3lrwxrwxrwx 1 root root 10 Jun 27 07:17 tmp1 -> ../../sda7lrwxrwxrwx 1 root root 10 Jun 27 07:17 var1 -> ../../sda6

3.  Automatic Storage Management Library (ASMLIB) setup:

Note:  For improved performance and easier administration, Oracle recommends that the Automatic Storage Management Library (ASMLIB) driver be used instead of raw devices to configure Automatic Storage Management disks.

3a. Verify that ASMLIB has not been installed already before installing (If Real Application Clusters (RAC), run this command on each node):

For example (as root):# rpm -qa | grep oracleasm

i. Output if installed:

oracleasm-2.6.18-164.el5PAE-2.0.5-1.el5 -----> optionaloracleasm-2.6.18-164.el5debug-2.0.5-1.el5 -----> optionaloracleasm-2.6.18-164.el5-2.0.5-1.el5oracleasmlib-2.0.4-1.el5oracleasm-support-2.1.3-1.el5oracleasm-2.6.18-164.el5xen-2.0.5-1.el5 -----> optional

ii. Output if not installed

package not installed

a. Install (if not installed).  Install MUST match kernel version. (System Administrator's Task)

b. Verify kernel version:

# uname -r2.6.18-164.el5PAE

c. Install the correct packages for the kernel version.

# rpm -i oracleasm-support-2.1.3-1.el5.i386.rpmoracleasmlib-2.0.4-1.el5.i386.rpmoracleasm-2.6.18-164.el5-2.0.5-1.el5.i386.rpm    

Important:  The version of the ASMLIB driver version has to be the same as the kernel version. Download the matching ASMLIB driver version from Oracle's website for ASMLIB drivers:   http://www.oracle.com/technetwork/server-storage/linux/downloads/index-088143.html

      3b.  Check status (If Real Application Clusters (RAC), run this command on each node):

Failed example:# /etc/init.d/oracleasm statusChecking if ASM is loaded: noChecking if /dev/oracleasm is mounted: no

If status failed ("no" displayed), then:

Configuring the Oracle ASM library driver, for example:

# /etc/init.d/oracleasm configure (If RAC, run this command on all nodes)

Default user to own the driver interface []: grid        -----> input Grid Infrastructure/ASM user name Default group to own the driver interface []: asmadmin   -----> input Grid Infrastructure/ASM group name Start Oracle ASM library driver on boot (y/n) [n]: yScan for Oracle ASM disks on boot (y/n) [y]: yWriting Oracle ASM library driver configuration: doneInitializing the Oracle ASMLib driver: [ OK ]Scanning the system for Oracle ASMLib disks: [ OK ]

Note: This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort.

3c. Check status again:

# /etc/init.d/oracleasm statusChecking if ASM is loaded: yesChecking if /dev/oracleasm is mounted: yes

4.  Create ASM diskgroups:

Important:  The /dev/dm-n devices are internal to device-mapper-multipath and are non-persistent, so should not be used. The /dev/mpath/ devices are created for multipath devices to be visible together, however, may not be available during the early stages of the boot process, so should not typically be used.  However, /dev/mapper/ devices are persistent and created early during boot...These are the only device names that should be used to access multipathed devices.

Note:  Use /dev/mapper/[WWID]p1 as the device name for createdisk.  If Real Application Clusters (RAC), do only on the first node.  All commands done as root.

4a. Check prior to createdisk command:

# /etc/init.d/oracleasm querydisk DATDisk "DAT" does not exist or is not instantiated

# /etc/init.d/oracleasm querydisk /dev/mapper/360060e80045b2b0000005b2b0000163cp1Device "/dev/mapper/360060e80045b2b0000005b2b0000163cp1" is not marked as an ASM disk

4b. After Check, do createdisk command:

# /etc/init.d/oracleasm createdisk DAT /dev/mapper/360060e80045b2b0000005b2b0000163cp1Marking disk "/dev/mapper/360060e80045b2b0000005b2b0000163cp[ OK ] ASM disk

Note:  If using multiple devices/disks within an ASM diskgroup, a good practice would be to use /etc/init.d/oracleasm createdisk with numbers appended to ASM alias name to create the members of the diskgroup for example:           # /etc/init.d/oracleasm createdisk DAT01 /dev/mapper/360060e80045b2b0000005b2b0000163cp1

5.  To make the disk available enter the following commands:      

Note:  If Real Application Clusters (RAC), run the following two commands, in the order of first node, second node, etc.   All commands done as root.        

5a.  Scan ASM disks:

# /etc/init.d/oracleasm scandisks

Scanning system for ASM disks: [ OK ] 

5b.  List ASM disks:

# /etc/init.d/oracleasm listdisksDAT

 6.  Check the ASM diskgroups:

For example (as root) (If Real Application Clusters (RAC), do on each node.):

# /etc/init.d/oracleasm querydisk DATDisk "DAT" is a valid ASM disk on device [253, 20]

or

# /etc/init.d/oracleasm querydisk -d DATDisk "DAT" is a valid ASM disk on device [253, 20]

# /etc/init.d/oracleasm querydisk /dev/mapper/360060e80045b2b0000005b2b0000163cp1Device "/dev/mapper/360060e80045b2b0000005b2b0000163cp1" is marked as an ASM disk

Note:  The numbers, [253, 20], indicate major and minor numbers that correspond to the major and minor numbers in the file, /proc/partitions.  These numbers can be used to validate the multipathed device by cross-referencing these numbers with the file, /proc/partitions, and the output of "multipath -ll" to ensure that the major and minor numbers match.

# cat /proc/partitionsmajor minor #blocks name...253 20 524281275 dm-20

7.  Ensure that the allocated devices can be seen in /dev/mpath:

           For example (as root) (f Real Application Clusters (RAC), do on each node.):

# cd /dev/mpath

# ls -ltotal 0lrwxrwxrwx 1 root root 8 May 8 10:32 360060e80045b2b0000005b2b0000163c -> ../dm-11lrwxrwxrwx 1 root root 8 May 8 10:32 360060e80045b2b0000005b2b0000163cp1 -> ../dm-20lrwxrwxrwx 1 root root 8 May 8 10:32 360060e80045b2b0000005b2b0000155a -> ../dm-14

lrwxrwxrwx 1 root root 8 May 8 10:32 360060e80045b2b0000005b2b0000155ap1 -> ../dm-22lrwxrwxrwx 1 root root 8 May 8 10:32 360060e80045b2b0000005b2b00001584 -> ../dm-16lrwxrwxrwx 1 root root 8 May 8 10:32 360060e80045b2b0000005b2b00001584p1 -> ../dm-23lrwxrwxrwx 1 root root 7 May 8 10:32 360060e80045b2b0000005b2b00003130 -> ../dm-1lrwxrwxrwx 1 root root 7 May 8 10:32 360060e80045b2b0000005b2b00003131 -> ../dm-2lrwxrwxrwx 1 root root 7 May 8 10:32 360060e80045b2b0000005b2b00003132 -> ../dm-3lrwxrwxrwx 1 root root 7 May 8 10:32 360060e80045b2b0000005b2b00003133 -> ../dm-4lrwxrwxrwx 1 root root 7 May 8 10:32 360060e80045b2b0000005b2b00003134 -> ../dm-5

8.  Ensure that the devices can be seen in /dev/mapper:

           For example (as root) (f Real Application Clusters (RAC), do on each node.): # ls -l /dev/mappertotal 0brw-rw---- 1 root disk 253, 0 Jun 27 07:17 360060e80045b2b0000005b2b000006b0brw-rw---- 1 root disk 253, 15 Jun 27 07:17 360060e80045b2b0000005b2b000006b0p1brw-rw---- 1 root disk 253, 1 Jun 27 07:17 360060e80045b2b0000005b2b000006c4brw-rw---- 1 root disk 253, 17 Jun 27 07:17 360060e80045b2b0000005b2b000006c4p1brw-rw---- 1 root disk 253, 2 Jun 27 07:17 360060e80045b2b0000005b2b000006d8brw-rw---- 1 root disk 253, 26 Jun 27 07:17 360060e80045b2b0000005b2b000006d8p1brw-rw---- 1 root disk 253, 11 Jun 27 07:17 360060e80045b2b0000005b2b0000163cbrw-rw---- 1 root disk 253, 20 Jun 27 07:17 360060e80045b2b0000005b2b0000163cp1brw-rw---- 1 root disk 253, 12 Jun 27 07:17 360060e80045b2b0000005b2b0000155abrw-rw---- 1 root disk 253, 29 Jun 27 07:17 360060e80045b2b0000005b2b0000155ap1brw-rw---- 1 root disk 253, 14 Jun 27 07:17 360060e80045b2b0000005b2b00001679brw-rw---- 1 root disk 253, 25 Jun 27 07:17 360060e80045b2b0000005b2b00001679p1brw-rw---- 1 root disk 253, 13 Jun 27 07:17 360060e80045b2b0000005b2b00001584brw-rw---- 1 root disk 253, 24 Jun 27 07:17 360060e80045b2b0000005b2b00001584p1brw-rw---- 1 root disk 253, 3 Jun 27 07:17 360060e80045b2b0000005b2b00001007brw-rw---- 1 root disk 253, 19 Jun 27 07:17 360060e80045b2b0000005b2b00001007p1brw-rw---- 1 root disk 253, 4 Jun 27 07:17 360060e80045b2b0000005b2b0000189cbrw-rw---- 1 root disk 253, 18 Jun 27 07:17 360060e80045b2b0000005b2b0000189cp1

9.  Check the device type:

           For example (as root) (f Real Application Clusters (RAC), do on each node.):# /sbin/blkid | grep oracleasm/dev/dm-20: LABEL="DAT" TYPE="oracleasm"   ---> >>here multipathed/dev/dm-22: LABEL="ARC" TYPE="oracleasm"/dev/dm-23: LABEL="FRA" TYPE="oracleasm"/dev/sdh1: LABEL="DAT" TYPE="oracleasm"    ---> here physical/dev/sdx1: LABEL="ARC" TYPE="oracleasm"/dev/sdj1: LABEL="FRA" TYPE="oracleasm"/dev/sdi1: LABEL="DAT" TYPE="oracleasm"    ---> >> here physical/dev/sdy1: LABEL="ARC" TYPE="oracleasm"/dev/sdz1: LABEL="FRA" TYPE="oracleasm">>>>

10.  Setup the ASM parameter (ORACLEASM_SCANORDER), in the file for ASMLIB configuration, /etc/sysconfig/oracleasm, for forcing ASM to bind with the multipath devices

Note:  If Real Application Clusters (RAC), do commands on each node.  All commands done as root.

10a.  Check the file, /etc/sysconfig/oracleasm:

# ls -la /etc/sysconfig/oracleasmlrwxrwxrwx 1 root root 24 Jun 13 09:58 /etc/sysconfig/oracleasm -> oracleasm-_dev_oracleasm

10b. Make a backup of the original file, /etc/sysconfig/oracleasm-_dev_oracleasm

# cp /etc/sysconfig/oracleasm-_dev_oracleasm /etc/sysconfig/oracleasm-_dev_oracleasm.orig

10c.  Modify the ORACLEASM_SCANORDER and ORACLEASM_SCANEXCLUDE parameters in /etc/sysconfig/oracleasm:

# vi /etc/sysconfig/oracleasm-_dev_oracleasm

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning

ORACLEASM_SCANORDER="mpath dm"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

ORACLEASM_SCANEXCLUDE="sd"

Note: Another valid value can be used for ORACLEASM_SCANORDER: ORACLEASM_SCANORDER="dm"

10d. save file

10e. Restart oracleasm:

# service oracleasm restart

or

# /etc/init.d/oracleasm restart

10f. Check mulitpath device against /proc/partitions file:

# cat /proc/partitionsmajor minor #blocks name...253 20 524281275 dm-20

10g. Check mulitpath device against the file, /dev/oracleasm/disks:

# ls -ltr /dev/oracleasm/disksbrw-rw---- 1 grid asmadmin 253, 20 Oct 4 13:37 DAT

10f. Check oracleasm disks again:

# /etc/init.d/oracleasm listdisksDAT

Note:  ASMLIB first tries/scans all disks that are in the /proc/partitions file. Within the multipath directory, /dev/mpath, the alias names and the WWIDs point to/are linked to the multipathed names dm[-n] names.  Furthermore, ASMLIB does not scan any disks (ORACLEASM_SCANEXCLUDE) that start with "sd". This is all the SCSI disks.

Additional Resources

Community Discussions: Storage Management MOS CommunityStill have questions? Use the above community to search for similar discussions or start a new discussion on this subject.

ReferencesNOTE:564580.1 - Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5NOTE:603868.1 - How to Dynamically Add and Remove SCSI Devices on LinuxNOTE:555603.1 - Configuration and Use of Device Mapper Multipathing on Oracle Enterprise Linux (OEL)NOTE:743949.1 - Unable To Create ASMLIB DiskNOTE:967461.1 - "Multipath: error getting device" seen in OS log causes ASM/ASMlib to shutdown by itselfNOTE:580153.1 - How To Setup ASM on Linux Using ASMLIB Disks, Raw Devices or Block Devices?NOTE:1089399.1 - Oracle ASMLib Software Update Policy for Red Hat Enterprise Linux Supported by Red HatNOTE:602952.1 - How To Setup ASM & ASMLIB On Native Linux Multipath Mapper disks?

 Related  

 Products  

Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Enterprise Edition > STORAGE > ASM Installation and Patching Issues

 Keywords  ASM; ASMLIB; CLUSTER; DISKGROUP; ENTERPRISE LINUX; LINUX; MULTIPATH;

ORACLEASM; SCSI; STORAGE Errors  

CSI-360060; CSI-3600508

Back to Top