Discovering VIO relationships for EMC arrays
Transcript of Discovering VIO relationships for EMC arrays
discovering vio relationships for emc arrays
EMC Proven Professional Knowledge Sharing 2011
Ewa Nowatkowska-GłowaczSenior Solutions [email protected]
EMC Proven Professional Knowledge Sharing 2
Table of contents
Introduction ............................................................................................................... 3
Inquiry utility .............................................................................................................. 5
Regular hdisk ......................................................................................................... 6
PowerPath device (hdiskpower) ............................................................................. 6
Logical volume ..................................................................................................... 12
Summary ................................................................................................................. 21
Scripts and EMCgrab .............................................................................................. 23
Bibliography ............................................................................................................ 26
Appendix ................................................................................................................. 27
Disclaimer: The views, processes or methodologies published in this article are
those of the author. They do not necessarily reflect EMC Corporation’s views,
processes, or methodologies.
EMC Proven Professional Knowledge Sharing 3
Introduction
This article describes a method for discovering VIO relationships for IBM p-based
systems connected to EMC Symmetrix® and CLARiiON® arrays. The Virtual I/O
Server (VIOS) provides virtual storage to client logical partitions by allowing a
physical adapter with attached disks on the VIOS partition to be shared by one or
more Virtual I/O Client (VIOC) partitions.
This type of configuration with use of VIOS/VIOC relationship becomes increasingly
popular among enterprise customers where IBM servers are installed. In large
migration projects, discovering and documenting relationships between servers and
their clients is a time consuming task.
EMC Proven Professional Knowledge Sharing 4
Currently, there are no tools available which could perform (from the client point of
view) fully automated volume or device mapping between servers and clients. The
identification method shown in this article could be useful for verifying current
configuration for migration purposes or can be used for daily reporting. In this article,
a real implementation project will be mentioned where the EMC team worked out
methodology and scripts needed to discover current relationships in order to perform
migration of VIOS resources between EMC arrays (technology refresh). Along with
VIO servers resource migration, VIO clients potentially involved in any given
migration were also indicated. Configuration and commands’ outputs shown in this
article are based on documentation of a real environment and helps to understand
VIO configuration challenges. Sample scripts for reports automation for large
configuration will be shown at the end of this article.
The main idea of how to discover VIOC/VIOS relationship during a migration project
came from reading EMC document: emc157385. In this document, methodology
overview and individual steps are described and illustrated by CLI commands.
However, during migration projects, other possible configurations were discovered
which are also worth mentioning and which inspired the author to write this article.
EMC Proven Professional Knowledge Sharing 5
Inquiry utility
Inquiry utility is required in this methodology to properly identify type of VIOC devices.
This utility can be downloaded from ftp://ftp.emc.com/pub/elab/inquiry/.
Here are examples of possible types of devices which could be identified in
miscellaneous configuration. The first five lines are typically directly connected
devices (both internal and external); the next three (line 6, 7, and 8) represent
devices seen by the VIO client.
---------------------------------------------------------------------
DEVICE :VEND :PROD :REV :SER NUM :CAP(kb)
---------------------------------------------------------------------
1 /dev/rhdisk6 :EMC :SYMMETRIX :5771 :8600cf3080 :3840000
2 /dev/rhdiskpower0 :EMC :SYMMETRIX :5771 :8600722080 :44193600
3 /dev/rhdisk1 :IBM :ST3300655LC :C806 :003185C :29296875
4 /dev/rhdisk2 :IBM H0:ST373455LC :C806 :0002F4A0 :71687000
5 /dev/rhdiskpower73 :DGC :RAID 5 :0326 :490000EC :1048576
6 /dev/rhdisk0 :AIX :VDASD : :hdisk66 :35354880
7 /dev/rhdisk4 :AIX :VDASD : :hdiskpow :35354880
8 /dev/rhdisk7 :AIX :VDASD : :001RT06H :13631488
Configuration shown in lines 6, 7, 8 will be described in detail later in this section.
In order to fully recognize the device behind VIO mapping, inquiry with option “-page0”
is needed against every client device involved in VIO configuration. In other words, if
client has three mapped devices shown above in lines 6, 7 and 8, inquiry should
query all these devices in the following syntax:
# inq.aix64_51 –r /dev/rhdisk0 –page0 (line 6)
# inq.aix64_51 –r /dev/rhdisk4 –page0 (line 7)
# inq.aix64_51 –r /dev/rhdisk7 –page0 (line 8)
EMC Proven Professional Knowledge Sharing 6
Regular hdisk
This is the simplest case which is described in detail in EMC document emc157385.
Inquiry utility in column “SER NUM” shows hdisk instance (hdisk66) of VIO server. Using
inquiry on VIO server we can discover more information about this disk. Detailed inquiry
(with option “-page0”) can also be used to find out more details about hdisk66.
---------------------------------------------------------------------
DEVICE :VEND :PROD :REV :SER NUM :CAP(kb)
---------------------------------------------------------------------
/dev/rhdisk0 :AIX :VDASD : :hdisk66 :35354880
PowerPath device (hdiskpower)
Line 7 informs us that this is configuration where PowerPath® is installed on VIO server.
This type of configuration could be described by inquiry output shown below.
---------------------------------------------------------------------
DEVICE :VEND :PROD :REV :SER NUM :CAP(kb)
---------------------------------------------------------------------
/dev/rhdisk4 :AIX :VDASD : :hdiskpow :35354880
Output from inquiry utility is not very helpful in this case because it doesn’t show full
hdiskpower instance; only 8 characters are shown (e.g. instance hdiskpower224 needs
13 characters to be a fully recognized device). Device hdiskpower is a pseudo device
created by EMC PowerPath software and can be EMC Symmetrix® symdev or EMC
CLARiiON® LUN. On the next two pages there are examples of outputs from inquiry
(inq.aix64_51 –r /dev/rhdiskX –page0) for devices from these two types of arrays.
EMC Proven Professional Knowledge Sharing 7
Here is an example for a CLARiiON device. For successful match devices between VIOS
and VIOC, unique Clariion LUN’s UID is used. This UID is visible in powermt display
output as well as inquiry for specific hdisk. LUN UID is shown in lines which start with:
“0060”, “0070” and “0080”.
EMC Proven Professional Knowledge Sharing 8
This is an example for Symmetrix symdev. In inquiry output (line “0060”) there is information about array serial number and symdev.
EMC Proven Professional Knowledge Sharing 9
In dual Virtual I/O Servers configuration, virtual SCSI can be configured redundantly
in order to provide continuous access when a Virtual I/O Server has to be taken
offline for planned outages (software updates as a example). This configuration
allows one Virtual I/O Server to be rebooted without affecting the virtual I/O clients. A
simple picture showing this type of configuration is shown here.
The volume (symdev 124F) is located on an external storage subsystem (EMC
Symmetrix DMX4, serial number ends with “55”) and is assigned to both Virtual I/O
Servers (vios1 & vios2). The client partition (vioc) sees the disk through two paths
using MPIO (lspath verifies this configuration). Each of the paths is going through a
different Virtual I/O server. Inquiry doesn’t show dual VIOS configuration.
inq.aix64_51 -no_dots (VIOC)
---------------------------------------------------------------------
DEVICE :VEND :PROD :REV :SER NUM :CAP(kb)
---------------------------------------------------------------------
/dev/rhdisk13 :AIX :VDASD : :hdiskpow : 13631488
EMC Proven Professional Knowledge Sharing 10
From inquiry, we know that serial number of arrays ends with 55 and symdev number
is 124F. In order to check if there is dual VIOS configuration, lspath can be used to
verify that there are two paths for this hdisk13.
lspath (VIOC)
Enabled hdisk13 vscsi0
Enabled hdisk13 vscsi1
EMC Proven Professional Knowledge Sharing 11
Uname command with option “-L” can be used to identify and verify VIO client
partition number.
uname –L (VIOC)
5 vioc
EMC Proven Professional Knowledge Sharing 12
Logical volume
In inquiry output shown below there are four lines showing instances when logical
volume is used by VIO servers to map hdisk to the client. This is a quite complicated
configuration which also involves dual VIO servers and is shown in the picture on the
following page. Here is part of inquiry output for hdisks for which “SER NUM” column
is the same. Again, based on this output there is not enough information to recognize
the device (logical volume in this case).
inq.aix64_51 -no_dots ---------------------------------------------------------------------
DEVICE :VEND :PROD :REV :SER NUM :CAP(kb)
---------------------------------------------------------------------
/dev/rhdisk0 :AIX :VDASD : :001MW06C : 2097152
/dev/rhdisk5 :AIX :VDASD : :001MW06C : 2097152
/dev/rhdisk12 :AIX :VDASD : :001MW06C : 3145728
/dev/rhdisk13 :AIX :VDASD : :001MW06C : 3145728
uname –L 12 vioc
Client partition number is “12” (this is 0x0c in hexadecimal notation shown by lsmap
command in Client Partition ID column).
Every hdisk has only one available path but what was discovered later (outputs from
adequate LVM commands are not shown in this document), logical volumes created
on hdisk0 and hdisk5 were also mirrored at LVM level.
lspath Enabled hdisk0 vscsi0
Enabled hdisk5 vscsi1
Enabled hdisk12 vscsi0
Enabled hdisk13 vscsi1
EMC Proven Professional Knowledge Sharing 13
EMC Proven Professional Knowledge Sharing 14
EMC Proven Professional Knowledge Sharing 15
lspv -l (VIOS1)
--------------------------------------------------------
lspv -l hdiskpower2 :-
hdiskpower2:
LV NAME LPs PPs DISTRIBUTION MOUNT POINT
001MW06CWO_lv01 20 20 00..00..00..10..10 N/A
001MW06CXP_lv01 13 13 11..00..00..00..02 N/A
001MW06CUP_lv01 1 1 01..00..00..00..00 N/A
001MW06CQP_lv02 24 24 00..12..11..01..00 N/A
--------------------------------------------------------
lspv -l hdiskpower4 :-
hdiskpower4:
LV NAME LPs PPs DISTRIBUTION MOUNT POINT
001MW06CZP_lv01 16 16 00..00..00..00..16 N/A
001MW06CXP_lv01 3 3 03..00..00..00..00 N/A
001MW06CSP_lv01 72 72 00..23..23..23..03 N/A
--------------------------------------------------------
lspv -l hdiskpower61 :-
hdiskpower61:
LV NAME LPs PPs DISTRIBUTION MOUNT POINT
001MW06CYP_lv02 24 24 00..03..21..00..00 N/A
001MW06CZP_lv02 10 10 02..00..05..03..00 N/A
001MW06CXP_lv02 24 24 24..00..00..00..00 N/A
001MW06CVP_lv01 3 3 00..03..00..00..00 N/A
001MW06CUP_lv01 39 39 00..00..00..24..15 N/A
EMC Proven Professional Knowledge Sharing 16
EMC Proven Professional Knowledge Sharing 17
lspv –l (VIOS2)
--------------------------------------------------------
lspv -l hdiskpower32 :-
hdiskpower32:
LV NAME LPs PPs DISTRIBUTION MOUNT POINT
001MW06CYP_lv01 16 16 00..00..07..09..00 N/A
001MW06CVP_lv02 26 26 12..00..00..02..12 N/A
001MW06CXP_lv01 16 16 00..12..04..00..00 N/A
--------------------------------------------------------
lspv -l hdiskpower62 :-
hdiskpower62:
LV NAME LPs PPs DISTRIBUTION MOUNT POINT
001MW06CXP_lv02 24 24 24..00..00..00..00 N/A
001MW06CYP_lv02 24 24 00..03..21..00..00 N/A
001MW06CZP_lv02 10 10 02..00..05..03..00 N/A
001MW06CVP_lv01 3 3 00..03..00..00..00 N/A
001MW06CUP_lv01 39 39 00..00..00..24..15 N/A
EMC Proven Professional Knowledge Sharing 18
EMC Proven Professional Knowledge Sharing 19
EMC Proven Professional Knowledge Sharing 20
As it is shown in inquiry output on the previous page, there is additional information included in lines “0060-0080” which can be useful to double check which VIO server (one of two) mapped hdisk5 to the client.
It is worth mentioning that outputs from above commands are collected by EMCgrab
- “odmget_CuAt.txt” file in “host” directory.
EMC Proven Professional Knowledge Sharing 21
Summary
The entire methodology which allows us to discover what physical devices are
behind hdisk seen on client side can be described in the following way:
First, use of “inquiry –no_dots” command to discover if any devices are
mapped from VIOS.
For every device which could potentialny be mapped by VIOS, inquiry with
option “-page0” should be invoked.
Output from “inquiry –page0” should be parsed in search of characteristic
strings from specific lines.
On the next page, “inquiry –page0” outputs from three configuration types described
earlier in this document are presented together in order to easily compare them.
EMC Proven Professional Knowledge Sharing 22
Based on outputs from inquiry command (-page0) it is possible to discover and
differentiate VIOC devices:
line “0090:” can be used to verify if this is CLARiiON LUN
(substr($18,1,5)==”GCfcp”)
o if this is a CLARiiON device, concatenate strings in line “0060,
0070, 0080” in order to create LUN UID
o if this is a Symmetrix device, substr($18,5,2) gives array serial
number, substr($18,7,4) gives symdev
EMC Proven Professional Knowledge Sharing 23
Scripts and EMCgrab
If the customer’s configuration consists of many VIO servers and clients, it is worth
considering the use of scripts which can help us to put devices in a given order and
match them between servers and clients. Here are a few examples of Shell scripts
(or procedures inside one script) which could automate this matching (real, used by
EMC script during migration project was much more complicated).
There is also a lot of information about server and client configuration available in the
form of files collected by EMCgrab. However, to fully match the client’s hdisk with
server hdiskpower (as an example), verification on the client side is needed (by using
“inquiry –page0” command) in order to discover this relationship.
Shell script vioc.sh shown in the Appendix identifies hdiskpower devices (both
Symmetrix and CLARiiON) shown in output from “inq.aix64_51 -no_dots” command
and for every identified hdiskpower device checks “backing device” using inquiry
utility with option “page0”. Here is an output from this script:
vioc|partition|size in GB|array serial|symdev or LUN UID|type
vioc|5|hdisk13|102.539|55|124F|SYMMETRIX
vioc|6|hdisk62|5||600601605A642200C4410D05FD9ADF11|CLARiiON
Please note that the fifth column is empty if the “backing device” is a CLARiiON array.
Next script “pp_config.sh” based on EMCgrab (file “powermt_display_dev=all.txt” in
“pp” directory) from VIO servers identifies all hdiskpower instances creating the
following output:
server|hdispower instance|array|symdev|
vios1|hdiskpower224|00029xxxxx55|124F|
vios2|hdiskpower224|00029xxxxx55|124F|
vios2|hdiskpower64|CK200xxxxxx25|LUN 77|
This script can be easly modified to show also LUN UID for CLARiiON LUNs.
Based on another collected by EMCgrab “_usr_ios_cli_ioscli_lsmap_-all.txt” file in
“host” directory, script check_lsmap.sh helps to sort what devices are presented to
which client partition.
vios|vioc|partition|vtd|backing device
vios1|vhost0|0x00000005|01DT100A5P_mp11|hdiskpower224
vios2|vhost0|0x00000005|01DT100A5P_mp11|hdiskpower224
EMC Proven Professional Knowledge Sharing 24
Below is a simple diagram of how easily collection of “inquiry –page0” can be
automated. Instead of executing this command manually on every VIOC, a short
script can be executed sequentially on every VIOC and results transferred over open
connection between server and client. There are two examples below, one was
written in Perl and the other is Windows Batch.
#!perl
#
($min, $hour, $day, $month, $year)=(localtime) [1,2,3,4,5];
$dir="logs";
$logfile=sprintf("%04d.%02d.%02d_%02d%02d_log.txt",$year+1900,$month+1,$day,
$hour,$min);
$errorfile=sprintf("%04d.%02d.%02d_%02d%02d_error.txt",$year+1900,$month+1,$
day,$hour,$min);
print "Logfile is: $logfile\n";
open( STDOUT, "> $dir\\$logfile" );
open( STDERR, "> $dir\\$errorfile" );
use Socket;
$port=XXXXX;
$net="192.168.1.";
foreach $ip ($net.1,$net.2)
{
print "$ip \n";
socket(SERVER, PF_INET, SOCK_STREAM, getprotobyname('tcp'));
$te=sockaddr_in($port,inet_aton($ip));
connect(SERVER,$te)
or die "Cannot connect to $ip:$port \n";
while (1)
{
$buffer=<SERVER>;
if ($buffer ne "" ) {print "$buffer\n";}
else {last; }
}
close(SERVER);
}
close (STDOUT);
close (STDERR);
@ECHO OFF
SET output=O:\output.csv
SET log=O:\log.txt
SET port=XXXXX
ECHO Servers
ECHO -------
ECHO OFF > %output%
ECHO OFF > %log%
FOR /f "tokens=* delims=" %%h in (vioc.txt) DO (
ECHO %%h
ECHO %%h >> %log%
O:\plink1 -telnet %%h -P %port% >> %output% 2>> %log% )
1 plink is available is a part of famous PUTTY download.
EMC Proven Professional Knowledge Sharing 25
EMC Proven Professional Knowledge Sharing 26
Bibliography
The following documents were used during writing this document and can be read to
fully understand the topic:
EMC Host Connectivity Guide for AIX (chapter “Virtual I/O Server”)
Primus solutions: emc157385, emc156902, emc110159, emc109158
Release Notes for EMC Solution Enabler 7.0 contains the following information:
VMware and AIX VIO virtualized clients support has been added to Solutions Enabler.
The feature allows symrslv to resolve array devices behind the virtual disk. The host,
with HBAs and Symmetrix array, creates virtualized environments, or guests. The
host pools together disks and then can carve out virtual disks for the guests. Now,
Solutions Enabler can identify Symmetrix devices behind the virtual disk.
During the project, new Solution Enabler capabilities described above weren’t tested
(project started before release of this version). Potentially, exploring this way of
discovering VIO client devices could make the identification process even easier than
described in this article.
EMC Proven Professional Knowledge Sharing 27
EMC Proven Professional Knowledge Sharing 28
EMC Proven Professional Knowledge Sharing 29
EMC Proven Professional Knowledge Sharing 30
EMC believes the information in this publication is accurate as of its publication
date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC
CORPORATION MAKES NO RESPRESENTATIONS OR WARRANTIES OF ANY
KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND
SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication
requires an applicable software license.