Creating a Red Hat Cluster
-
Upload
praveen-kumar-karli -
Category
Documents
-
view
139 -
download
7
description
Transcript of Creating a Red Hat Cluster
Creating a Red Hat Cluster: Part 4In this article we will create a GFS filesystem that will allows us to share data between nodes. In the next and last article we’ll finalise the cluster by completing our ftp and web service so they really work. We will also show you how to manually move service from one server to another. So we still have some work to do, so let’s start right away.
Adding a SAN disk to our serversThe Linux operating system is installed on the internal disks for each of our server. We will now add a SAN disk that will be visible be each of our server. I assume here that your SAN and your Brocade switch are configure accordingly. Explaining how to set up the SAN and the Brocade switch is not in the scope of this article. But I think that you get the idea that the new disk must be visible by every node in our cluster. In the example below we already have a SAN disk (sda) with one partition (sda1) on it. Adding a disk to the server, can be done (live) without any interruption of service, if you follow the steps below. I would suggest you practice on a test server, to become familiar with the procedure.
Before we add a disk, let’s see what are the visible disks on the system, by looking at the /proc/partitions file. We can see that we already have a disk (sda) with one partition on it. So the new disk that we’re going to add, should be seen as “sdb”.
root@gollum~# grep sd /proc/partitions8 0 104857920 sda8 1 104856223 sda1
Let’s rescan the SCSI bus by typing the command below. This command must be run on each of the server within the cluster. Here, we have only one HBA (Host Base Adapter) card connected to the SAN on each server. If you have a second HBA, you need to run the same command for the second HBA, but replace the “host0″ by “host1″.
root@gollum~# echo “- – -” > /sys/class/scsi_host/host0/scanroot@gandalf~# echo “- – -” > /sys/class/scsi_host/host0/scanroot@bilbo~# echo “- – -” > /sys/class/scsi_host/host0/scan
Let’s see if we have some new disk(s) that were detected (sdb) (check each servers)
root@gollum~# grep sd /proc/partitions8 0 104857920 sda8 1 104856223 sda18 16 15728640 sdb
Let’s create a LVM partition on our new disk (sdb) by running the “fdisk” command.
# fdisk /dev/sdb
Command (m for help): p
Disk /dev/sdh: 107.3 GB, 107374510080 bytes255 heads, 63 sectors/track, 13054 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System
Command (m for help): n (n=new partition)Command actione extendedp primary partition (1-4) p (p=primary partition)
Partition number (1-4): 1 (first partition =1)First cylinder (1-13054, default 1): 1 (Start at the beginning of disk)Last cylinder or +size or +sizeM or +sizeK (1-13054, default 13054): 13054 (End of the Disk)Command (m for help): p (p=Print partition information)
Disk /dev/sdh: 107.3 GB, 107374510080 bytes255 heads, 63 sectors/track, 13054 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System/dev/sdb1 1 13054 104856223+ 83 Linux
Command (m for help): t (t=type of partition)Selected partition 1 (Change type of partition 1)Hex code (type L to list codes): 8e (8e=LVM partition – Type L to list partition code)Changed system type of partition 1 to 8e (Linux LVM)Command (m for help): p (p=print partition information)
Disk /dev/sdh: 107.3 GB, 107374510080 bytes255 heads, 63 sectors/track, 13054 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System/dev/sdb1 1 13054 104856223+ 8e Linux LVM
Command (m for help): w (w=write partition to disk)
The partition table has been altered!Calling ioctl() to re-read partition table.Syncing disks.
If we look again at our /proc/partition, we should see our new disk and partition are being seen by this server.
root@gollum~# grep sd /proc/partitions8 0 104857920 sda8 1 104856223 sda18 16 15728640 sdb8 16 15727658 sdb1
Now we need to make sure that the new disk and partition are seen by every servers within the cluster. We now need to go on every servers in the cluster and run the command “partprobe” (Partition probe). After running the command, you should check like we did on “gollum”, if all the disks and partitions are seen by each servers.
root@gollum~# partproberoot@gandalf~# partproberoot@bilbo~# partprobe
Creating our physical volume
Now that we know that the disk is seen by every node, let’s create the physical volume on one of the server and then check on all others servers in the cluster, the physical should be seen on all the servers. The command to create a physical volume is “pvcreate”, so what we are really doing here, is creating a physical volume of the partition (/dev/sdb1) we created earlier.
# pvcreate /dev/sdb1Physical volume “/dev/sdb1″ successfully created
Let’s run a pvscan on every node, to validate that every node can actually see the new disk.
# pvscanPV /dev/sda1 VG datavg lvm2 [100.00 GB / 22.02 GB free]PV /dev/sdb1 lvm2 [100.00 GB]
Create our clustered volume group
We will now create a new volume group named “sharevg” and we will assign the physical volume “/dev/sdb1″ as part of that group. If we ever ran out of disk space within “sharevg”, we could add another physical volume to the volume group and continue to work without any service disruption. This is a real advantage when working in production environment.
# vgcreate sharevg /dev/sdb1Clustered volume group “sharevg” successfully created
Display the “sharevg” volume group properties
We can display the volume group properties by issueing the “vgdisplay” command. We can see that our volume group is “Clustered”, so it is cluster aware. This will allow later on, to create “LV” (Logical volume/Filesystem) on one server and have the cluster software automatically advise the cluster member that a new logical volume (filesystem) is available.
root@gollum~# vgdisplay sharevg
--- Volume group ---
VG Name sharevg
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 25
VG Access read/write
VG Status resizable
Clustered yes
Shared no
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 100.00 GB
PE Size 4.00 MB
Total PE 24999
Alloc PE / Size 1 / 2.10 MB
Free PE / Size 24980 / 98.00 GB
VG UUID V8Ag76-vdW2-NAk4-JjOo-or3l-GuPz-x5LEKP
Create a logical volume of 1024MB named “cadmin” of the “sharevg” volume group
We will create a logical volume named “cadminlv” (Cluster Admin), in our sharevg volume group. The command below is asking to create a logical volume of 1024MB, name “cadminlv” in the volume group “sharevg”. This command can be done one server and the logical volume will be seen by every member of the cluster.
#/usr/sbin/lvcreate -L1024M -n cadminlv sharevgLogical volume “cadminlv” created
The “lvs” command allows you to display a list of all your logical volumes. Since this is currently the only one on the volume group “sharevg”, we filter the list (with the grep command) to only display the logical volume on “sharevg” volume group.Let’s check if it seen by all nodes.
root@gandalf~# lvs | grep sharevgcadminlv sharevg -wi-a- 1024.00M
root@gollum~# lvs | grep sharevgcadminlv sharevg -wi-a- 1024.00M
root@bilbo~# lvs | grep sharevgcadminlv sharevg -wi-a- 1024.00M
Creating the /cadmin GFS FilesystemFinally, we are going to create a GFS filesystem within that logical volume “cadminlv” we’ve just created. But first we need to create our GFS filesystem mount point on every nodes. We need to do that, because we want this filesystem to be mounted on every node and be available for the 3 nodes.
root@gandalf~# mkdir /cadmin
root@bilbo~# mkdir /cadmin
root@gollum~# mkdir /cadmin
We have choosen to have the GFS filesystem “/cadmin” to be mounted on all servers at all time. We could have include it as part of our service, so it would be mounted only when a service is started. But we found out, that the process of unmounting and mounting a couple of GFS take time and this time adds up to the time it take to move a service from one server to another. In production we have 5 servers in a cluster for more that 2 years now, we have around 30 GFS mounted at all time on all the five servers and we had very little problem. The only thing you have to be careful about is the number of journals that you assign to each GFS. One journal is needed for each concurrent mount in the cluster, in our case we have at least 5 journals for each of our GFS filesystem (more on that below).Create the GFS on the LVM cadmin created previously. This need to be done only on one node, the creation is done once and all the nodes are made aware of the new GFS by the cluster daemon.
The command we use to create a GFS filesystem is “gfs_mkfs”. We need to use a couple of options and I will explain them all.
First, the “-O“ prevents “gfs_mkfs” from asking for confirmation before creating the filesystem.
The option “-p lock_dlm“, indicate the name of the locking protocol to use. The locking protocol should be “lock_dlm” for a clustered file system.
The “-t our_cluster:cadminlv” It’s the cluster-name, followed by “:” and the logical volume name. The cluster name must match the one you have defined in your cluster configuration file (in our case “our_cluster”), only members of this cluster are permitted to use this file system. The filesystem name (cadminlv) is a unique file system name used to distinguish this GFS file system from others created (1 to 16 characters).
The “-j 4” is the number of journals for gfs_mkfs to create. You need at least one journal per machine that will mount the filesystem. This number should have been 3, but I always add one more, in case a add a member in the cluster. This number is important, if I had put 3 and I added a node within the cluster and I wanted the 4 nodes to mount simultaneously this filesystem, I would need to make sure that the filesystem have 4 journals, because the GFS wound not mount. You can always add a journal to an existing GFS filesystem with the “gfs_jadd” command. Each journal reserve 128 MB in the filesystem, so you need take into consideration. Let look at our example, we want all our nodes to mount the “/cadmin” GFS filesystem, we created an logical volume of 1024M and on it we created a GFS, we reserved 4 journals (4*128=512MB) , so will have only around 500 MB available for data out of the 1024MB we allocated to our logical volume.
The last parameter “/dev/sharevg/cadmlv” is the name of the logical volume we created previously.
# /sbin/gfs_mkfs -O -p lock_dlm -t our_cluster:cadmlv -j 4 /dev/sharevg/cadmlvDevice: /dev/sharevg/cadmlvBlocksize: 4096Filesystem Size: 98260Journals: 4Resource Groups: 8Locking Protocol: lock_dlmLock Table: our_cluster:cadminlvSyncing…All Done
We are now able to mount our GFS filesystem on all the servers, by using the command below on all the servers,
# mount -t gfs /dev/sharevg/cadminlv /cadmin
We want that filesystem to be mounted every time a server boot, so don’t forget to add your filesystem to /etc/fstab, so it will mount after the next reboot and don’t forget to change the owner and protection of the filesystem.
# echo “/dev/sharevg/cadminlv /cadmin gfs defaults 0 0″ >> /etc/fstab
The filesystem should be available on all our nodes.
# df -h /cadminFilesystem Size Used Avail Use% Mounted on/dev/mapper/sharevg-cadminlv 510 M 1.5M 508M 1% /cadmin
So we’ve just created our first GFS filesystem and it is mounted on all our nodes in the cluster.
In our last article we will finalise our cluster, by creating the needed scripts for our ftp/web services to start and to move from server to server. We will add these scripts to our cluster configuration and we will show you how to move service from one server to another using the command line and GUI. So stay tune, for this last article on how to build a Red Hat cluster.
Creating a Red Hat Cluster: Part 5Welcome back to LINternUX, for our last article of this series on how to build a working Red Hat cluster. So far we have a working cluster, but it only move the IP from server to server. In this article, we will put in place everything so that we have an FTP and a web service that will be fully redundant within our cluster. In our previous article, we have create a GFS filesystem under the mount point “/cadmin”, this is where we will put our scripts, configuration files and log used for our cluster. The content of the “/cadmin” filesystem can be downloaded here, it include all the directories structure and scripts used in our cluster articles. After this article, you will have a fully configured cluster, running an ftp and a web service. We will have a lot to do, so let’s begin.
FTP prerequisiteWe need to make sure that the ftp server “vsftpd” is installed on every server in our cluster. You can check if it is installed by typing the following command;
root@gandalf:~# rpm -q vsftpdvsftpd-2.0.5-16.el5_5.1root@gandalf:~#
If is not installed, we need to run the following command to install it on the servers where it’s not installed;
root@bilbo:~# yum install vsftpd
We must make sure the vsftpd is not started and doesn’t start upon reboot. To do so use the following commands on all servers;
root@bilbo:~# service vsftpd stopShutting down vsftpd: [FAILED]root@bilbo:~# chkconfig vsftpd off
Script to stop/start/status our FTP serviceNow we need to create a script for each of our services (ftp and web) that the cluster software will use to stop and start the appropriate service and add it to our cluster configuration. We’ll put these scripts if our /cadmin GFS filesystem, so it’s accessible by our 3 servers. We will start by creating the script for the ftp service. The script used by the Red Hat Cluster Suite, receive one parameter when called by the cluster software. This parameter can be “stop”, “start” and “status”. You can download a copy of the script and the vsftpd configuration file if you want. But remember that if you want to use them as is, you must put them in the /cadmin filesystem. The “srv_ftp.sh” script will go in a subdirectory name “/cadmin/srv” and the configuration file “srv_ftp.conf” must go in “/cadmin/cfg” directory. But nothing beat an example; let’s build the one for our FTP service.
#! /bin/bash
# ---------------------------------------------------------------------------------
# Script to stop/start and give a status of ftp service in the cluster.
# This script is build to receive 3 parameters.
# - start : Executed by cluster to start the application(s) or service(s)
# - stop : Executed by cluster to stop the application(s) or service(s)
# - status: Executed by cluster every 30 seconds to check service status.
# -------------------------------------------------------------------------
# Author : Jacques Duplessis - April 2011
# -------------------------------------------------------------------------
#set -x
CDIR="/cadmin" ; export CDIR # Root directory for Services
CSVC="$CDIR/srv" ; export CSVC # Service Scripts Directory
CCFG="$CDIR/cfg" ; export CCFG # Service Config. Directory
INST="srv_ftp" ; export INST # Service Instance Name
LOG="$CDIR/log/${INST}.log" ; export LOG # Service Log file name
HOSTNAME=`hostname -a` ; export HOSTNAME # HostName
VSFTPD="/usr/sbin/vsftpd" ; export VSFTPD # Service Program name
FCFG="${CCFG}/${INST}.conf" ; export FCFG # Service Config. file name
RC=0 ; export RC # Service Return Code
DASH="---------------------"; export DASH # Dash Line
# Where the Action Start
# -------------------------------------------------------------------------
case "$1" in
start) echo -e "\n${DASH}" >> $LOG 2>&1
echo -e "Starting service $INST on $HOSTNAME at `date`" >> $LOG 2>&1
echo -e "${VSFTPD} ${FCFG}" >> $LOG 2>&1
${VSFTPD} ${FCFG} >> $LOG 2>&1
RC=$?
FPID=`ps -ef |grep -v grep |grep ${FCFG} |awk '{ print $2 }'|head -1`
echo "Service $INST started on $HOSTNAME - PID=${FPID} RC=$RC">> $LOG
echo "${DASH}" >> $LOG 2>&1
;;
stop ) echo -e "\n${DASH}" >> $LOG 2>&1
echo -e "Stopping Service $INST on $HOSTNAME at `date` " >> $LOG
ps -ef | grep ${FCFG}| grep -v grep >> $LOG 2>&1
FPID=`ps -ef |grep -v grep |grep ${FCFG} |awk '{ print $2 }'|head -1`
echo -e "Killing PID ${FPID}" >> $LOG 2>&1
kill $FPID >> $LOG 2>&1
echo -e "Service $INST is stopped ..." >> $LOG 2>&1
RC=0
echo "${DASH}" >> $LOG 2>&1
;;
status) COUNT=`ps -ef | grep ${FCFG}| grep -v grep | wc -l`
FPID=`ps -ef |grep -v grep |grep ${FCFG} |awk '{ print $2 }'|head -1`
echo -n "`date` Service $INST ($COUNT) on $HOSTNAME">> $LOG 2>&1
if [ $COUNT -gt 0 ]
then echo " - PID=${FPID} - OK" >> $LOG 2>&1
RC=0
else echo " - NOT RUNNING" >> $LOG 2>&1
ps -ef | grep -i ${FCFG} | grep -v grep >> $LOG 2>&1
RC=1
fi
;;
esac
exit $RC
This script is placed in the directory “/cadmin/svc” and name “svc_ftp.sh”. Now, let’s add this script to our ftp service, run the “system-config-cluster” command to start the cluster configuration GUI.
Add ftp script to our ftp cluster serviceNow, let’s add this script to our ftp service, run the “system-config-cluster” command.
root@gandalf:~# system-config-cluster &
Click on “Resources” on the left side and then on the “Create a Resource”‘ button at the bottom right of the screen. This will allow us insert our ftp service script into the cluster configuration.
Select “Script” from the “Resource Type” list and then enter the name of our ressource “srv_ftp” and then specify the name of the script the service will use, with it’s full path. Here, like I said we decided to place it in our “/cadmin” GFS filesystem so it is seen by every node in the cluster.
Now we need to edit our “srv_ftp” service to add the resource we just created.
Select the “srv_ftp” service at the bottom left of the screen and then press the “Edit Service Properties” button.
Click on the “Add a Shared Resource to this service” button. This will bring up the screen below, where we select the “srv_ftp” script that we want to add our service.
After adding our script to the resource, press the “Close” button.
We are now ready to push our new configuration to the member of our cluster, press the “Send to Cluster” button to do so.
Web site prerequisiteMake sure that the “httpd” and the “php” package is installed on every server in our cluster. You can check if it is installed by typing the following command;
root@gandalf # rpm -q httpd phphttpd-2.2.3-45.el5php-5.1.6-27.el5_5.3root@gandalf #
#
If is not installed, we need to run the following command to install them on the servers where it’s not installed;
root@bilbo:~# yum install httpd php
We must make sure the “httpd” is not started and doesn’t start upon reboot. To do so use the following commands on all servers;
root@bilbo:~# service httpd stopShutting down httpd: [FAILED]root@bilbo:~# chkconfig httpd off
Script to stop/start/status our Web serviceWe have simplified the configuration of our web site to the minimum. This was done intentionally; we wanted to demonstrate the cluster functionally and not the “httpd” possibilities. But our web site will be functional and redundant. As with the ftp script, the function of our web server script is very similar. You can download this script and the httpd configuration file if you want, but remember that if you want to use them as is, you must put them in the /cadmin filesystem. The “srv_www.sh” script will go in a subdirectory name “/cadmin/srv” and the configuration file “srv_www.conf” must go in “/cadmin/cfg” directory.
#! /bin/bash
# -------------------------------------------------------------------------
# Script to stop/start and give a status of our web service in the cluster.
# This script is build to receive 3 parameters.
# - start : Executed by cluster to start the application(s) or service(s)
# - stop : Executed by cluster to stop the application(s) or service(s)
# - status: Executed by cluster every 30 seconds to check service status.
# ------------------------------------------------------------------------
# Author : Jacques Duplessis - April 2011
# -------------------------------------------------------------------------
#set -x
CDIR="/cadmin" ; export CDIR # Root directory for Services
CSVC="$CDIR/srv" ; export CSVC # Service Scripts Directory
CCFG="$CDIR/cfg" ; export CCFG # Service Config. Directory
INST="srv_www" ; export INST # Service Instance Name
LOG="$CDIR/log/${INST}.log" ; export LOG # Service Log file name
HOSTNAME=`hostname -a` ; export HOSTNAME # HostName
HTTPD="/usr/sbin/httpd" ; export HTTPD # Service Program name
HCFG="${CCFG}/${INST}.conf" ; export HCFG # Service Config. file name
RC=0 ; export RC # Service Return Code
DASH="---------------------"; export DASH # Dash Line
# Where the Action Start
# -------------------------------------------------------------------------
case "$1" in
start) echo -e "\n${DASH}" >> $LOG 2>&1
echo -e "Starting service $INST on $HOSTNAME at `date`" >> $LOG 2>&1
echo -e "${HTTPD} ${HCFG}" >> $LOG 2>&1
${HTTPD} -f ${HCFG} >> $LOG 2>&1
RC=$?
HPID=`cat ${CCFG}/${INST}.pid`
echo "Service $INST started on $HOSTNAME - PID=${HPID} RC=$RC">> $LOG
echo "${DASH}" >> $LOG 2>&1
;;
stop ) echo -e "\n${DASH}" >> $LOG 2>&1
echo -e "Stopping Service $INST on $HOSTNAME at `date` " >> $LOG
HPID=`cat ${CCFG}/${INST}.pid`
echo -e "Killing PID ${HPID}" >> $LOG 2>&1
kill $HPID > /dev/null 2>&1
echo -e "Service $INST is stopped ..." >> $LOG 2>&1
RC=0
echo "${DASH}" >> $LOG 2>&1
;;
status) COUNT=`ps -ef | grep ${HCFG}| grep -v grep | wc -l`
HPID=`cat ${CCFG}/${INST}.pid`
echo -n "`date` Service $INST ($COUNT) on $HOSTNAME">> $LOG 2>&1
if [ $COUNT -gt 0 ]
then echo " - PID=${HPID} - OK" >> $LOG 2>&1
RC=0
else echo " - NOT RUNNING" >> $LOG 2>&1
ps -ef | grep -i ${HCFG} | grep -v grep >> $LOG 2>&1
RC=1
fi
;;
esac
exit $RC
Updating our cluster Configuration
To add our web service, please follow the same sequence as we did when we inserted our ftp service into the cluster configuration. You only need to replace “srv_ftp.sh” by “srv_www.sh” and the script path will be the same, we have decide to place our scripts into the directory “/cadmin/srv”. Once we have push the new configuration to all servers in the cluster, we should now have a working cluster.The web site define in the configuration have its “Root Directory” set to “/cadmin/www/html” it contains only one file that will display the name of the it is running on. The will help us testing our cluster configuration.
I you wish to use the cluster configuration, scripts and configuration files we have used in this series of articles, I would encourage you to download the “cadmin.tar” file. The file is the actual content of the “/cadmin” directory used throught out this article. To use it, download the “cadmin.tar” file then copy it to your “/cadmin” directory and enter the command “tar -xvf ./cadmin.tar”. This will explode the tar file and then you will have the working envirionnment I used in this article.
Testing our ftp serviceSo here we are (finally you would say … hum me too), we have now a fully working cluster. So if we issue the “clustat” command this is what we should see.
root@gollum:/# clustat
Cluster Status for our_cluster @ Sat Apr 16 11:37:25 2011
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
hbbilbo.maison.ca 1 Online, rgmanager
hbgandalf.maison.ca 2 Online, rgmanager
hbgollum.maison.ca 3 Online, Local, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:srv_ftp hbgollum.maison.ca started
service:srv_www hbbilbo.maison.ca started
root@gollum:/#
From the information above, we can see that all our cluster member status are online and that the resource manager is running on all of them. The resource manager is important, it is responsable for moving service around when needed. Our service “srv_ftp” is started (running) on the “hbgollum” server and “srv_www” is running on the “hbbilbo” like we decided at the beginning (remember ?
root@gollum:/# ip addr show | grep 192
inet 192.168.1.104/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.204/24 scope global secondary eth0
root@gollum:/#
root@gollum:/# ps -ef | grep vsftpd | grep -v grep
root 7858 1 0 10:05 ? 00:00:00 /usr/sbin/vsftpd /cadmin/cfg/srv_ftp.conf
root@gollum:/#
The command “ip addr show | grep 192″ is confirming that the virtual IP is defined on “hbgollum” server and if we check if the ftp process is also running, we can see that it is. So let’s try to do an FTP to our virtual IP that we have name “ftp.maison.ca” (192.168.1.204). We will try it from the “gandalf” server and we see that it is working.
root@gandalf:/# ftp ftp.maison.ca
Connected to ftp.maison.ca.
220 ftp.maison.ca
530 Please login with USER and PASS.
530 Please login with USER and PASS.
KERBEROS_V4 rejected as an authentication type
Name (ftp.maison.ca:root):
Now let’s move the ftp service from “hbgollum” to “hbbilbo”, to see if the ftp service will continue to work. To move the service we will use the “clusvcadm” command, we need to specify the service name we need to relocate (-r) and the server (-m for machine) we wish to move it. You can issue the “clusvccmd” command on any of the server within our cluster. So enter the following command to move our service to “hbbilbo” ;
root@gandalf:/# clusvcadm -r srv_ftp -m hbbilbo
'hbbilbo' not in membership list
Closest match: 'hbbilbo.maison.ca'
Trying to relocate service:srv_ftp to hbbilbo.maison.ca...Success
service:srv_ftp is now running on hbbilbo.maison.ca
root@gandalf:/#
Notice that just after pressing [Enter], we got “‘hbbilbo’ nit in membership list”, this is because we did not mention the domain name ‘maison.ca’, but it managed to assume that we were refering to “hbbilbo.maison.ca”. So our command succeeded, so let’s see if everything went like it should have.
Fisrt, let’s execute the clustat command to see in the “srv_ftp” service is now running on ‘hbbilbo’.
root@gandalf:/# clustat
Cluster Status for our_cluster @ Sat Apr 16 12:10:01 2011
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
hbbilbo.maison.ca 1 Online, rgmanager
hbgandalf.maison.ca 2 Online, Local, rgmanager
hbgollum.maison.ca 3 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:srv_ftp hbbilbo.maison.ca started
service:srv_www hbbilbo.maison.ca started
root@gandalf:/#
We can see that our ftp service is now running on “hbbilbo”, let’s see if it reallly is. If we check if the 192.168.1.204 (ftp.maison,ca) is now defined on “hbbilbo’ we can see that it is. The FTP process is also running now on the server.
root@bilbo:~# ip addr show | grep 192
inet 192.168.1.111/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.211/24 scope global secondary eth0
inet 192.168.1.204/24 scope global secondary eth0
root@bilbo:~# ps -ef | grep vsftpd | grep -v grep
root 8616 1 0 12:05 ? 00:00:00 /usr/sbin/vsftpd /cadmin/cfg/srv_ftp.conf
root@bilbo:~#
But what happen on ‘hbgollum’, the IP 192.168.1.204 should not be there anymore and the FTP process should not be running anymore. So that’s what happen the IP is gone and the ftp process is no longer running. So far so good, the moves to ‘hbbilbo’ server have worked.
root@gollum:/etc/profile.d# ip addr show | grep 192
inet 192.168.1.104/24 brd 192.168.1.255 scope global eth0
root@gollum:/etc/profile.d# ps -ef | grep vsftpd | grep -v grep
root@gollum:/etc/profile.d#
The last test if to try to do an ftp to ftp.maison.ca and see if it responds.
root@gandalf:/# ftp ftp.maison.ca
Connected to ftp.maison.ca.
220 ftp.maison.ca
530 Please login with USER and PASS.
530 Please login with USER and PASS.
KERBEROS_V4 rejected as an authentication type
Name (ftp.maison.ca:root):
Great evrything worked !! Let’s move the ftp process back to ‘hbgollum’ before testing our web site. Open another terminal window and enter the “clustat -i 2″ command and see watch the status change from ‘started/stopping/starting/started’ while the move is going on. Check your /var/log/message and familiar yourself with the line recorded when the move happen.
root@gandalf:/# clusvcadm -r srv_ftp -m hbgollum
'hbgollum' not in membership list
Closest match: 'hbgollum.maison.ca'
Trying to relocate service:srv_ftp to hbgollum.maison.ca...Success
service:srv_ftp is now running on hbgollum.maison.ca
root@gandalf:/#
One of the test we should make, is to unplug the network cable or poweroff “hbgollum” and see if the service move to the next server in the failover domain (It will). So we have now completed and tested our ftp service. It as been a long road but it woth it, no ?
Testing our web serviceYou know now, how to move service from server to another, Let’s do the same test with our web server service. The web site is actually just one simple page. It just display the name of the server that it is running on, this simplify our testing.
If we issue the “clustat” command we should have the following picture;
# clustat
Cluster Status for our_cluster @ Sat Apr 16 14:15:37 2011
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
hbbilbo.maison.ca 1 Online, rgmanager
hbgandalf.maison.ca 2 Online, Local, rgmanager
hbgollum.maison.ca 3 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:srv_ftp hbgollum.maison.ca started
service:srv_www hbbilbo.maison.ca started
Let’s see if it is working, open your browser type this URL “http://www.maison.ca”, you should have a the following;
Now, let’s move the wev site to “gandalf” server, type the following command;
root@gollum:/cadmin/cfg# clusvcadm -r srv_www -m hbgandalf
'hbgandalf' not in membership list
Closest match: 'hbgandalf.maison.ca'
Trying to relocate service:srv_www to hbgandalf.maison.ca...Success
service:srv_www is now running on hbgandalf.maison.ca
root@gollum:/cadmin/cfg#
root@gollum:/cadmin/cfg# clustat
Cluster Status for our_cluster @ Sat Apr 16 14:27:14 2011
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
hbbilbo.maison.ca 1 Online, Local, rgmanager
hbgandalf.maison.ca 2 Online, rgmanager
hbgollum.maison.ca 3 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:srv_ftp hbgollum.maison.ca started
service:srv_www hbgandalf.maison.ca started
We can see that the web site is now running on “gandalf” server.
Disabling and Enabling ServicesThere may come a time, when you need to stop a service completely. We will demonstrate how to acheive that, first let’s display the status of our cluster
root@bilbo:~# clustat
Cluster Status for our_cluster @ Sat Apr 16 14:39:48 2011
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
hbbilbo.maison.ca 1 Online, Local, rgmanager
hbgandalf.maison.ca 2 Online, rgmanager
hbgollum.maison.ca 3 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:srv_ftp hbgollum.maison.ca started
service:srv_www hbgandalf.maison.ca started
We are going to disable the “srv_www” service, to do so enter the following command;
root@bilbo:~# clusvcadm -d srv_www
Local machine disabling service:srv_www...Success
The clustat command show us that the service is now disable.
root@bilbo:~# clustat
Cluster Status for our_cluster @ Sat Apr 16 14:40:04 2011
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
hbbilbo.maison.ca 1 Online, Local, rgmanager
hbgandalf.maison.ca 2 Online, rgmanager
hbgollum.maison.ca 3 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:srv_ftp hbgollum.maison.ca started
service:srv_www (hbgandalf.maison.ca) disabled
We will now enable the service, but this time we will enable it on another server than “hbgandalf”. This command enables the “srv_www” service on the server “hbbilbo”.
root@bilbo:~# clusvcadm -e srv_www -m hbbilbo
'hbbilbo' not in membership list
Closest match: 'hbbilbo.maison.ca'
Member hbbilbo.maison.ca trying to enable service:srv_www...Success
service:srv_www is now running on hbbilbo.maison.ca
root@bilbo:~#
We can see the it is now running on “hbbilbo”.
root@bilbo:~# clustat
Cluster Status for our_cluster @ Sat Apr 16 14:47:36 2011
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
hbbilbo.maison.ca 1 Online, Local, rgmanager
hbgandalf.maison.ca 2 Online, rgmanager
hbgollum.maison.ca 3 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:srv_ftp hbgollum.maison.ca started
service:srv_www hbbilbo.maison.ca started
root@bilbo:~#
This concludes our implementation of a small cluster. It was intended just to show everyone how the Red Hat Cluster Suite actually work and to give a brief overview how it work. We will now move on to other interesting topic. Don’t know what it will be, but I can assure that it should fit into one article, so I hope you appreciate it and hope to see you soon.