How to failover a VCS Service Group ?

For Example: You have a service group(sg1) currently running on nodeA and you want to fail it over to node B. Then you need to run the below command to perform the failover.

hagrp -switch sg1 to nodeB

VCS- Implementing Veritas Cluster File System (CFS)

Veritas Cluster File System (CFS)

CFS allows the same file system to be simultaneously mounted on multiple nodes in the cluster.

The CFS is designed with master/slave architecture. Though any node can initiate an operation to create, delete, or resize data, the master node carries out the actual operation. CFS caches the metadata in memory, typically in the memory buffer cache or the vnode cache. A distributed locking mechanism, called GLM, is used for metadata and cache coherency among the multiple nodes.

The examples here are :

1. Based on VCS 5.x but should also work on 4.x
2. A new 4 node cluster with no resources defined.
3. Diskgroups and volumes will be created and shared across all nodes.

Before you configure CFS

1. Make sure you have an established Cluster and running properly.
2. Make sure these packages are installed on all nodes:

VRTScavf Veritas cfs and cvm agents by Symantec
VRTSglm Veritas LOCK MGR by Symantec

3. Make sure you have a license installed for Veritas CFS on all nodes.
4. Make sure vxfencing driver is active on all nodes (even if it is in disabled mode).

Check the status of the cluster

Here are some ways to check the status of your cluster. On these examples, CVM/CFS are not configured yet.

# cfscluster status
 NODE         CLUSTER MANAGER STATE            CVM STATE
serverA        running                        not-running                   
serverB        running                        not-running                   
serverC        running                        not-running                   
serverD        running                        not-running                   

 Error: V-35-41: Cluster not configured for data sharing application

# vxdctl -c mode
mode: enabled: cluster inactive

# /etc/vx/bin/vxclustadm nidmap
Out of cluster: No mapping information available

# /etc/vx/bin/vxclustadm -v nodestate
state: out of cluster

# hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen             

A  serverA             RUNNING              0                   
A  serverB             RUNNING              0                   
A  serverC             RUNNING              0                   
A  serverD             RUNNING              0

Configure the cluster for CFS

During
configuration, veritas will pick up all information that is set on your
cluster configuration. And will activate CVM on all the nodes.

# cfscluster config

       The cluster configuration information as read from cluster
       configuration file is as follows.
               Cluster : MyCluster
               Nodes   : serverA serverB serverC serverD


       You will now be prompted to enter the information pertaining
       to the cluster and the individual nodes.

       Specify whether you would like to use GAB messaging or TCP/UDP
       messaging. If you choose gab messaging then you will not have
       to configure IP addresses. Otherwise you will have to provide
       IP addresses for all the nodes in the cluster.
 
       ------- Following is the summary of the information: ------
               Cluster         : MyCluster
               Nodes           : serverA serverB serverC serverD
               Transport       : gab
       -----------------------------------------------------------


       Waiting for the new configuration to be added.

       ========================================================

       Cluster File System Configuration is in progress...
       cfscluster: CFS Cluster Configured Successfully

Check the status of the cluster

Now let's check the status of the cluster. And notice that there is now a new service group cvm. CVM is required to be online before we can bring up any clustered filesystem on the nodes.

# cfscluster status

 Node             :  serverA
 Cluster Manager  :  running
 CVM state        :  running
 No mount point registered with cluster configuration


 Node             :  serverB
 Cluster Manager  :  running
 CVM state        :  running
 No mount point registered with cluster configuration


 Node             :  serverC
 Cluster Manager  :  running
 CVM state        :  running
 No mount point registered with cluster configuration


 Node             :  serverD
 Cluster Manager  :  running
 CVM state        :  running
 No mount point registered with cluster configuration

# vxdctl -c mode
mode: enabled: cluster active - MASTER
master: serverA

# /etc/vx/bin/vxclustadm nidmap
Name                             CVM Nid    CM Nid     State
serverA                         0          0          Joined: Master
serverB                         1          1          Joined: Slave
serverC                         2          2          Joined: Slave
serverD                         3          3          Joined: Slave

# /etc/vx/bin/vxclustadm -v nodestate
state: cluster member
       nodeId=0
       masterId=1
       neighborId=1
       members=0xf
       joiners=0x0
       leavers=0x0
       reconfig_seqnum=0xf0a810
       vxfen=off

# hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen             

A  serverA             RUNNING              0                   
A  serverB             RUNNING              0                   
A  serverC             RUNNING              0                   
A  serverD             RUNNING              0                   

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State         

B  cvm             serverA             Y          N               ONLINE        
B  cvm             serverB             Y          N               ONLINE        
B  cvm             serverC             Y          N               ONLINE        
B  cvm             serverD             Y          N               ONLINE

How to find a file or files using maximum space on Unix Systems like Solaris , Linux, HP-UX or AIX?

Example For Finding out the file bigger than 100MB

find /var -xdev -size +100000000c -exec ls -lh {} \;

-xdev is being used so that you can find out the file in the filesystem where /var is located. Many times we have other filesystems as well part of /var like /var/crash(which is mounted from a another Filesystem) . 

So, my advice is to always xdev to ensure that you are searching in the right directory.

HP Service Guard Basic Commands and Important Files

1. How to check the Service Guard Cluster Version?
root@nodeA:/# cmversion
A.11.17.00

2.  How to search for the existing SG clusters in the network?
cmquerycl to search existing Serviceguard clusters in the Network.

root@nodeA:/# cmquerycl -v
Looking for Serviceguard nodes ... Done
Cluster Name Node Name Version Status
Swrecovery
nodeA up to date
nodeB up to date

3. How to check the status of the cluster?

root@nodeA:/etc/cmcluster/sw# cmviewcl -v

CLUSTER STATUS

swrecovery up

NODE STATUS STATE

nodeA up running

nodeB up running

4. How to ensure that the Cluster will start at boot time?

Update: /etc/rc.config.d/cmcluster

AUTOSTART_CMCLD =1 

5. How to stop the cluster on both nodes?

#cmhaltcl -v

6. How to start the node in the cluster?

cmrunnode nodeB

7. How to halt a node in the cluster?

cmhaltnode -v nodeB

8. How to start and halt a package?

cmrunpkg -v sw-pkg

cmhaltpkg -v sw-pkg

Few more details.

# cmmodpkg –e pkg1 Globally enable switching for a package.

# cmmodpkg –d pkg1 Globally disable switching for a package.

# cmmodpkg –n nodeA –e pkg1 Enable a node to run a package.

# cmmodpkg –n nodeA –d pkg1 Disable a node to run a package