VCS- Implementing Veritas Cluster File System (CFS)

Veritas Cluster File System (CFS)

CFS allows the same file system to be simultaneously mounted on multiple nodes in the cluster.

The CFS is designed with master/slave architecture. Though any node can initiate an operation to create, delete, or resize data, the master node carries out the actual operation. CFS caches the metadata in memory, typically in the memory buffer cache or the vnode cache. A distributed locking mechanism, called GLM, is used for metadata and cache coherency among the multiple nodes.

The examples here are :

1. Based on VCS 5.x but should also work on 4.x
2. A new 4 node cluster with no resources defined.
3. Diskgroups and volumes will be created and shared across all nodes.

Before you configure CFS

1. Make sure you have an established Cluster and running properly.
2. Make sure these packages are installed on all nodes:

VRTScavf Veritas cfs and cvm agents by Symantec
VRTSglm Veritas LOCK MGR by Symantec

3. Make sure you have a license installed for Veritas CFS on all nodes.
4. Make sure vxfencing driver is active on all nodes (even if it is in disabled mode).

Check the status of the cluster

Here are some ways to check the status of your cluster. On these examples, CVM/CFS are not configured yet.

# cfscluster status
 NODE         CLUSTER MANAGER STATE            CVM STATE
serverA        running                        not-running                   
serverB        running                        not-running                   
serverC        running                        not-running                   
serverD        running                        not-running                   

 Error: V-35-41: Cluster not configured for data sharing application

# vxdctl -c mode
mode: enabled: cluster inactive

# /etc/vx/bin/vxclustadm nidmap
Out of cluster: No mapping information available

# /etc/vx/bin/vxclustadm -v nodestate
state: out of cluster

# hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen             

A  serverA             RUNNING              0                   
A  serverB             RUNNING              0                   
A  serverC             RUNNING              0                   
A  serverD             RUNNING              0

Configure the cluster for CFS

During
configuration, veritas will pick up all information that is set on your
cluster configuration. And will activate CVM on all the nodes.

# cfscluster config

       The cluster configuration information as read from cluster
       configuration file is as follows.
               Cluster : MyCluster
               Nodes   : serverA serverB serverC serverD


       You will now be prompted to enter the information pertaining
       to the cluster and the individual nodes.

       Specify whether you would like to use GAB messaging or TCP/UDP
       messaging. If you choose gab messaging then you will not have
       to configure IP addresses. Otherwise you will have to provide
       IP addresses for all the nodes in the cluster.
 
       ------- Following is the summary of the information: ------
               Cluster         : MyCluster
               Nodes           : serverA serverB serverC serverD
               Transport       : gab
       -----------------------------------------------------------


       Waiting for the new configuration to be added.

       ========================================================

       Cluster File System Configuration is in progress...
       cfscluster: CFS Cluster Configured Successfully

Check the status of the cluster

Now let's check the status of the cluster. And notice that there is now a new service group cvm. CVM is required to be online before we can bring up any clustered filesystem on the nodes.

# cfscluster status

 Node             :  serverA
 Cluster Manager  :  running
 CVM state        :  running
 No mount point registered with cluster configuration


 Node             :  serverB
 Cluster Manager  :  running
 CVM state        :  running
 No mount point registered with cluster configuration


 Node             :  serverC
 Cluster Manager  :  running
 CVM state        :  running
 No mount point registered with cluster configuration


 Node             :  serverD
 Cluster Manager  :  running
 CVM state        :  running
 No mount point registered with cluster configuration

# vxdctl -c mode
mode: enabled: cluster active - MASTER
master: serverA

# /etc/vx/bin/vxclustadm nidmap
Name                             CVM Nid    CM Nid     State
serverA                         0          0          Joined: Master
serverB                         1          1          Joined: Slave
serverC                         2          2          Joined: Slave
serverD                         3          3          Joined: Slave

# /etc/vx/bin/vxclustadm -v nodestate
state: cluster member
       nodeId=0
       masterId=1
       neighborId=1
       members=0xf
       joiners=0x0
       leavers=0x0
       reconfig_seqnum=0xf0a810
       vxfen=off

# hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen             

A  serverA             RUNNING              0                   
A  serverB             RUNNING              0                   
A  serverC             RUNNING              0                   
A  serverD             RUNNING              0                   

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State         

B  cvm             serverA             Y          N               ONLINE        
B  cvm             serverB             Y          N               ONLINE        
B  cvm             serverC             Y          N               ONLINE        
B  cvm             serverD             Y          N               ONLINE