Re-adding a repaired node to a clustered system by using the CLI

Edit online

You can use the command-line interface (CLI) to re-add a failed node back into a clustered system after it was repaired.

Before you begin

Before you add a node to a clustered system, you must make sure that the switchd\ zoning is configured such that the node that is being added is in the same zone as all other nodes in the clustered system. If you are replacing a node and the switch is zoned by worldwide port name (WWPN) rather than by switch port, make sure that the switch is configured such that the node that is being added is in the same VSAN/zone.

Attention:
  1. If you are re-adding a node to the SAN, ensure that you are adding the node to the same I/O group from which it was removed. Failure to select the correct I/O group can result in data corruption. You must use the information that was recorded when the node was originally added to the clustered system. If you do not have access to this information, call the IBM® Support Center to add the node back into the clustered system without corrupting the data.
  2. The LUNs that are presented to the ports on the new node must be the same as the LUNs that are presented to the nodes that currently exist in the clustered system. You must ensure that the LUNs are the same before you add the new node to the clustered system.
  3. LUN masking for each LUN must be identical on all nodes in a clustered system. You must ensure that the LUN masking for each LUN is identical before you add the new node to the clustered system.
  4. You must ensure that the model type of the new node is supported by the SAN Volume Controller software level that is installed on the clustered system. If the model type is not supported by the SAN Volume Controller software level, update the clustered system to a software level that supports the model type of the new node. See the following website for the latest supported software levels:

    www.ibm.com/support

About this task

Special procedures when you add a node to a clustered system

Applications on the host systems direct I/O operations to file systems or logical volumes that are mapped by the operating system to virtual paths (vpaths), which are pseudo disk objects that are supported by the Subsystem Device Driver (SDD). SDD maintains an association between a vpath and a SAN Volume Controller volume. This association uses an identifier (UID) which is unique to the volume and is never reused. The UID permits SDD to directly associate vpaths with volumes.

SDD operates within a protocol stack that contains disk and Fibre Channel device drivers that are used to communicate with the SAN Volume Controller using the SCSI protocol over Fibre Channel as defined by the ANSI FCS standard. The addressing scheme that is provided by these SCSI and Fibre Channel device drivers uses a combination of a SCSI logical unit number (LUN) and the worldwide node name (WWNN) for the Fibre Channel node and ports.

If an error occurs, the error recovery procedures (ERPs) operate at various tiers in the protocol stack. Some of these ERPs cause I/O to be redriven by using the same WWNN and LUN numbers that were previously used.

SDD does not check the association of the volume with the vpath on every I/O operation that it performs.

Before you add a node to the clustered system, you must check to see if any of the following conditions are true:
  • The clustered system has more than one I/O group.
  • The node that is being added to the clustered system uses physical node hardware or a slot that has previously been used for a node in the clustered system.
  • The node that is being added to the clustered system uses physical node hardware or a slot that has previously been used for a node in another clustered system and both clustered systems have visibility to the same hosts and back-end storage.
If any of the previous conditions are true, the following special procedures apply:
  • The node must be added to the same I/O group that it was previously in. You can use the command-line interface (CLI) command lsnode or the management GUI to determine the WWN of the clustered system nodes.
  • Before you add the node back into the clustered system, you must shut down all of the hosts using the clustered system. The node must then be added before the hosts are rebooted. If the I/O group information is unavailable or it is inconvenient to shut down and reboot all of the hosts by using the clustered system, then do the following:
    • On all of the hosts that are connected to the clustered system, unconfigure the Fibre Channel adapter device driver, the disk device driver, and multipathing driver before you add the node to the clustered system.
    • Add the node to the clustered system, and then reconfigure the Fibre Channel adapter device driver, the disk device driver, and multipathing driver.

Scenarios where the special procedures can apply

The following two scenarios describe situations where the special procedures can apply:
  • Four nodes of an eight-node clustered system have been lost because of the failure of a pair of 2145 UPS or four 2145 UPS-1U . In this case, the four nodes must be added back into the clustered system by using the CLI command addnode or the management GUI.
    Note: You do not need to run the addnode command on a node with a partner that is already in a clustered system; the clustered system automatically detects an online candidate.
  • A user decides to delete four nodes from the clustered system and add them back into the clustered system using the CLI command addnode or the management GUI.

For 5.1.0 nodes, the SAN Volume Controller automatically re-adds nodes that failed back to the clustered system. If the clustered system reports an error for a node missing (error code 1195) and that node has been repaired and restarted, the clustered system automatically re-adds the node back into the clustered system. This process can take up to 20 minutes, so you can manually re-add the node by completing the following steps:

Procedure

  1. Issue the lsnode CLI command to list the nodes that are currently part of the clustered system and determine the I/O group for which to add the node.

    The following is an example of the output that is displayed:

    lsnode -delim :

id:name:UPS_serial_number:WWNN:status:IO_group_id:IO_group_name
:config_node:UPS_unique_id:hardware:iscsi_name:iscsi_alias
:panel_name:enclosure_id:canister_id:enclosure_serial_number
1:node1::50050868010050B2:online:0:io_grp0:yes::100:iqn.1986-03.com.ibm
:2145.cluster0.node1::02-1:2:1:123ABCG
2:node2::50050869010050B2:online:0:io_grp0:no::100:iqn.1986-03.com.ibm
:2145.cluster0.node2::02-2:2:2:123ABDG
    lsnode -delim :

Id:name:UPS_serial_number:WWNN:status:IO_group_id:IO_group_name
:config_node:UPS_unique_id:hardware:iscsi_name:iscsi_alias
:panel_name:enclosure_id:canister_id:enclosure_serial_number
1:node1:UPS_Fake_SN:50050768010050B1:online:0:io_grp0:yes
:10000000000050B1:8G4:iqn.1986-03.com.ibm:2145.cluster0.node1:000368:::
  2. Issue the lsnodecandidate CLI command to list nodes that are not assigned to a clustered system and to verify that a second node is added to an I/O group.

    The following is an example of the output that is displayed:

    lsnodecandidate -delim :

id:panel_name:UPS_serial_number:UPS_unique_id:hardware
5005076801000001:000341:10L3ASH:202381001C0D18D8:8A4
5005076801000009:000237:10L3ANF:202381001C0D1796:8A4
50050768010000F4:001245:10L3ANF:202381001C0D1796:8A4
....
  3. Issue the addnode CLI command to add a node to the clustered system.
    Important: Each node in an I/O group must be attached to a different uninterruptible power supply.

    The following is an example of the CLI command you can issue to add a node to the clustered system by using the panel name parameter:

    addnode -panelname 000237
 -iogrp io_grp0

    Where 000237 is the panel name of the node, io_grp0 is the name of the I/O group that you are adding the node to.

    The following is an example of the CLI command you can issue to add a node to the clustered system by using the WWNN parameter:

    addnode -wwnodename 5005076801000001
 -iogrp io_grp1

    Where 5005076801000001 is the WWNN of the node, io_grp1 is the name of the I/O group that you are adding the node to.

  4. Issue the lsnode CLI command to verify the final configuration.

    The following example shows output that is displayed:

    lsnode -delim :

id:name:UPS_serial_number:WWNN:status:IO_group_id:IO_group_name:config_node:UPS_unique_id:
hardware:iscsi_name:iscsi_alias
1:node1:10L3ASH:0000000000000000:offline:0:io_grp0:no:1000000000003206:
8A4:iqn.1986-03.com.ibm:2145.ndihill.node1:
    Record the following information for the new node:
    • Node name
    • Node serial number
    • WWNN
    • IQNs (if using hosts attached by using iSCSI connections)
    • All WWPNs
    • I/O group that contains the node
    Note: If this command is issued quickly after you add nodes to the clustered system, the status of the nodes might be adding. The status is shown as adding if the process of adding the nodes to the clustered system is still in progress. You do not have to wait for the status of all the nodes to be online before you continue with the configuration process.

Results

The nodes are added to the clustered system.