Replacing nodes nondisruptively

These procedures describe how to replace most nodes nondisruptively.

Before you begin

These procedures are nondisruptive because changes to your SAN environment are not required. The replacement (new) node uses the same worldwide node name (WWNN) as the node that you are replacing. An alternative to this procedure is to replace nodes disruptively either by moving volumes to a new I/O group or by rezoning the SAN. However, the disruptive procedures require additional work on the hosts.

This task assumes that these conditions are met:

  • The existing system software must be at a version that supports the new node. If a node is being replaced by a SAN Volume Controller 2145-CG8 node, the system software version must be 6.2.0 or later. If a node is being replaced by a SAN Volume Controller 2145-CF8 node, the system software version must be 5.1.0 or later. If a node is being replaced by a SAN Volume Controller 2145-8A4 node, the system software version must be 4.3.1 or later.
    Note: For nodes that contain flash drives: if the existing Flash drives are being moved to the new node, the new node must contain the necessary serial-attached SCSI (SAS) adapter to support Flash drives.
  • All nodes that are configured in the system are present and online.
  • All errors in the system event log are addressed and marked as fixed.
  • There are no volumes, managed disks (MDisks), or external storage systems with a status of degraded or offline.
  • The replacement node is not powered on.
  • The replacement node is not connected to the SAN.
  • You have a 2145 UPS-1U unit (feature code 8115) for each new SAN Volume Controller 2145-CG8 SAN Volume Controller 2145-CF8, or SAN Volume Controller 2145-8A4 node.
  • You backed up the system configuration and saved the svc.config.backup.xml file.
  • The replacement node must be able to operate at the Fibre Channel or Ethernet connection speed of the node it is replacing.
  • If the node being replaced contains flash drives, transfer all Flash drives and SAS adapters to the new node if it supports the drives. To prevent losing access to the data, if the new node does not support the existing Flash drives, transfer the data from the Flash drives before replacing the node.
Important:
  1. Do not continue this task if any of the conditions that are listed are not met unless you are instructed to do so by the IBM® Support Center.
  2. Review all of the steps that follow before you perform this task.
  3. Do not perform this task if you are not familiar with SAN Volume Controller environments or the procedures that are described in this task.
  4. If you plan to reuse the node that you are replacing, ensure that the WWNN of the node is set to a unique number on your SAN. If you do not ensure that the WWNN is unique, the WWNN and WWPN are duplicated in the SAN environment and can cause problems.
    Tip: You can change the WWNN of the node you are replacing to the factory default WWNN of the replacement node to ensure that the number is unique.
  5. The node ID and possibly the node name change during this task. After the system assigns the node ID, the ID cannot be changed. However, you can change the node name after this task is complete.

About this task

Perform these steps to replace active nodes in a system:

Procedure

  1. (If the system software version is at 5.1 or later, complete this step).

    Confirm that no hosts have dependencies on the node.

    When you shut down a node that is part of a system or you delete the node from a system, you can use either the management GUI or a command-line interface (CLI) command. In the management GUI, select Monitoring > System > Manage. Click Show Dependent Volumes to display all the volumes that depend on a node. You can also use the node parameter with the lsdependentvdisks CLI command to view dependent volumes.

    If dependent volumes exist, determine whether the volumes are being used. If the volumes are being used, either restore the redundant configuration or suspend the host application. If a dependent quorum disk is reported, repair the access to the quorum disk or modify the quorum disk configuration.

  2. Use these steps to determine the system configuration node, and the ID, name, I/O group ID, and I/O group name for the node that you want to replace. If you already know the physical location of the node that you want to replace, you can skip this step and proceed to step 3.
    Tip: If one of the nodes that you want to replace is the system configuration node, replace it last.
    1. Issue this command from the command-line interface (CLI): 
      lsnode -delim :

      This output is an example of the output that is displayed for this command:

      id:name:UPS_serial_number:WWNN:status:IO_group_id:IO_group_name:
config_node:UPS_unique_id:hardware:iscsi_name:iscsi_alias
3:dvt113294:100089J137:5005076801005A07:online:0:io_grp0:yes:
20400002096810C7:8A4:iqn.1986-03.com.ibm:2145.ldcluster-80.dvt113294:
14:des113004:10006BR010:5005076801004F0F:online:0:io_grp0:no:
2040000192880040:8G4:iqn.1986-03.com.ibm:2145.ldcluster-80.des113004:
    2. In the config_node column, find the value yes and record the values in the id and name columns.
    3. Record the values in the id and the name columns for each node in the system.
    4. Record the values in the IO_group_id and the IO_group_name columns for each node in the system.
    5. Issue this command from the CLI for each node in the system to determine the front panel ID: 
      lsnodevpd node_name or node_id

      where node_name or node_id is the name or ID of the node for which you want to determine the front panel ID.

    6. Record the value in the front_panel_id column. The front panel ID is displayed on the front of each node. You can use this ID to determine the physical location of the node that matches the node ID or node name that you want replace.
  3. Perform these steps to record the WWNN or iSCSI name of the node that you want to replace:
    1. Issue this command from the CLI: 
      lsnode -delim : node_name or node_id

      where node_name or node_id is the name or ID of the node for which you want to determine the WWNN or iSCSI name.

    2. Record the WWNN or iSCSI name of the node that you want to replace. Also, record the order of the Fibre Channel and Ethernet ports.
  4. Issue this command from the CLI to power off the node: 
    stopsystem -node node_name
    Important:
    1. Record and mark the order of the Fibre Channel or Ethernet cables with the node port number (port 1 - 4 for Fibre Channel, or port 1 - 2 for Ethernet) before you remove the cables from the back of the node. The Fibre Channel ports on the back of the node are numbered 1 - 4 from left to right. You must reconnect the cables in the exact order on the replacement node to avoid issues when the replacement node is added to the system. If the cables are not connected in the same order, the port IDs can change, which impacts the ability of the host to access volumes. See the hardware documentation specific to your model to determine how the ports are numbered.
    2. Do not connect the replacement node to different ports on the switch or director. The SAN Volume Controller can have 4 Gbps or 8 Gbps HBAs. However, do not move them to faster switch or director ports at this time to avoid issues when the replacement node is added to the system. This task is separate and must be planned independently of replacing nodes in a system.
  5. Issue this CLI command to delete this node from the system and I/O group: 
    rmnode node_name or node_id

    Where node_name or node_id is the name or ID of the node that you want to delete. You can use the CLI to verify that the deletion process is complete..

  6. Issue this CLI command to ensure that the node is no longer a member of the system: 
    lsnode
    A list of nodes is displayed. Wait until the removed node is not listed in the command output.
  7. Perform these steps to change the WWNN or iSCSI name of the node that you just deleted from the system to FFFFF:
    For SAN Volume Controller V6.1.0 or later:
    1. Power on the node. With the Cluster panel displayed, press the up or down button until the Actions option is displayed.
    2. Press and release the select button.
    3. Press the up or down button until Change WWNN? is displayed.
    4. Press and release the select button to display the current WWNN.
    5. Press and release the select button to switch into edit mode. The Edit WWNN? panel is displayed.
    6. Change the WWNN to FFFFF.
    7. Press and release the select button to exit edit mode.
    8. Press the right button to confirm your selection. The Confirm WWNN? panel is displayed.
    9. Press and release the select button to confirm.
  8. Install the replacement node and the uninterruptible power supply in the rack and connect the uninterruptible power supply cables.See the IBM SAN Volume Controller Model 2145-XXX Hardware Installation Guide to determine how to connect the node and the uninterruptible power supply.
    Important: Do not connect the Fibre Channel or Ethernet cables during this step.
  9. If you are removing Flash drives from an old node and inserting them into a new node, see the IBM SAN Volume Controller Hardware Maintenance Guide for specific instructions.
  10. Power on the replacement node.
  11. Record the WWNN of the replacement node. You can use this name if you plan to reuse the node that you are replacing.
  12. Perform these steps to change the WWNN name of the replacement node to match the name that you recorded in step 3:

    For SAN Volume Controller V6.1.0 or later:

    1. With the Cluster panel displayed, press the up or down button until the Actions option is displayed.
    2. Press and release the select button.
    3. Press the up or down button until Change WWNN? is displayed.
    4. Press and release the select button to display the current WWNN.
    5. Press the select button to switch into edit mode. The Edit WWNN? panel is displayed.
    6. Change the WWNN to the numbers that you recorded in step 3.
    7. Press and release the select button to exit edit mode.
    8. Press the right button to confirm your selection. The Confirm WWNN? panel is displayed.
    9. Press the select button to confirm.

    Wait one minute. If Cluster: is displayed on the front panel, the node is ready to be added to the system. If Cluster: is not displayed, see the troubleshooting information to determine how to address this problem or contact the IBM Support Center before you continue with the next step.

  13. Connect the Fibre Channel or Ethernet cables to the same port numbers that you recorded for the original node in step 4.
  14. Issue this CLI command to verify that the last five characters of the WWNN are correct: 
    lsnodecandidate
    Important: If the WWNN is not what you recorded in step 3, you must repeat step 12.
  15. Issue this CLI command to add the node to the system and ensure that the node has the same name as the original node and is in the same I/O group as the original node. See the addnode CLI command documentation for more information. 
    addnode -wwnodename WWNN -iogrp iogroupname/id

    WWNN and iogroupname/id are the values that you recorded for the original node.

    The SAN Volume Controller V5.1 and later automatically reassigns the node with the name that was used originally. For versions before V5.1, use the name parameter with the svctask addnode command to assign a name. If the original name of the node name was automatically assigned by SAN Volume Controller , it is not possible to reuse the same name. It was automatically assigned if its name starts with node. In this case, either specify a different name that does not start with node or do not use the name parameter so that SAN Volume Controller automatically assigns a new name to the node.

    If necessary, the new node is updated to the same SAN Volume Controller software version as the system. This update can take up to 20 minutes.

    Important:
    1. Both nodes in the I/O group cache data; however, the cache sizes are asymmetric. The replacement node is limited by the cache size of the partner node in the I/O group. Therefore, it is possible that the replacement node does not use the full cache size until you replace the other node in the I/O group.
    2. You do not have to reconfigure the host multipathing device drivers because the replacement node uses the same WWNN and WWPN as the previous node. The multipathing device drivers should detect the recovery of paths that are available to the replacement node.
    3. The host multipathing device drivers take approximately 30 minutes to recover the paths. Do not update the other node in the I/O group until for at least 30 minutes after you successfully updated the first node in the I/O group. If you have other nodes in different I/O groups to update, you can perform those updates while you wait.
  16. Query paths to ensure that all paths have been recovered before proceeding to the next step. If you are using the IBM Multipath Subsystem Device Driver (SDD), the command to query paths is datapath query device. Documentation that is provided with your multipathing device driver shows how to query paths.
  17. Repair the faulty node.

    If you want to use the repaired node as a spare node, perform these steps.

    For SAN Volume Controller V6.1.0 or later:
    1. With the Cluster panel displayed, press the up or down button until the Actions option is displayed.
    2. Press and release the select button.
    3. Press the up or down button until Change WWNN? is displayed.
    4. Press and release the select button to display the current WWNN.
    5. Press and release the select button to switch into edit mode. The Edit WWNN? panel is displayed.
    6. Change the WWNN to 00000.
    7. Press and release the select button to exit edit mode.
    8. Press the right button to confirm your selection. The Confirm WWNN? panel is displayed.
    9. Press and release the select button to confirm.

    This node can now be used as a spare node.

  18. Repeat steps 3 to 17 for each node that you want to replace.