Replacing nodes nondisruptively: 2145-CG8 or 2145-CF8

You can replace most nodes nondisruptively with SAN Volume Controller 2145-CG8 or 2145-CF8 nodes.

Before you begin

These procedures are nondisruptive because changes to your SAN environment are not required. The replacement (new) node uses the same worldwide node name (WWNN) as the node that you are replacing. An alternative to this procedure is to replace nodes disruptively either by moving volumes to a new I/O group or by rezoning the SAN. However, the disruptive procedures require more work on the hosts.

This task assumes that the following conditions are met. Do not continue this task if any of the conditions are not met, unless you are instructed to do so by the IBM® Remote Technical Support Center.

  • The existing system software must be at a version that supports the new node. If a node is being replaced by a SAN Volume Controller 2145-CG8 node, the system software version must be 6.2.0 or later. If a node is being replaced by a SAN Volume Controller 2145-CF8 node, the system software version must be 5.1.0 or later.
    Note: If the current node contains flash drives that are being moved, the new node must contain the necessary serial-attached SCSI (SAS) adapter to support flash drives.
  • All nodes that are configured in the system are present and online.
  • All errors in the system event log are addressed and marked as fixed.
  • No volumes, managed disks (MDisks), or external storage systems are in degraded or offline status.
  • The replacement node is not powered on.
  • The replacement node is not connected to the SAN.
  • You have a 2145 UPS-1U unit (feature code 8115) for each new SAN Volume Controller 2145-CG8 or SAN Volume Controller 2145-CF8 node.
  • You backed up the system configuration and saved the svc.config.backup.xml file.
  • The replacement node must be able to operate at the Fibre Channel or Ethernet connection speed of the node it is replacing.
  • If the node that is being replaced contains flash drives, transfer all flash drives and SAS adapters to the new node if it supports the drives. To prevent losing access to the data, if the new node does not support the existing flash drives, transfer the data from the flash drives before you replace the node.
Important:
  1. Before you perform this task, review all of the steps that follow.
  2. Do not perform this task if you are not familiar with SAN Volume Controller environments or the procedures that are described in this task.
  3. If you plan to reuse the node that you are replacing, ensure that the WWNN of the node is set to a unique number on your SAN. If you do not ensure that the WWNN is unique, the WWNN and WWPN are duplicated in the SAN environment and can cause problems.
    Tip: You can change the WWNN of the node you are replacing to the factory default WWNN of the replacement node to ensure that the number is unique.
  4. The node ID and possibly the node name change during this task. After the system assigns the node ID, the ID cannot be changed. However, you can change the node name after this task is complete.

Procedure

  1. Confirm that no hosts have dependencies on the node.

    When you shut down a node that is part of a system or you delete the node from a system, you can use either the management GUI or a command-line interface (CLI) command. In the management GUI, select Monitoring > System > Manage. Click Show Dependent Volumes to display all the volumes that depend on a node. You can also use the node parameter with the lsdependentvdisks CLI command to view dependent volumes.

    If dependent volumes exist, determine whether the volumes are being used. If the volumes are being used, either restore the redundant configuration or suspend the host application. If a dependent quorum disk is reported, repair the access to the quorum disk or modify the quorum disk configuration.

  2. Use these steps to determine the system configuration node, and the ID, name, I/O group ID, and I/O group name for the node that you want to replace. If you already know the physical location of the node that you want to replace, you can skip this step and proceed to step 3.
    Tip: If one of the nodes that you want to replace is the system configuration node, replace it last.
    1. Issue the following command from the command-line interface (CLI): 
      lsnode -delim :
    2. In the config_node column, find the value yes. Record the values in the id and name columns for each node in the system.
    3. Record the values in the IO_group_id and the IO_group_name columns for each node in the system.
    4. Issue the following command for each node in the system to determine the front panel ID, where node_name or node_id is the name or ID of the node. 
      lsnodevpd node_name or node_id
    5. Record the value in the front_panel_id column. The front panel ID is displayed on the front of each node. You can use this ID to determine the physical location of the node that matches the node ID or node name that you want replace.
  3. Perform these steps to record the WWNN or iSCSI name of the node that you want to replace:
    1. Issue this command from the CLI, where node_name or node_id is the name or ID of the node for which you want to determine the WWNN or iSCSI name. 
      lsnode -delim : node_name or node_id
    2. Record the WWNN or iSCSI name of the node that you want to replace. Also, record the order of the Fibre Channel and Ethernet ports.
  4. Issue this command to power off the node. 
    stopsystem -node node_name
    Important:
    1. Record and mark the order of the Fibre Channel or Ethernet cables with the node port number (port 1 - 4 for Fibre Channel, port 1 - 2 for Ethernet) before you remove the cables from the back of the node. The Fibre Channel ports on the back of the node are numbered 1 - 4 from left to right. You must reconnect the cables in the exact order on the replacement node to avoid issues when the replacement node is added to the system. If the cables are not connected in the same order, the port IDs can change, which impacts the ability of the host to access volumes. See the hardware documentation specific to your model to determine how the ports are numbered.
    2. Do not connect the replacement node to different ports on the switch or director. The SAN Volume Controller can have 4 Gbps or 8 Gbps adapters. However, do not move them to a faster switch or director ports now to avoid issues when the replacement node is added to the system. This task is separate and must be planned independently of replacing nodes in a system.
  5. Issue this CLI command to delete this node from the system and I/O group, where node_name or node_id is the name or ID of the node that you want to delete. 
    rmnode node_name or node_id
  6. Issue this CLI command to verify that the node is no longer a member of the system: 
    lsnode
    A list of nodes is displayed. Wait until the removed node is not listed in the command output.
  7. Change the WWNN or iSCSI name of the node that you deleted from the system to 1FFFF:
    1. Power on the node. With the Cluster panel displayed, press the up or down button until the Actions option is displayed.
    2. Press and release the select button.
    3. Press the up or down button until Change WWNN? is displayed.
    4. Press and release the select button to display the current WWNN.
    5. Press and release the select button to switch into edit mode. The Edit WWNN? panel is displayed.
    6. Change the WWNN to 1FFFF.
    7. Press and release the select button to exit edit mode.
    8. Press the right button to confirm your selection. The Confirm WWNN? panel is displayed.
    9. Press and release the select button to confirm.
  8. Install the replacement node and the uninterruptible power supply in the rack and connect the uninterruptible power supply cables.See the IBM SAN Volume Controller Model 2145-XXX Hardware Installation Guide to determine how to connect the node and the uninterruptible power supply.
    Important: Do not connect the Fibre Channel or Ethernet cables during this step.
  9. If you are removing flash drives from an old node and inserting them into a new node, see the IBM SAN Volume Controller Hardware Maintenance Guide for specific instructions.
  10. Power on the replacement node.
  11. Record the WWNN of the replacement node. If you plan to reuse the node that you are replacing, you can use this name.
  12. Change the WWNN name of the replacement node to match the name that you recorded in step 3:
    1. With the Cluster panel displayed, press the up or down button until the Actions option is displayed.
    2. Press and release the select button.
    3. Press the up or down button until Change WWNN? is displayed.
    4. Press and release the select button to display the current WWNN.
    5. Press the select button to switch into edit mode. The Edit WWNN? panel is displayed.
    6. Change the WWNN to the numbers that you recorded in step 3.
    7. Press and release the select button to exit edit mode.
    8. Press the right button to confirm your selection. The Confirm WWNN? panel is displayed.
    9. Press the select button to confirm.

    Wait one minute. If Cluster: is displayed on the front panel, the node is ready to be added to the system. If Cluster: is not displayed, see the troubleshooting information to determine how to address this problem or contact the IBM Support Center before you continue with the next step.

  13. Connect the Fibre Channel or Ethernet cables to the same port numbers that you recorded for the original node in step 4.
  14. Issue this CLI command to verify that the last 5 characters of the WWNN are correct: 
    lsnodecandidate
    Important: If the WWNN is not what you recorded in step 3, you must repeat step 12.
  15. Issue this CLI command to add the node to the system. Ensure that the new node has the same name as the original node and is in the same I/O group as the original node. See the addnode CLI command documentation for information. 
    addnode -wwnodename WWNN -iogrp iogroupname/id

    In the command, WWNN and iogroupname/id are the values that you recorded for the original node.

    SAN Volume Controller automatically reassigns the node with the name that was used originally. For versions before V5.1, use the name parameter with the svctask addnode command to assign a name. If the original name of the node name was automatically assigned by SAN Volume Controller, it is not possible to reuse the same name. It was automatically assigned if its name starts with node. In this case, either specify a different name that does not start with node or do not use the name parameter so that SAN Volume Controller automatically assigns a new name to the node.

    If necessary, the new node is updated to the same SAN Volume Controller software version as the system. The update can take up to 20 minutes.

    Important:
    1. Both nodes in the I/O group cache data but the cache sizes are asymmetric. The replacement node is limited by the cache size of the partner node in the I/O group. Therefore, it is possible that the replacement node does not use the full cache size until you replace the other node in the I/O group.
    2. You do not have to reconfigure the host multipathing device drivers because the replacement node uses the same WWNN and WWPN as the previous node. The multipathing device drivers can detect the recovery of paths that are available to the replacement node.
    3. The host multipathing device drivers take approximately 30 minutes to recover the paths. Do not update the other node in the I/O group until for at least 30 minutes after you successfully update the first node in the I/O group. If you have other nodes in different I/O groups to update, you can do those updates while you wait.
  16. Query the paths to ensure that all paths are recovered. If you are using the IBM System Storage® Multipath Subsystem Device Driver (SDD), the command to query paths is datapath query device. Documentation that is provided with your multipathing device driver shows how to query paths.
  17. If you want to use the replaced node as a spare node, follow these steps:
    1. With the Cluster panel displayed, press the up or down button until the Actions option is displayed.
    2. Press and release the Select button.
    3. Press the up or down button until Change WWNN? is displayed.
    4. Press and release the Select button to display the current WWNN.
    5. Press and release the Select button to switch into edit mode. The Edit WWNN? panel is displayed
    6. Change the WWNN to 00000.
    7. Press and release the Select button to exit edit mode.
    8. Press the right button to confirm your selection. The Confirm WWNN? panel is displayed.
    9. Press and release the Select button to confirm.

    This node can now be used as a spare node.

  18. Repeat steps 3 to 17 for each node that you want to replace.
Parent topic: Replacing nodes nondisruptively