Replacing nodes nondisruptively: 2145-SV1

The following procedures describe how to nondisruptively replace most nodes with SAN Volume Controller 2145-SV1 nodes.

1. Confirm that the node you are replacing is running software level 7.7.1. If the node is not running SAN Volume Controller system software level 7.7.1, the system software must be upgraded before you continue this procedure. You can use the technician port and the service assistant GUI to view and update the software level. For information, see Updating the system.

Collect important information about the node you are replacing

2. Determine the ID, name, I/O group ID, I/O group name, and system configuration node status for the node that you want to replace.
   1. Issue the lsnode command from the command-line interface.

      svcinfo lsnode -delim : 

      The system displays information about the nodes that are currently defined in the system.
   2. Record the information from the lsnode command output in Table 1. This information identifies the node, the I/O group in which it belongs, and iSCSI information.
      Tip: If one of the nodes that you want to replace is the system configuration node (config_node:yes), replace it last.

      Table 1. Configuration information about the nodes to be replaced

      lsnode command output							lsnodevpd command output
      id	name	WWNN	IO_group_id	IO_group_name	config_node	iscsi_name	front_panel_id

   3. Find the front panel ID of the node you want to replace. You can use this ID to determine the physical location of the node. Issue the lsnodevpd command, where node_name or node_id is the name or ID of the node. (If you already know the physical location of the node that you want to replace, you can go to the next step.)

      lsnodevpd node_name or node_id

      The system displays detailed information about the node.
   4. Record the value in the front_panel_id column in Table 1.
3. Confirm that no hosts have dependencies on the node that you are replacing. You can use either the management GUI or issue a command.
   • If you use the management GUI, select Monitoring > System. Right-click the node and select Show Dependent Volumes to display all the volumes that depend on a node.
   • If you use the lsdependentvdisks command to view dependent volumes, specify the node parameter, where node_name or node_id is the name or ID of the node.

     lsdependentvdisks -node node_id_or_name

   1. If dependent volumes exist, determine whether the volumes are being used. If the volumes are being used, either restore the redundant configuration or suspend the host application.
   2. If a dependent quorum disk is reported, repair the access to the quorum disk or modify the quorum disk configuration.
4. Issue the lsservicestatus command to display information about the FC ports of the node to be replaced.

   sainfo lsservicestatus

5. Record the fc_io_port_id and fc_io_port_WWPN for each port in Table 2. This information is required to check the port mapping when you add the new node.

   Table 2. Information about the FC ports of the node to be replaced

   lsservicestatus command output
   fc_io_port_id	fc_io_port_WWPN

6. If Ethernet port IP addresses are configured on the system, enter the lsportip command to display the current settings so that they can be applied to the replacement nodes.

   lsportip -delim : 

   The system displays information about the Ethernet ports that are defined on the specified node.
7. Record the information about the Ethernet ports on the node that you want to replace in Table 3.

   Table 3. Information about the Ethernet ports of the node to be replaced

   lsportip command output
   node_id	node_name	IP_address	subnet_mask	IP_address_6	prefix	gateway_port_id

Remove the node from the system

8. Record and mark the order of the Fibre Channel or Ethernet cables with the node port number before you remove the cables from the back of the node. You must reconnect the cables in the exact order on the replacement node to avoid issues when the replacement node is added to the system. If the cables are not connected in the same order, the port IDs can change. If the port IDs change, the host system might not be able to access volumes. See the hardware documentation specific to your model to determine how the ports are numbered.
9. Do not connect the replacement node to different ports on the switch or to a different switch.
10. If the node has 10 Gbps Ethernet IP addresses configured, delete these settings by using the rmportip command, ensuring that you note the current settings.

    rmportip -node node_ID_or_name port_ID

11. Issue the rmnode command to delete this node from the system and I/O group. The node_ID_or_name value identifies the node that you want to delete.

    rmnode node_ID_or_name

12. Enter the lsnode command to ensure that the node is no longer a member of the system:

    lsnode

    The system displays a list of nodes. Before you continue to the next step, ensure that the removed node is not listed in the command output.
13. Optional: If you want to use the removed node as a spare node, change the WWNN and iSCSI name of each node that you deleted to 1FFFF.
    • For a SAN Volume Controller 2145-CG8 or SAN Volume Controller 2145-CF8 node.
      1. Power on the node. With the Cluster panel displayed, press the up or down button until the Actions option is displayed.
      2. Press and release the select button.
      3. Press the up or down button until Change WWNN? is displayed.
      4. Press and release the select to display the current WWNN.
      5. Press and release the select to switch into edit mode. The Edit WWNN? panel is displayed.
      6. Change the WWNN to 1FFFF.
      7. Press and release the select to exit edit mode.
      8. Press the right button to confirm your selection. The Confirm WWNN? panel is displayed.
      9. Press and release the select to confirm.
    • For a 2145-DH8 or 2145-SV1 node.
      1. Power on the node.
      2. Issue the following chvpd command.

         satask chvpd -wwnn FFFFFFFFFFFFFFFF

Prepare the replacement 2145-SV1 node

14. Install the replacement node in the rack. See Installing the SAN Volume Controller 2145-SV1 hardware for more information.
    Important: Do not connect the Fibre Channel or Ethernet cables during this step.
15. Power on the replacement node.
16. Enter the following command to find the WWNN of the replacement 2145-SV1 node. This name can be reused by another 2145-SV1 node.

    sainfo lsservicestatus

17. Assign a hardware location in the new 2145-SV1 node for each FC port that is defined on the node you are replacing. To do so, enter the appropriate chvpd command for the port mapping information you recorded in Setting the Fibre Channel port mapping: 2145-SV1.

    satask chvpd -wwnn wwnn -fcportmap AB-CD,AB-CD,AB-CD,AB-CD 

    Note: You must create the port mapping before you can add the new node to the system.
    When the command completes, the system creates the new port mappings on the replacement 2145-SV1 node. The node will then reboot to apply the new settings.
18. Enter the lsnodecandidate command to verify that the last 5 characters of the WWNN are correct.

    satask lsnodecandidate

19. Enter the lsservicestatus command to verify that the fc_io_port_id and fc_io_port_WWPN on the 2145-SV1 node match the values that are recorded from the lsservicestatus output from the original node.

    sainfo lsservicestatus

    1. If there are differences, review Setting the Fibre Channel port mapping: 2145-SV1 and correct the mapping, as needed.
    2. If the values match, connect the Fibre Channel or Ethernet cables to the host adapters.
20. Add the new 2145-SV1 replacement node to the system. You can use the service assistant interface or enter the addnode command, where WWNN and iogroup_name_or_id are the values that you recorded for the original node.

    addnode -wwnodename WWNN -iogrp iogroup_name_or_id

    Ensure that the new node has the same name as the original node and is in the same I/O group as the original node. Refer to the data that you recorded in Table 1 in Step 2.b.

    The system reassigns the 2145-SV1 node with the name that was used originally for the node that was replaced. If the original name of the node was automatically assigned by SAN Volume Controller, it is not possible to reuse that name. It was automatically assigned if its name starts with node. In this case, either specify a different name that does not start with node or do not use the name parameter so that SAN Volume Controller automatically assigns a new name to the node.

    Important: Ensure that all other nodes in the cluster are running system software level 7.7.1 or later. Otherwise, the replacement 2145-SV1 node will not be recognized. For information, see Updating the system.
21. If Ethernet IP addresses were previously configured on the replaced node, configure the Ethernet ports on the new node to reuse those settings. Ethernet port IP addresses can be configured by using the management GUI or the cfgportip command. Specify the appropriate values that you noted in Table 3 in Step 7.
    • For IPv4 IP addresses

      cfgportip -node node_name_or_ID -ip IPv4_addr
      -mask subnet_mask -gw gateway port ID

    • For IPv6 IP addresses

      cfgportip -node node_name_or_ID -ip_6 IPv6_addr
      -prefix_6 prefix -gw_6 gateway port ID

    Important:

    1. Both nodes in the I/O group cache data; however, the cache sizes are asymmetric. The replacement node is limited by the cache size of the partner node in the I/O group. Therefore, it is possible that the replacement node does not use the full cache size until you replace the other node in the I/O group.
    2. You do not need to reconfigure the host multipathing device drivers because the replacement node uses the same WWNN and WWPN as the previous node. The multipathing device drivers detect the recovery of paths that are available to the replacement node.
    3. The host multipathing device drivers take approximately 30 minutes to recover the paths. Do not update the other node in the I/O group for at least 30 minutes after you successfully update the first node in the I/O group. If you have other nodes in different I/O groups to update, you can do those updates while you wait.
    4. If you were not able to check that the Fibre Channel device driver of every host is set to time out a Fibre Channel path in 3 seconds or less, reboot the new 2145-SV1 node now to guarantee that the fiber path becomes active when the node becomes active again.
22. Important Ask the host administrator to query the paths on each host to ensure that all paths to the replacement node are active before you proceed to the next step. If you are using the IBM System Storage® Multipath Subsystem Device Driver (SDD), issue the datapath query device command to query the paths. Documentation that is provided with your multipathing device driver shows how to query paths. Force the multipath driver to rescan for paths if the expected paths are not active.
23. Create the appropriate storage arrays (MDisks) with the flash drives in the expansion enclosure to hold the data for all the volumes that are being migrated from the internal disks of this I/O group in the same storage pools (MDisk groups) that contain the MDisks for the internal storage in this I/O group. To do so, enter the mkarray or mkdistributedarray command.
24. To remove the MDisks for the internal drives, enter the following rmmdisk command.

    rmmdisk -mdisk mdisk_list -force mdisk_group_id_or_name 

25. Check the progress of the active migrations by entering lsmigrate command.

    svcinfo lsmigrate

    Note: The command processing might appear to complete asynchronously before the actual data migration is completed.
26. Check that the flash drives in the old node are not in member state by entering the lsdrive command.
27. Remove the original drives from the system configuration by changing their use to unused.
28. Repeat Step 2 through Step 22 for each node that you replace.