Replacing nodes nondisruptively

You can nondisruptively replace an existing node with a new IBM Spectrum Virtualize node.

Before you begin

The replacement procedures are nondisruptive because changes to your SAN environment are not required. The replacement IBM Spectrum Virtualize node uses the same worldwide node name (WWNN) as the node that you are replacing. An alternative to this procedure is to replace nodes disruptively either by moving volumes to a new I/O group or by rezoning the SAN. However, the disruptive procedures require more work on the hosts.

There might be some loss of system performance when the nodes are being replaced. For example, volumes that are managed by the I/O group containing the node to be replaced will become degraded when one of the nodes is shut down at the start of this procedure. The volumes will remain degraded until both IBM Spectrum Virtualize nodes are running.

This task assumes that the following conditions are met.

The existing system software on the nodes that are being replaced must be 7.7.1 or later.
All nodes that are configured in the system are present and online.
All errors in the system event log are addressed and marked as fixed.
No volumes, managed disks (MDisks), or external storage systems have a status of degraded or offline.
You backed up the system configuration and saved the svc.config.backup.xml file.
The replacement node must be able to operate at the Fibre Channel or Ethernet connection speed of the node that it is replacing.
The replacement node must have the same configuration of I/O adapters in the same slots as the old node.
The Fibre Channel device driver on each Fibre Channel attached host should be set to time out a missing fibre path in 3 seconds or less. If it is not practical to check the parameters of the Fibre Channel driver on each host, you will need to reboot the new node shortly after it is added to the system. By doing so, the fibre paths to the host stop long enough to ensure that they are recovered properly when the IBM Spectrum Virtualize node is active again.
Tip: The timeout setting for the Emulux Fibre Channel device driver might default to 30 seconds, so it needs to be changed.

Important Notes:

Do not continue this task if any of the conditions that are listed are not met unless you are instructed to do so by IBM® Remote Technical Support.
Review all of the steps that follow before you proceed with this task.
Do not continue this task if you are not familiar with the IBM Spectrum Virtualize environment or the procedures that are described in this task.
If you plan to reuse the node that you are replacing, ensure that the WWNN of the node is set to a unique number on your SAN. If you do not ensure that the WWNN is unique, the WWNN and WWPN are duplicated in the SAN environment and can cause problems.
The node ID and possibly the node name change during this task. After the system assigns the node ID, the ID cannot be changed. However, you can change the node name after this task is complete.

About this task

To replace active nodes in a system, complete the following steps.

Procedure

Optional: If the current software level on the node you are replacing is not the same as the software level on the active IBM Spectrum Virtualize system, you might want to install the current IBM Spectrum Virtualize system software level on to the node. By performing this step, you can save up to 20 minutes when a node is added to the IBM Spectrum Virtualize system at step 16.

Optionally, using the service assistant, you can also change the WWNN now to the value used by the node that you are replacing with this node.
Complete the following steps.
1. Confirm that no hosts have dependencies on the node.
  You can use either the management GUI or a command-line interface (CLI) command:
  1. In the management GUI, select Monitoring > System .
  2. On the System -- Overview page, use the directional arrow near the node Node Details page.
  3. Select Node Actions > Dependent Volumes
  - If you use the CLI command, use the node parameter with the lsdependentvdisks command to view dependent volumes.
    lsdependentvdisks -node node_id | node_name
2. If dependent volumes exist, determine whether the volumes are being used.
  If the volumes are being used, either restore the redundant configuration or suspend the host application.
3. If a dependent quorum disk is reported, repair the access to the quorum disk or modify the quorum disk configuration.
If you already know the physical location of the node that you want to replace, you can skip this step and proceed to the next step. Follow these steps to determine the system configuration node, and the ID, name, I/O group ID, and I/O group name for the node that you want to replace.

Tip: If one of the nodes that you want to replace is the system configuration node, replace it last.
1. Issue the lsnode command from the command-line interface (CLI).
```
lsnode -delim : 
```
2. In the config_node column, find the value yes and record the values in the id and name columns.
3. Record the values in the id and the name columns for each node in the system.
4. Record the values in the IO_group_id and the IO_group_name columns for each node in the system.
5. To determine the front panel ID, issue the lsnodevpd command, where node_name or node_id is the name or ID of the node. Issue the command for each node in the system.
```
lsnodevpd node_name or node_id
```
6. Record the value in the front_panel_id column.
  The front panel ID is displayed on the front of each node. You can use this ID to determine the physical location of the node that matches the node ID or node name that you want replace.
Identify and record the WWNN and iSCSI name of the node that you want to replace.
1. Issue the lsnode command, where node_name or node_id is the name or ID of the node for which you want to determine the WWNN and iSCSI name.
```
lsnode -delim : node_name_or_id
```
2. Record the WWNN and iSCSI name of the node that you want to replace.
3. Record the order of the Fibre Channel and Ethernet ports.
4. If the system has Ethernet port IPs configured, store the current settings so that they can be applied to the replacement nodes. To do so, enter the lsportip command.
```
lsportip -delim :
```
Required: Complete the following steps.
1. Record and mark the order of the Fibre Channel or Ethernet cables with the node port number before you remove the cables from the back of the node.
  You must reconnect the cables in the exact order on the replacement node to avoid issues when the replacement node is added to the system. If the cables are not connected in the same order, the port IDs can change, which impacts the ability of the host to access volumes. See the hardware documentation specific to your model to determine how the ports are numbered.
2. Do not connect the replacement node to different ports on the switch or to a different switch.
  If the Fibre Channel switches are going to be changed so that the 8 Gbps speed can be reached, then this task must be a separate task that is done before or after this node replacement procedure.
If the node has 10 Gbps Ethernet IPs configured, delete these settings by using the rmportip command, ensuring that you note the current settings.
```
rmportip -node node_ID_or_name port_ID
```
Issue the rmnode command to delete this node from the system and I/O group. The node_ID_or_name value identifies the node that you want to delete.
```
rmnode node_ID_or_name
```
You can use the CLI to verify that the deletion process was completed.
Optional: If you want to use the removed node as a spare node, enter the lsnodecommand to ensure that the node is no longer a member of the system.
```
lsnode 
```
A list of nodes is displayed. Wait until the removed node is not listed in the command output.
Change the WWNN and iSCSI name of the node that you deleted from the system to FFFFF.
1. Power on the node.
2. Issue the following chvpd command.
```
satask chvpd -wwnn FFFFFFFFFFFFFFFF
```
Install the replacement node in the rack.

Important: Do not connect the Fibre Channel or Ethernet cables during this step.
Power on the replacement node.
Record the WWNN of the replacement node. This name can be reused by another IBM Spectrum Virtualize node.
Change the WWNN name of the replacement node to match the name that you recorded in step 4.
To do so, you can use the service assistant interface to change the WWNN. You can also issue the following satask chvpd command, where WWNN is the value you recorded from the original node.
```
satask chvpd -wwnn WWNN 
```
Enter the lsnodecandidate command to verify that the last 5 characters of the WWNN are correct.
```
lsnodecandidate
```
Important: If the WWNN is not what you recorded in step 4, you must repeat step 13.
Connect the Fibre Channel or Ethernet cables to the same port numbers that you recorded for the original node in step 5.
Add the node to the system. You can use the service assistant interface or enter the following command, where WWNN and iogroupname_id are the values that you recorded for the original node. Ensure that the new node has the same name as the original node and is in the same I/O group as the original node. For more information, see the addnode command.
```
addnode -wwnodename WWNN -iogrp iogroupname_id
```
IBM Spectrum Virtualize reassigns the node with the name that was used originally. If the original name of the node name was automatically assigned by IBM Spectrum Virtualize, it is not possible to reuse the same name. It was automatically assigned if its name starts with node. In this case, either specify a different name that does not start with node or do not use the name parameter so that IBM Spectrum Virtualize automatically assigns a new name to the node.

If necessary, the new node is updated to the same IBM Spectrum Virtualize software version as the system. This update can take up to 20 minutes.

If Ethernet IPs were previously configured, configure the Ethernet ports to reuse the settings from the replaced node. Ethernet port IPs can be configured by using the management GUI or the cfgportip command.
- For IPv4 IP addresses
```
cfgportip -node node_name_or_ID -ip IPv4_addr
-mask subnet_mask -gw gateway port ID
```
- For IPv6 IP addresses
```
cfgportip -node node_name_or_ID -ip_6 IPv6_addr
-prefix_6 prefix -gw_6 gateway port ID 
```
Important:
1. The replacement node must have at least as many CPUs and at least as much RAM as the node being replaced.
2. Both nodes in the I/O group cache data, however, the cache sizes are asymmetric. The replacement node is limited by the cache size of the partner node in the I/O group. Therefore, it is possible that the replacement node does not use the full cache size until you replace the other node in the I/O group.
3. You do not need to reconfigure the host multipathing device drivers because the replacement node uses the same WWNN and WWPN as the previous node. The multipathing device drivers detect the recovery of paths that are available to the replacement node.
4. The host multipathing device drivers take approximately 30 minutes to recover the paths. Do not upgrade the other node in the I/O group for at least 30 minutes after you successfully upgrade the first node in the I/O group. If you have other nodes in different I/O groups to upgrade, you can do those upgrades while you wait.
5. If you were not able to check that the Fibre Channel device driver of every host is set to time out a Fibre Channel path in 3 seconds or less, then it is best to reboot the new IBM Spectrum Virtualize node now to guarantee that the fibre path becomes active when the node becomes active again.
Required: Important: Ask the host administrator to query the paths on each host to ensure that all paths to the replacement node are active before you proceed to the next step. If you are using the IBM Multipath Subsystem Device Driver (SDD), the command to query paths is datapath query device. Documentation that is provided with your multipathing device driver shows how to query paths. Force the multipath driver to rescan for paths if the expected paths are not active.
Optional: If you want to use the replaced node as a spare node, follow these steps.
1. Connect to the service assistant interface on the node by using the technician port.
2. Ensure that you are connected to the correct node and then select Configure Node.
3. Select Update WWNN.
4. Under Specify WWNN, enter 00000.
5. Click Modify to confirm.
This node can now be used as a spare node.
Repeat steps 4 to 18 for each node that you want to replace.