Configuring a system to use RDMA-capable Ethernet ports for node-to-node communications

The system supports node-to-node connections that use Ethernet protocols that support remote direct memory access (RDMA) technology, such as RDMA over Converged Ethernet (RoCE) or iWARP. To use these protocols, the system requires that an RDMA-capable adapter is installed on each node and dedicated RDMA-capable Ethernet ports are only configured for node-to-node communication.

RDMA technologies, like RoCE and iWARP, enable the RDMA-capable adapter to transfer data directly between nodes, bypassing CPU and caches, making transfers faster. RDMA technologies provide faster connection and processing time than traditional iSCSI connections.

Prerequisites

The following prerequisites are required for all RDMA-capable Ethernet ports that are used between nodes:

All installation of the node hardware is complete.
The 25-Gbps Ethernet adapter that supports RDMA technology is installed on each node. If you are using RDMA-technology for node-to-node communications, ensure that the RDMA-capable adapters use the same technology, such as RoCE or iWARP. These RDMA-capable adapters must be installed in the same slots across all the nodes of the system. These installation requirements ensure that port identifiers are same across all nodes in the system.
Ethernet cables between each node are connected correctly.
The protocol technology on the source and destination adapters is the same.
The local and remote IP addresses can be reached.
Each IP address for RDMA-capable Ethernet ports and their associated subnet masks are unique on each node.
Router must not be placed between nodes that use RDMA-capable Ethernet ports for node-to-node communication.
The negotiated speeds on the local and remote adapters are the same.
The local and remote port virtual LAN identifiers are the same. Use virtual LAN to create physical separation of networks for unrelated systems, wherever possible. All the ports that are used for node-to node communication must be assigned with one VLAN ID and ports that are used for host attachment must have a different VLAN ID. If you plan to use VLAN to create this separation, you must configure VLAN support on the all the Ethernet switches in your network before you define the RDMA-capable Ethernet ports on nodes in the system. On each switch in your network, set VLAN to Trunk mode and specify the VLAN ID for the RDMA-ports that will be in the same VLAN. In addition, if VLAN settings for a RDMA-capable Ethernet port needs to be updated, these settings cannot be updated independently of other configuration settings. Before you update VLAN settings on specific RDMA-capable Ethernet ports, you must unconfigure the port, make any necessary changes to the switch configuration, then reconfigure RDMA-capable Ethernet ports on each of the nodes in the system.
A minimum of two dedicated RDMA-capable Ethernet ports are required for node-to-node communications to ensure best performance and reliability. These ports must be configured for inter-node traffic only and must not be used for host attachment, virtualization of Ethernet-attached external storage, or IP replication traffic.
A maximum of four RDMA-capable Ethernet ports per node are allowed for node-to-node communications.

The following graphic depicts an example configuration of an system that uses RDMA-capable Ethernet ports for node-to-node communications . The configuration uses the prerequisites and recommendations for environment and network configuration. In this example, Ports 1 and 2 (purple) on all the nodes in the system are configured for node-to-node communications while Port 3 and 4 (red) are used for host attachment. Ports 1 and 2 on all nodes are in a different subnet from ports 3 and 4 on all nodes to separate node-to-node traffic from host traffic. All the ports are connected to two redundant 100-GB switches that route traffic appropriately.

Configuring a system with RDMA-capable Ethernet ports — Figure 1. Configuring a system with RDMA-supported connections

Using the service assistant and management GUIs

To create a system that uses RDMA-capable Ethernet ports, complete these steps:

Define all IP addresses for all RDMA-capable Ethernet ports on each node in the system

Use the service assistant GUI to define the port IP addresses on each node by completing these steps:

In a supported browser, enter the service IP address for one of the nodes in the system.
In the service assistant GUI, select the one of the nodes and select Change Node IP.
On the Change Node IP panel, select the port to define IP addresses and click Modify.
Enter the IP address, subnet mask, gateway, and VLAN ID for the selected RDMA-capable Ethernet port.
Note: The IP address and subnet mask for each port must be unique and cannot be used anywhere else on the system. However, the VLAN ID for RDMA-capable Ethernet ports on all nodes must be the same, or connections between the nodes that use these IP addresses will fail. When you configure RDMA-capable Ethernet ports, do not exceed the maximum of four ports for node-to-node communications.
Click Save.
Repeat these steps for each RDMA-capable Ethernet port for the selected node. Depending on the number of nodes in your system, up to 8 RDMA-capable Ethernet ports can be configured, with two ports per node. These ports must be dedicated to inter-node communications and cannot be used for host attachment, virtualization of Ethernet-attached external storage, or IP replication.
After all IP addresses for the first node are configured, select the other node from the drop-down and select Go.
Repeat steps 2 through 6 for the other nodes in the system. After the second node is updated, you can add it to the system with the management GUI.

Add the nodes to the system

Using the management GUI, add the node to the system by completing these steps:

Select Monitoring > System.
Select Add Node from the System Actions menu.
After the node is added to the system, verify that the status of the node is online on the System - Overview page. If the node is listed as offline or if errors occur, select Monitoring > Events to view error messages or select Settings > Network > Ethernet Connectivity to view port connectivity for the node to view potential connection problems. You can also use the ping command to troubleshoot connectivity issues between the nodes. This page displays connection statuses between nodes. If a connection issue is present, error data information is provided to help determine what is causing the problem:

Discovered

Indicates that an Ethernet port on the selected node is configured, but a connection cannot be established. This status indicates a potential problem that needs to be resolved. The Error Data column indicates the reason for the Discovered status. The following values are possible in the Error Data column:

Protocol mismatch

Indicates that the protocol on the source and destination adapters is not the same. This error occurs when one node in the system does not have the 25 Gbps Ethernet adapter installed.

Unreachable

Indicates that the local and remote IP addresses cannot be reached. This error can occur if one of the nodes in the system is offline. Select Monitoring > Events to view errors and run any necessary fix procedures to return the node to an online status.

Duplicate IP addresses

Indicates that one or more IP addresses are being used in the network. Each node IP address must be unique. To fix this error, you can use the Service Assistant interface to change the node IP address.

Degraded

Indicates that the negotiated speed on both the local and remote adapters is not the same. Degraded status occurs when one or both adapters are configured at lower speed rather than the maximum speed that the adapters support. To fix this issue, ensure that adapters on both nodes are configured at the maximum speed.

VLAN ID Mismatch

Indicates that the local and remote port virtual LAN identifiers are not the same. To fix this error, ensure that the local and remote nodes belong to the same VLAN. If you are using VLAN in your network, you must configure VLAN on switches by setting VLAN to "Trunk" mode and specifying the VLAN ID on the switch before you configure IP addresses and other settings for the RDMA-capable Ethernet ports on the nodes in the system.

Using the command-line interface

To create a system that uses RDMA-capable Ethernet ports,, complete these steps:

To define port IP addresses for RDMA-capable Ethernet ports, enter the following command on each port for each node within the system:
```
satask chnodeip  -ip ip_address -mask mask -gw gateway -port_id port_number -vlan vlanid panel_name
```
where ip_address is the IP address for the port that is identified by its port identifier (port_number) and node name (panel_name). Ensure that all ports are in the same VLAN by specifying the same vlanid for each IP address on the RDMA-capable Ethernet port.
Note: The IP address and subnet mask for each port must be unique and cannot be used anywhere else on the system. However, the VLAN ID for RDMA-capable Ethernet port, on all nodes must be the same, or connections between the nodes that use these IP addresses will fail.
To add the node to the system, complete these steps:
1. Ensure that the node is listed as a candidate node by entering the following command:
```
svcinfo lsnodecandidate
```
  In the results that display, verify that the id parameter displays the WWNN for the node. If the node is not detected, verify cabling to the node.
2. Enter the following command to determine the I/O group where the node must be added:
```
lsiogrp
```
3. Record the name or ID of the first I/O group that has a node count of zero. You need the name or ID for the next step.
  Note: You must do this step for the first node that is added. You do not do this step for the second node of the pair because it uses the same I/O group number.
4. Enter this command to add the node to the system:
```
addnode -wwnodename WWNN -iogrp iogrp_name -name new_name_arg
```
  Where WWNN is the WWNN of the node, iogrp_name is the name of the I/O group that you want to add the node to, and new_name_arg is the name that you want to assign to the node. If you do not specify a new node name, a default name is assigned.
Verify connectivity between the nodes that use RDMA-capable Ethernet ports, by entering this command:
```
sainfo lsnodeipconnectivity
```
In the results that display, a value of Status:Connected indicates a successful connection. A value of Status:Discovered indicates that port on the selected node is configured, but a connection cannot be established. If connection errors occur, possible reasons for these errors are displayed in the error_data parameter. For descriptions of these error states, see the management GUI instructions. You can also use the ping command to troubleshoot connectivity issues between the nodes.