Managing nodes that use RDMA-capable Ethernet ports

The system supports node-to-node connections that use Ethernet protocols that support remote direct memory access (RDMA) technology, such as RDMA over Converged Ethernet (RoCE) or iWARP. To use these protocols, the system requires that an RDMA-capable adapter is installed on each node and dedicated RDMA-capable Ethernet ports are only configured for node-to-node communication.

RDMA technologies, like RoCE and iWARP, enable the RDMA-capable adapter to transfer data directly between nodes, bypassing CPU and caches, making transfers faster. RDMA technologies provide faster connection and processing time than traditional iSCSI connections.

The following prerequisites are required for all RDMA-capable Ethernet ports that are used between nodes:
  • All installation of the node hardware is complete.
  • The 25-Gbps Ethernet adapter that supports RDMA technology is installed on each node. If you are using RDMA-technology for node-to-node communications, ensure that the RDMA-capable adapters use the same technology, such as RoCE or iWARP. These RDMA-capable adapters must be installed in the same slots across all the nodes of the system. These installation requirements ensure that port identifiers are same across all nodes in the system.
  • Ethernet cables between each node are connected correctly.
  • The protocol technology on the source and destination adapters is the same.
  • The local and remote IP addresses can be reached.
  • Each IP address for RDMA-capable Ethernet ports and their associated subnet masks are unique on each node.
  • Router must not be placed between nodes that use RDMA-capable Ethernet ports for node-to-node communication.
  • The negotiated speeds on the local and remote adapters are the same.
  • The local and remote port virtual LAN identifiers are the same. Use virtual LAN to create physical separation of networks for unrelated systems, wherever possible. All the ports that are used for node-to node communication must be assigned with one VLAN ID and ports that are used for host attachment must have a different VLAN ID. If you plan to use VLAN to create this separation, you must configure VLAN support on the all the Ethernet switches in your network before you define the RDMA-capable Ethernet ports on nodes in the system. On each switch in your network, set VLAN to Trunk mode and specify the VLAN ID for the RDMA-ports that will be in the same VLAN. In addition, if VLAN settings for a RDMA-capable Ethernet port needs to be updated, these settings cannot be updated independently of other configuration settings. Before you update VLAN settings on specific RDMA-capable Ethernet ports, you must unconfigure the port, make any necessary changes to the switch configuration, then reconfigure RDMA-capable Ethernet ports on each of the nodes in the system.
  • A minimum of two dedicated RDMA-capable Ethernet ports are required for node-to-node communications to ensure best performance and reliability. These ports must be configured for inter-node traffic only and must not be used for host attachment, virtualization of Ethernet-attached external storage, or IP replication traffic.
  • A maximum of four RDMA-capable Ethernet ports per node are allowed for node-to-node communications.

The basic node tasks, such as adding an node or removing a node, are the same for both Fibre Channel and RDMA-based connections between nodes, but you may need to complete management actions on the RDMA-capable Ethernet ports prior to completing node-level management tasks.

Before completing managing tasks that are related to RDMA-capable Ethernet ports on a node, use the following best practices to manage these ports:
  • If you already have a system configured to use RDMA-capable Ethernet ports, you must ensure that one redundant path is available before adding, removing, or updating settings for RDMA-capable Ethernet ports.
  • Only add, remove, or update settings on one RDMA-capable Ethernet port at a time. Wait 15 seconds between these changes before updating other RDMA-capable Ethernet ports.
  • If you are using virtual LAN to create physical separation of networks, ensure that you follow these additional guidelines when completing managing related tasks:
    • VLAN IDs cannot be update or added independently of other settings on a RDMA-capable Ethernet port, such as IP address.
    • Before adding or updating VLAN ID information to RDMA-capable Ethernet ports, you must configure VLAN support on the all the Ethernet switches in your network. For example, on each switch, set VLAN to "Trunk" mode and specify the VLAN ID for the RDMA-capable Ethernet ports that will be in the same VLAN.