Configuration details for using RDMA-capable Ethernet ports for node-to-node communications

Before you configure a system to use RDMA-capable Ethernet ports for node-to-node communications, consider the following recommendations.

Important: Before you configure a system that uses RDMA-capable Ethernet ports for node-to-node communications and includes IBM® HyperSwap® configurations, contact your product support representative.
When you configure RDMA-capable Ethernet ports for node-to-node communication, use the following recommendations:
  • Assign different subnets for ports that are used for node-to-node communication and ports that are used for host attachment. For node-to-node communications with RDMA-capable Ethernet ports, physical port identifiers of all the nodes in the system must match and must be in the same subnet.
  • Use dual, redundant switch configuration to avoid single point of failure in Ethernet connectivity.
  • Verify RDMA support on all switches before you configure RDMA-capable Ethernet ports on nodes in the system.
  • Ensure that round-trip time (RTT) is up to 3 milliseconds without any packet drops.
  • To make the lossless Ethernet fabric, always use Ethernet flow control mechanism (IEEE 802.3x) for standard topology.
  • Use virtual LAN to create physical separation of networks for unrelated systems, wherever possible. All the ports that are used for node-to node communication must be assigned with one VLAN ID and ports that are used for host attachment must have a different VLAN ID. If you plan to use VLAN to create this separation, you must configure VLAN support on the all the Ethernet switches in your network before you define the RDMA-capable Ethernet ports on nodes in the system. On each switch in your network, set VLAN to Trunk mode and specify the VLAN ID for the RDMA-ports that will be in the same VLAN. In addition, if VLAN settings for a RDMA-capable Ethernet port needs to be updated, these settings cannot be updated independently of other configuration settings. Before you update VLAN settings on specific RDMA-capable Ethernet ports, you must unconfigure the port, make any necessary changes to the switch configuration, then reconfigure RDMA-capable Ethernet ports on each of the nodes in the system.
  • All systems that are not part of the current system that is being configured must be physically separated from the Ethernet fabric. You can configure different VLANs to achieve separation among systems.
  • Verify the configuration on IP addresses on your network to ensure that there are no duplicate IP addresses assigned.
  • If you use firewalls in your network configuration, ensure that traffic is open for TCP port 21455 and UDP ports 4791, 21451, and 21452. Node-to-node communications with RDMA-capable Ethernet connections use TCP port 21455 for data traffic and UDP ports 21451 and 21452 for service discovery on the system. If you are using the RoCE protocol, ensure that traffic is also open for UDP ports. Additionally, RDMA-capable Ethernet ports use Internet Group Management Protocol (IGMP) for group multicast communication for service discovery, so ensure that IGMP traffic is enabled on the firewall for redundant site configurations.
  • Assign an IP address with a different subnet for each individual RDMA-capable Ethernet port on each node so that the system can select a unique route depending on the destination IP address that it is connecting to.
  • RDMA-capable Ethernet ports can be used for both node-to-node communications and host attachment; however, do not share RDMA-capable Ethernet ports for hosts and node-to-node communications. RDMA-capable ports are not supported for connections to external storage. A variety of other protocols are also supported for host attachment and virtualization of external storage.
  • For HyperSwap configurations, ensure that the number of interswitch links is sufficient for RDMA-capable connections between nodes. Use the peak traffic that can flow between sites in failure scenarios, such as a single-site failure, Metro Mirror connections, or volume mirroring between sites to determine the number of inter-switch links that are required for the system.
For systems that use RDMA-capable Ethernet ports for node-to-node communication, the system has the following limitations:
  • Only IPv4 addresses are supported on RDMA-capable Ethernet ports.
  • Only the default value of 1500 is supported for maximum transmission unit (MTU) for node-to-node communications that use RDMA-capable Ethernet ports.
  • Port masking is not supported on RDMA-capable Ethernet ports. Due to this limitation, do not exceed the maximum of four ports for node-to-node communications.
  • Hot-spare nodes are not supported on systems that use RDMA-capable Ethernet ports for node-to-node communications.
  • Node-to-node communications that use RDMA-capable Ethernet ports are not supported in a network configuration that contains more than 2 hops in the fabric of switches.
  • Some environments might not include a stretched layer 2 subnet. In such scenarios, a layer 3 network such as in standard topologies or long-distance RDMA node-to-node HyperSwap configurations is applicable. To support the layer 3 Ethernet network, the unicast discovery method can be employed for RDMA node-to-node communication. This method relies on unicast-based fabric discovery rather than multicast discovery. To configure unicast discovery, see the addnodediscoverysubnet, rmnodediscoverysubnet, or lsnodediscoverysubnet commands.
  • Only certain models of the system support RDMA-capable Ethernet ports for node-to-node communications. For a list of the system hardware models that support RDMA-capable Ethernet ports, see the following website:

    http://www-03.ibm.com/systems/support/storage/ssic/interoperability.wss

    For other information about using RDMA-capable Ethernet ports on the system, go to the following website and search for Configuration Limits and Restrictions:

    www.ibm.com/support