Planning for 25 Gbps Ethernet adapters

The system supports 25 Gbps Ethernet adapters that provide both iSCSI and iSCSI Extensions over RDMA (iSER) connections. Review prerequisite system requirements and installation considerations before deployment.

It is helpful to be familiar with the following terms when you are considering the features of the 25 Gbps Ethernet adapter for these connections.
Remote Direct Memory Access (RDMA)
RDMA is an networking standard that allows the adapters to transfer data directly to or from the endpoints in a connection without using CPU resources on either of the endpoints. These transfers occur simultaneously with other system operations and do not impact overall system performance. When the system performs an I/O operation over an RDMA-based connection, data is sent directly to the network, which reduces latency and increases the speed of data transfers.
iSCSI Extensions for RDMA (iSER)
iSER is a network protocol that extends the iSCSI protocol to use RDMA. You can implement RDMA-based connections that use existing Ethernet networking structures and connections without upgrading current hardware. Currently the system supports RDMA-based connections with RDMA over Converged Ethernet (RoCE) or Internet Wide Area RDMA Protocol (iWARP). If you are undecided whether to use RoCE or iWARP in your data center, search the internet for iSCSI Extensions for RDMA to obtain more information.
Review the following elements when you are deciding to install the optional 25 Gbps adapter:
  • The minimum software level that is needed to support the 25 Gbps Ethernet adapter is 8.1.1.1 for host attachment and external storage system connections. For node-to-node connections, the minimum software level of 8.2.1 or later is required.
  • Two types of 25 Gbps Ethernet adapters are available:
    • 2-port 25 Gbps Ethernet (RoCE) adapter
    • 2-port 25 Gbps Ethernet (iWARP) adapter

    These types of adapters can be used for connections to hosts, external storage systems, or between nodes to create a system. For host attachment, these 25 Gbps adapters support iSCSI and RDMA-based connections. For external storage systems, only iSCSI connections are supported through these adapters. When the 25 Gbps adapter is installed on nodes in the system, RDMA technology can be used for node-to-node communications.

  • Depending on your system model, the number of Ethernet ports that are supported per node or node canister varies. Verify your system model and the support before installing or using 25 Gbps Ethernet adapters. To find the maximum number of Ethernet ports that can be used by the software on this node, search for Configuration Limits and Restrictions and your product name at the following website:

    http://www.ibm.com/support/

    Three or four onboard ports are available depending on the model, plus up to eight Ethernet ports on PCIe adapters.

  • If you are using RDMA-technology for node-to-node communications, ensure that the RDMA-capable adapters use the same technology, such as RoCE or iWARP. These RDMA-capable adapters must be installed in the same slots across all the nodes of the system. These installation requirements ensure that port identifiers are same across all nodes in the system.
  • 10 Gbps and 25 Gbps adapters are supported on the same platform.
  • Most host operating systems that support 10 Gbps Ethernet also support 25 Gbps Ethernet. To verify host operating system support, search for Configuration Limits and Restrictions and your product name at the following website:

    http://www.ibm.com/support/

  • The 25 Gbps adapter has better latency, port consolidation, and approximately 25-40% better bandwidth usage when compared to the 10 Gbps adapter.
  • IP Replication is supported with 25 Gbps Ethernet adapters.
    Note: There is no performance benefit or advantage for IP Replication with these adapters. However, for the purpose of consolidation where these cards are used for other purposes like iSER Host Attach or iSCSI Host Attach/Backend Virtualization, they can be used for IP replication.
  • The correct 25 Gbps Ethernet adapter feature code must be used, depending on whether the SAN implements iWARP or RoCE, as the protocols do not inter-operate.
    • iWARP adapters can communicate RDMA only to iWARP adapters.
    • RoCE adapters can communicate RDMA only to RoCE adapters.
  • An iSER connection from a RoCE host or storage system Ethernet port works only to a RoCE Ethernet port on a node canister.
  • An iSER connection from an iWARP host or storage system Ethernet port works only to an iWARP Ethernet port on a node canister.
  • RoCE is supported by the InfiniBand® Trade Association (IBTA) and the RoCE Alliance.
  • iWARP is supported by Chelsio and Intel.