Fibre Channel SAN configuration details

All back-end storage systems must always be connected to SAN switches only. Multiple connections are permitted from redundant storage systems to improve data bandwidth performance. A connection between each redundant storage system and each counterpart SAN is not required. For example, in an IBM® DS4000® configuration that contains two redundant storage systems, only two storage system minihubs are usually used. Storage system A is connected to counterpart SAN A, and storage system B is connected to counterpart SAN B. Any configuration that uses a direct physical connection between the node and the storage system is not supported.

When you attach a node to a SAN fabric that contains core directors and edge switches, connect the node ports to the core directors. Then, connect the host ports to the edge switches. In this type of fabric, the next priority for connection to the core directors is the storage systems, leaving the host ports connected to the edge switches.

A SAN must follow all switch manufacturer configuration rules, which might place restrictions on the configuration. Any configuration that does not follow switch manufacturer configuration rules is not supported.

Mixing manufacturer switches in a single SAN fabric

Within an individual SAN fabric, only mix switches from different vendors if the configuration is supported by the switch vendors. When you use this option for FCF Switch to FC Switch connectivity, review and plan as documented in ISL oversubscription.

Fibre Channel switches and interswitch links

The system supports distance-extender technology, including dense wavelength division multiplexing (DWDM) and Fibre Channel over IP (FCIP) extenders, to increase the overall distance between local and remote systems. If this extender technology involves a protocol conversion, the local and remote fabrics are regarded as independent fabrics, limited to three ISL hops each.

With ISLs between nodes in the same system, the inter-switch links (ISLs) are considered a single point of failure. Figure 1 illustrates this example.

  • If Link 1 or Link 2 fails, the system communication does not fail.
  • If Link 3 or Link 4 fails, the system communication does not fail.
  • If ISL 1 or ISL 2 fails, the communication between Node A and Node B fails for some amount of time. The node is not recognized, even though the nodes still have a connection.
Figure 1. Fabric with ISL between nodes in a system
This figure depicts fabric with inter-switch links between nodes in a system.

To ensure that a Fibre Channel link failure does not cause nodes to fail when ISLs are between nodes, you must use a redundant configuration. This configuration is illustrated in Figure 2. With a redundant configuration, if any one of the links fails, communication on the system does not fail.

Figure 2. Fabric with ISL in a redundant configuration
This figure depicts fabric with Inter-Switch Links in a redundant configuration.

ISL oversubscription

Complete a thorough SAN design analysis to avoid ISL congestion. Do not configure the SAN to use system to system traffic or a system to storage system traffic across ISLs that are oversubscribed. For host to system traffic, do not use an ISL oversubscription ratio that is greater than 7 to 1. Congestion on the ISLs can result in severe performance degradation and I/O errors on the host.

When you calculate oversubscription, you must account for the speed of the links. For example, if the ISLs run at 4 Gbps and the host runs at 2 Gbps, calculate the port oversubscription as 7 * (4/2). In this example, the oversubscription can be 14 ports for every ISL port.
Note: The port speed is not used in the oversubscription calculation.

The system in a SAN with director class switches

You can use director class switches within the SAN to connect large numbers of RAID controllers and hosts to a system. Because director class switches provide internal redundancy, one director class switch can replace a SAN that uses multiple switches. However, the director class switch provides only network redundancy; it does not protect against physical damage (for example, flood or fire), which might destroy the entire function. A tiered network of smaller switches or a core-edge topology with multiple switches in the core can provide comprehensive redundancy. This configuration provides more protection against physical damage for a network in a wide area. Do not use a single director class switch to provide more than one counterpart SAN because this configuration does not constitute true redundancy.