You can use more than four fabric ports per
node to improve
peak load I/O performance, but careful planning is needed.
A fabric port is a Fibre Channel port or a Fibre Channel
over Ethernet (FCoE) port. If you use more than four fabric ports
per node, you must either use the localfcportmask and partnerfcportmask commands
or be careful with your fabric zoning configuration.
If a node receives more than
16 logins from another node, then it causes node error 860.
You can ensure that no more than 16 logins are received by following these guidelines:
- Zone the SAN fabric to reduce the number of paths between the nodes.
- Apply a local Fibre Channel port mask (if the nodes are within the same system) or partner Fibre
Channel port mask (if the nodes are in different systems) to reduce the number of ports that are
being used for node-to-node communication.
- Provide a combination of the two (zoning and port masks).
To avoid receiving node error 860 and to maximize performance on your system, follow these
guidelines:
- For redundancy, use a minimum of two ports. If ports are being dedicated for different types of
traffic, assign a minimum of two ports for each type of traffic.
- Within a system, up to 50% of the overall data that is transferred across the SAN is transmitted
between the nodes. However, for read-intensive workloads, the figure is much less. Therefore, if
ports are being dedicated for different types of traffic, assign between 1/4 and 1/2 of the overall
ports for node-to-node communication within a system.
- For replication between systems, the connection between the systems is usually the bottleneck.
Other than for redundancy, there is no point in having more SAN connections than there is bandwidth.
For example, if two sites are connected with a 10 Gbps link, two 8 Gbps ports are sufficient.
Systems usually are configured with two ports per node for replication traffic primarily for
redundancy. For systems with larger numbers of nodes, it might be appropriate to have connections
only from a subset of the nodes for replication to a remote system. The system automatically
forwards replication traffic between local nodes so that all nodes can still participate in
replication.
In the examples that follow, consider this configuration that is shown in
Figure 1:
Figure 1. Fabric port configuration
- Four nodes, where nodes 1 and 2 are in a system on site A. Nodes 3 and 4 are in a system on site
B.
- Each site has 2 switches (switch 1 and 2 on site A; switch 3 and 4 on site B).
- Each node has 2 adapters (A and B).
- Each adapter has 4 ports where the port is named P nodeid
adapterid
portnumber. For example, port 3 on adapter B of node 4 is named P4B3.
Example 1: Zoning only
Each node in the system is configured in the following way:
- Four ports are used for host, controller, and local node connections.
- Four ports are used for Global Mirror and Metro Mirror.
The following zones are required for two sites, A and
B:
Zone sets for site A:
- Zone for storage controllers and local node connections (Fibre Channel adapter HBA)
- Zone 1 - [Node ports P1A1, P1A2, P2A1, P2A2 plus all storage controller ports in site A on switch 1]
- Zone 2 - [Node ports P1A3, P1A4, P2A3, P2A4 plus all storage controller ports in site A on switch
2]
- Zones for host attachment (Fibre Channel adapter HBA A)
- The same ports on the first adapter are zoned to be visible to hosts. It is recommended that
each host is in a separate zone, and a host with two ports that are connected to two fabrics have
two zones, one per fabric. The actual number of zones might be two multiplied by the number of
hosts. A host with four ports that are connected to two fabrics should have four zones. In this
case, the actual number of zones might be four multiplied by the number of hosts. In the four-port
host case, the zones for one host might be:
- Zone 3 - [host port, P1A1 and P2A1 for multipath fail over] on switch 1
- Zone 4 - [host port, P1A2 and P2A2 for multipath fail over] on switch 1
- Zone 5 - [host port, P1A3 and P2A3 for multipath fail over] on switch 2
- Zone 6 - [host port, P1A4 and P2A4 for multipath fail over] on switch 2
If the host only had two ports, it might have Zone 3 and Zone 5, or Zone 4 and Zone 6.
- Zones for Global Mirror and Metro Mirror (Fibre Channel adapter HBA B
only):
To ensure
Global Mirror and
Metro Mirror use the second Fibre
Channel adapter exclusively, it is required that each local or remote port pair has its own zone:
- Zone 7 - [P1B1,P3B1]
- Zone 8 - [P1B1,P3B2]
- Zone 9 - [P1B1,P4B1]
- Zone 10 - [P1B1,P4B2]
- Zone 11 - [P1B2,P3B1]
- Zone 12 - [P1B2,P3B2]
- Zone 13 - [P1B2,P4B1]
- Zone 14 - [P1B2,P4B2]
- Zone 15 - [P2B1,P3B1]
- Zone 16 - [P2B1,P3B2]
- Zone 17 - [P2B1,P4B1]
- Zone 18 - [P2B1,P4B2]
- Zone 18 - [P2B1,P4B2]
- Zone 19 - [P2B1,P4B2]
- Zone 20 - [P2B2,P3B2]
- Zone 21 - [P2B2,P4B1]
- Zone 22 - [P2B2,P4B2]
Zone sets for site B:
This zone set is essentially a mirror image of the zoning for site A.
Example 2: Port masking only
Assuming the same configuration as in example 1, you can achieve the same result by using port
masking rather than zoning. The positions in the mask represent the Fibre Channel I/O port IDs with
ID 1 in the rightmost position. These IDs are shown by the lsportfc command. In
this example ports A1, A2, A3, A4, B1, B2, B3, and B4 correspond to FC I/O port IDs 1, 2, 3, 4, 5,
6, 7 and 8.
For remote partnership communication, apply a mask that limits communication to ports B1, B2, B3,
and B4 by applying a port mask of 11110000 to both systems. Use the command chsystem
-partnerfcportmask 11110000.
For local node-to-node communication, apply a mask that limits communication to ports A1, A2, A3,
and A4 by applying a port mask of 00001111 to both systems. Use the command chsystem
-localfcportmask 00001111.
It is then possible to have a single zone
that contains all the node ports, as the port masking ensures that
the number of logins is no more than 16.
With this setup, zoning for host connectivity and storage connectivity does not need to consider
the rules about more than 16 logins.
