Cabling of Fabric MetroCluster
The cabling of a MetroCluster is the key. Outside of some licensing it’s the cabling that’s really the only different between MetroCluster and Mirrored HA pair. Yes it’s a bit more complex for failover and failback but really the main difference from a setup point of view is the cabling. There’s a large number of cables and the configuration should be all mapped out before beginning putting equipment into your racks. I would heartily recommend reading MetroCluster and High Availability Guide before starting to understand your cabling requirements. Below is not a how-to on how to connect everything, it’s just an overview with a brief explanation. The above NetApp document is very detailed and should answer any questions you may have.
A simplified view of a Fabric MetroCluster is as follows:
I found this workflow from NetApp documentation which is quite useful as a guideline on how the bridges should be cabled.
The ATTO FibreBridge devices provide the ability to extend the storage loops across sites by converting the SAS connections to fibre channel. The MDS switches are also used as part of this to pass through the connecting SAS bridge fibre connections and to also create a loop to one of the ATTO bridges on the other site. A picture is worth a thousand words so below you’ll see the diagram of how the SAS shelves link into the FibreBridges.
Once the shelves and the bridges have been cabled you’ll see that both controllers have connectivity to shelves on the other site and this forms the loop required for redundancy and failover. ATTO Bridges are not rocket science, they’re just converter devices and the back of them look similar to the following:
Note that in the below image the MDS switches and bridges are not visible, it’s just an overview of how the SAS connectivity would look
To get the controllers to see the disks attached to the fibre bridges you’ll need to connect the HBAs on the controller to the fibre switch. The best method to connect the HBAs is to have the odd number HBAs go to one switch and even to the other. The key thing however is to ensure that the connections are in different virtual channels.
When performing cabling of the MDS switches you need to ensure you follow the correct virtual channel rules. To improve switch traffic it distributed across VCs to avoid bottlenecks. The FC-VI and inter-switch links are cabled to ports in one VC, and the disk shelf and controller connections are cabled to ports in another VC.
This is an extract from Netapp documentation on how the virtual channels work.
To cable the FC-VI you need to connect both the FC-VI ports on the controller to different switches. These can be either Brocade or MDS 9000 SAN switches. I’d advise to check the compatibility matrix on this beforehand as the may change. FC-VI A goes to switch 1, FC-VI B goes to switch 2. On the same switches two ports from each will be used to create the inter-switch links. This is then also completed on the second site.
Once everything has been connected and you’re feeling like giving yourself an adrenaline kick make sure to power everything off and then you can start assigning disks.