NetApp cabling and hardware basics

In this post we will review some of the NetApp backend connectivity and hardware basics, like disk ownership, single path and multipath, partner cabling, etc. I decided to write this article since I’ve been working with NetApp for some time, but always managing them remotely, never had the chance to stand in front of one of these amazing storage boxes, and never found a NetApp document explaining this stuff, so learning and figuring this has been like a challenge to me and I think it would be nice to share my progress with other ones starting working with NetApp storage.

Let’s see what we are going to talk about:

• Head FC ports
• Shelves and modules
• Disk ownership (hardware and software)
• Cabling (single path and multipath, single nodes and clusters)

Head FC ports

Filers need FC ports in order to be able to connect to fabrics and shelves (there is a chance of having SAS shelves but we will only cover FC shelves on this post). This is the rear view of a FAS3020, as you can see, it comes with 4 FC ports (marked in orange):

These ports are named as 0a, 0b, 0c and 0d, and as said they can be used to connect shelves (configure them as initiators) or to connect to hosts (configure them as targets). You can define whether a FC port is  target or initiator using the fcadmin config command.

As you can see there are 4 expansion slots, here you can place new HBAs to provide the filer increased FC port availability, in this case the ports will be named ‘#X’, where ‘#’ stands for the number of slot where the HBA has been installed and ‘X’ stands for the port index which can be ‘a’ or ‘b’ since most of the HBAs are dual ports.

You can use sysconfig command in order to see how many ports your system has and what it is connected to them. What is connected under each port is known as loop or stack, for example, if under port 0a you have 3 shelves, it would be a loop or a stack composed by 3 shelves. Here it is an example from a NetApp simulator, in this case, as it is a simulator, different adapters will be named as v1, v2, v3, etc.:

Shelves and Modules

As any storage system besides having the controllers or heads, it has shelves where disks should be contained. NetApp has different types of disks, shelves and modules. Depending on the type of disk you select you will be able to use certain types of shelves and certain types of modules. Check the following diagram to understand the possible combinations (as we already said, we won’t cover SAS shelves, but you can take a look at http://www.netapp.com/mx/products/storage-systems/disk-shelves-and-storage-media/disk-shelves-tech-specs-la.html for further information):

Let’s see what each box means:

Disk type:

• ATA Disks: well… you must be already used with these types of disks.
• SATA Disks: Also for this one, already well known.
• FC disks: Fiber channel disks.

To read more on disks types, specially about supported types and sizes, there is a very good NOW article on disk supported types: Available disk capacity by disk size (you might require registering to read the article).

Shelf Models

• ds14mk2: This shelf accepts 14 FC or ATA disks, it takes 3 rack units and accepts ESH2 and AT-FCX disks.
• ds14mk4: This shelf accepts 14 FC disks, it takes 3 rack units and only accepts ESH4 disks.

Modules

Modules allow different disk shelves to be connected to the storage FC loops.

• ESH2: This module is used for FC disks and it is connected to a FC 2GB bus.
• ESH4: This module is used for FC disks and it is connected to a FC 4GB bus.
• AT-FCX: This module is used for ATA or SATA disks and it is connected to a FC 2GB bus.

This is a general guide so you can understand the concepts and differences between disks, shelf types and modules, for further information you can see the technical specs of you storage box. Here you can see the rear view of a disk shelf with only one module installed. Each shelf has space for 2 modules, module A (in the top slot) and module B (in the bottom slot, empty in the next picture).

As we said before, shelves are connected to a loop or stack, each module has an IN port and an OUT port which allow shelf being chained, and between the modules you will see a little green display which will allow you to set the number of shelf inside the loop. In the following picture you can see a single FAS3020 controller with 2 shelves in loop 0a:

This is a single path configuration, but we will talk on that later. The picture shows how rx (receive) and tx (transfer) fibers from 0a port on the filer head are connected to IN port in module A in shelf 1, the from module A in shelf one rx and tx fibers are run from OUT port to IN port in shelf 2 module A.

Disk ownership

Well, here we will make a quick stop and just say there are two types of ownership for disks. But first, let’s define ownership, disks are owned by a filer which is the one that manages the LUNs, shares, exports, snapshots and the rest of the operations over the volumes it hosts, in a cluster scenario the filers must know it’s partner’s disks and only take ownership of those resources in case of a takeover.

About the types of ownership, we have:

• Software based: The less common one, the ownership of the disks in managed with the disk assign command, disks owned by a filer might be distributed across all shelves belonging to the cluster.
• Hardware based: The most common one, the filer connected to the A module of the shelves owns the disks.

Cabling

NetApp supports lot’s of cabling configurations, let’s start reviewing from the simplest one, to the more complex ones. As said in the previous sections of this post there are other supported configurations, such as metro cluster, but here we will cover the most common ones:

• Single node, single path
• Single node, multipath
• Cluster, single path
• Cluster, multipath

Single node, single path

The most simple configuration, this is the less redundant solutions since it has many points of failures, such as a single controller, only 1 loop to each stack and only one module per shelf. In this configuration, rx (receive) and tx (transfer) fibers are connected from any of the FC ports on the controller to IN port in module A in the first shelf of the loop or stack, then from module A in that shelf rx and tx fibers are run from OUT port to IN port in shelf 2 module A, and so on for all the shelves in each stack.

Single node, multipath

This configuration is more secure than the previous one since it reduces all the single point of failures but one (there is still only one controller):

As you can see, in this case shelves have 2 modules each and there are two loops connected to the same stack of shelves (0a, solid line, and 0c, dotted line).

Why have  we used 0a for primary loop and 0c for secondary? well… there is no actual limitation, you can use any adapter you like, but NetApp recommends to use 0a as primary and 0c as secondary and same for 0b / 0d, even across nodes, will talk on this later.

Remember when we talked about disk ownership? If hardware disk ownership has been set (which is in 99% of the cases) 0a loop, connected to the A modules, will own the disks, in case of a faulty module or fiber cable, resources would start being accessed using 0c loop. If you run sysconfig as we have seen early on this post, you would see there are 6 shelves on the system, 3 attached to 0a and three attached to 0c,  usingenvironment shelf command and storage show disk –p command you can identify which shelves are duplicated and which is the loop connected to A module and which to B modules.

For example if you run environment shelf command, you would obtain something like this for each shelf on the system, if you have 3 shelves in 2 loops then you would see this 6 times:

Channel: v0
Shelf: 1
SES device path: local access: v0.17
Module type: LRC; monitoring is active
Shelf status: normal condition
SES Configuration, via loop id 17 in shelf 1:
logical identifier=0x0b00000000000000
vendor identification=XYRATEX
product identification=DiskShelf14
product revision level=1111
Vendor-specific information: Product Serial Number:          Optional Settings: 0×00
Status reads attempted: 844; failed: 0
Control writes attempted: 3; failed: 0
Shelf bays with disk devices installed:
13, 12, 11, 10, 9, 8, 6, 5, 4, 3, 2, 1, 0
with error: none
Power Supply installed element list: 1, 2; with error: none
Power Supply information by element:
[1] Serial number: sim-PS12345-1  Part number: <N/A>
Type: <N/A>
Firmware version: <N/A>  Swaps: 0
[2] Serial number: sim-PS12345-2  Part number: <N/A>
Type: <N/A>
Firmware version: <N/A>  Swaps: 0
Cooling Unit installed element list: 1, 2; with error: none
Temperature Sensor installed element list: 1, 2, 3; with error: none
Shelf temperatures by element:
[1] 24 C (75 F) (ambient)  Normal temperature range
[2] 24 C (75 F)  Normal temperature range
[3] 24 C (75 F)  Normal temperature range
Temperature thresholds by element:
[1] High critical: 50 C (122 F); high warning 40 C (104 F)
Low critical:  0C (32 F); low warning 10 C (50 F)
[2] High critical: 63 C (145 F); high warning 53 C (127 F)
Low critical:  0C (32 F); low warning 10 C (50 F)
[3] High critical: 63 C (145 F); high warning 53 C (127 F)
Low critical:  0C (32 F); low warning 10 C (50 F)
ES Electronics installed element list: 1, 2; with error: none
ES Electronics reporting element: none
ES Electronics information by element:
[1] Serial number: sim-LS12345-1  Part number: <N/A>
CPLD version: <N/A>  Swaps: 0
[2] Serial number: sim-LS12345-2  Part number: <N/A>
CPLD version: <N/A>  Swaps: 0

The first record marked in bold is the serial number for the shelf, since this is an output obtained from a simulated filer the serial number here is missing, and the other two serial numbers marked in bold identifies A and B modules in that shelf respectively. So serial numbers will help to understand which shelves are connected to which loops, then storage show disk –p command will help you to identify which is the primary loop:

As you can see the 3 simulated shelves we saw in the sysconfig output are connected only to v0 adapter (since NetApp simulator does not emulate multipathing to shelves), and also you can see primary port is A.

Cluster, single path

Take a look at the following configuration:

we have 2 nodes interconnected by an infiniband cable (this interconnect cable is used for heartbeat and other cluster related operations and checks), and then we have 2 stacks, one with 3 shelves, and another one with 2 shelves. The first stack has all the A modules connected to 0a loop in controller 1 while B modules are connected to 0c modules in controller 2, as in a single node multipath configuration 0a/0c is used, but the difference now resides in 0a loop belongs to the owning filer and 0c to the partner, not 0a and 0c in the same node. And then we have the very same configuration for node 2, 0a loop is connected to A modules in the second stack and 0c adapter in controller 1 is used to connect the partner to this stack.

In this configuration there is no single point of failure, but there is still one down side, if an A module, or a fiber between primary loops fails a takeover of the resources would be executed by the partner. For example, if module A in shelf 1 (upper one) on 0a loop in controller 1 fails (I know it might sound confusing, read it twice if necessary, I had to hehehe), controller 1 would lost connectivity to the whole stack, in this case, controller 2 would have to takeover resources from controller 1 in order to be able to continue servicing storage. Unfortunately, the takeover process implies cifs service to be restarted, all cifs connections are dropped, so it can be really disruptive.

Cluster, multipath

Ok, now take a look at the following graph and go crazy!

Let’s do some writing to describe connections because it is really hard to follow the lines hehe:

Controller 1 (left) has:

• 0a loop connected to A modules in stack 1 (3 shelves, left).
• 0c adapter is connected to B modules in stack 2 (2 shelves, right).
• 0b loop is used to provide a second patch for controller 1 to stack 1, it is connected to B module OUT port in last shelf on the loop, this way if A module fails in any of the shelves on this stack Controller 1 would still have access to the disks on this stack without need to failover resources over the partner.
• 0d port is connected to shelf 2 in stack 2 module A OUT port, this way if resources have been failed from controller 2 to controller 1 you still don’t have a single point of failure.

Controller 2 (right) has:

• 0a loop connected to A modules in stack 2 (2 shelves, right).
• 0c adapter is connected to B modules in stack 1 (3 shelves, left).
• 0b loop is used to provide a second patch for controller 2 to stack 2, it is connected to B module OUT port in last shelf on the loop, this way if A module fails in any of the shelves on this stack Controller 2 would still have access to the disks on this stack without need to failover resources over the partner.
• 0d port is connected to shelf 3 in stack 1 module A OUT port, this way if resources have been failed from controller 1 to controller 2 you still don’t have a single point of failure.

As you might have already guessed this is the most redundant configuration (at least among the standard ones, never worked with metro cluster for example so can’t talk about it), then only down side of this configuration is you have to use 2 FC ports per head in order to provide access to a stack of shelves, this might become really expensive in FAS6240 environment which might have LOTS of shelves in different stacks..

Hope you liked the article and helped you to understand some of the basics, as always, any questions or comments are welcomed.

Cisco UCS Fabric Interconnect Server Profile Setup

In this tutorial we will be setting up a Server Profile that will be applied to our Cisco UCS Blade. Before creating a Service Policy there are a few pre-requisites that need to be done. We will need to create a few policies and pools to be used when we go through the Service Profile creation process. Please follow the links below:

Create a Bios Policy
Create a UUID Suffix Pool
Create a MAC Address Pool
Create a Maintenance Policy
Create a boot policy
Create a local disk policy
If you are using Fiber Channel – Cisco UCS Fabric Interconnect WWNN and WWPN Pool
If you are using Fiber Channel – Cisco UCS Fabric Interconnect VSAN Setup

1. Open Cisco UCS Manager and click on the Servers Tab – Servers – Service Profiles. Right click on either Root or your organization and select Create Service Profile. Give the Service Profile a name. You can create either a static Service Profile or an Updating Service Profile. Updating Service Profile means any update you make to the Service Profile i.e. add in a vlan will be automatically applied to all servers instantly. As with a Static no changes are made to the existing service profiles even if created with the static service profile.

2. For UUID Assignment select the UUID Pool that we created from the pre-requisites. Click Next.

3. For Local Storage, select the Local Disk Policy that we created in the pre-requisites.

4. Next to How would you like to configure SAN connectivity, select Expert.

5. In the World Wide Node Name Assignment box select the WWNN Pool that we created in the pre-requisites.

6. Click Add at the bottom of the screen. Here we will create 2 vHBA named FC0 and FC1. Under World Wide Port Name Assignment, select the WWPN pool that we created in the pre-requisites. For FC0 select Fabric ID A and for FC1 select Fabric ID B. Select the VSAN that you also created in the pre-requisites and for the Adapter Policy select VMWare. Click Ok.

8. The vHBA’s are now listed. Click Next.

9. In the Networking section we will be adding 2 vNIC’s to our Service Profile.

10. Click Add. We will be creating 2 vNIC’s called eth0 and eth1. Both using the MAC-Pool that we created in the pre-requisites, both with a 9000MTU and Adapter Policy of VMWare. However eth0 will be connected to Fabric A and eth1 with be connected to Fabric B.

11. You can see the two vNIC’s are created displaying which vNIC is connected to which Fabric ID. Click Next.

12. You have the option to create a placement policy of let the system perform placement. This is similar to PCI order or priority on a system motherboard. Click Next.

13. Select the Boot Policy that we created in the pre-requisites. The boot order is displayed at the bottom of the page. Click Next.

14. Select the Maintenance Policy that we created in the Pre-Requisites and click Next.

15. You have the option to assign this Service Profile to a server/blade now or later. I’ll select to assign later. Click Next.

16. Select the BiosPolicy that we created in our pre-requisites and click Finish. The Server Profile is now created and ready to associate to a one of your blades.

Cisco UCS Fabric Interconnect VSAN Setup

Referring to our Cisco UCS lab setup, we will now configure our Virtual SAN or vSAN setup. As described and well explained on The Unified Computing Blog, there are two types of vSANs that we can create. One being in the SAN Cloud and the second being in the Storage Cloud. If you create a VSAN in the Storage Cloud you would be direct connecting your Storage into the Fabric Interconnect. If you have Fiber Channel switches external to your Cisco UCS Infrastructure and you wish to connect your Fabric Interconnect to your external switches, you will use the SAN Cloud option. In this tutorial we are connecting to external fiber channel switches so we will create our VSAN under the SAN Cloud tree.

1. Open Cisco UCS manager and click on the SAN tab. Expand SAN Cloud.

2. Right click on VSANs and select Create VSAN. Give your VSAN a name, leave Default Zoning as disabled – the default, select Both Fabrics Configured Differently. We will now give Fabric A VSAN a different VSAN ID than Fabric B VSAN to isolate traffic. For the Fabric Interconnect to isolate traffic to the Chassis which travels via FCoE we need to assign two separate vlans.

3. Click Finish to conclude setting up your VSAN. Next click on Equipment Tab and expand Fabric Interconnects – Fabric Interconnect A – Expansion Module 2 – Uplink FC Ports – FC Port 1. On the right hand side select the new VSAN you created in the previous steps next to VSAN. Click Save Changes. Do this for each FC Port that you wish to connect to your fiber channel switches. Repeat the process for Fabric Interconnect B. For Uplink FC Ports that are not in use, right click each of them and select disable to avoid unnecessary alerts.

4. Before connecting the fiber cables to your fiber channel switch, you must configure the fiber channel switch port as NPIV. NPIV makes the fiber channel port act similar to a trunk port in a switch and allows fiber channel connectivity from each blade to come through the one port. Blade Server vHBA’s will be discussed later in creating Service Profiles.

Cisco UCS Fabric Interconnect Basic System Configuration

1. The following dialog will appear:

—- Basic System Configuration Dialog —-

This setup utility will guide you through the basic configuration of
the system. Only minimal configuration including IP connectivity to
the Fabric interconnect and its clustering mode is performed through these steps.

Type Ctrl-C at any time to abort configuration and reboot system.
To back track or make modifications to already entered values,
complete input till end of section and answer no when prompted
to apply configuration.

2. Here we will be configuring the initial part of the Fabric Interconnect via console

Enter the configuration method. (console/gui) ? console

3. As this is a clean install and I’m not restoring any data select setup.

Enter the setup mode; setup newly or restore from backup. (setup/restore) ? setup

4. Select Yes to setup a new Fabric Interconnect.

You have chosen to setup a new Fabric interconnect. Continue? (y/n): y

5. Enter in the admin password you’d like to use. Must be at least 8 characters.

Enter the password for “admin”:
Confirm the password for “admin”:

6. As we are configuring our Fabric Interconnects as a cluster we will select yes.

Is this Fabric interconnect part of a cluster(select ‘no’ for standalone)? (yes/no) [n]: yes

7. As this is the first switch we are configuring we will select A. When we configure our secondary switch we will select B.

Enter the switch fabric (A/B) []: A

8. Enter a system name for the cluster. This name will apply to the cluster and each Fabric Interconnect will append either -A or -B depending on what you selected in the previous step.

Enter the system name: ucs-lab

9. Enter in this physical interconnects ip address, subnet mask and default gateway.

Physical Switch Mgmt0 IPv4 address : 192.168.1.10
Physical Switch Mgmt0 IPv4 netmask : 255.255.255.0
IPv4 address of the default gateway : 192.168.1.1

10. Enter in the shared clustered IP address. This IP address will be shared amongst both Fabric Interconnects and will be used to connect to when you launch the Cisco UCS Manager.

Cluster IPv4 address : 192.168.1.30

11. Optionally configure your DNS servers. Select yes and enter in the IP address of your DNS server.

Configure the DNS Server IPv4 address? (yes/no) [n]: y
DNS IPv4 address : 192.168.1.50

12. Optionally configure a default domain name. I want to skip this step so i’ll select no.

Configure the default domain name? (yes/no) [n]:n

13. A summary is provided with your settings. Select yes to save and reboot or select no to go back and make changes.
Following configurations will be applied:

Switch Fabric=A
System Name=ucs-lab
Physical Switch Mgmt0 IP Address=192.168.1.10
Physical Switch Mgmt0 IP Netmask=255.255.255.0
Default Gateway=192.168.1.1
DNS Server=192.168.1.50

Cluster Enabled=yes
Cluster IP Address=192.168.1.30

Apply and save the configuration (select ‘no’ if you want to re-enter)? (yes/no): yes
Applying configuration. Please wait.

Configuration file – Ok

12. This completes the setup for Fabric Interconnect 1. Leave Fabric Interconnect 1 powered on and connect to the console port of the Fabric Interconnect 2. Power on Fabric Interconnect 2.

13. We are presented with the Basic System Configuration Dialog again.

—- Basic System Configuration Dialog —-

This setup utility will guide you through the basic configuration of
the system. Only minimal configuration including IP connectivity to
the Fabric interconnect and its clustering mode is performed through these steps.

Type Ctrl-C at any time to abort configuration and reboot system.
To back track or make modifications to already entered values,
complete input till end of section and answer no when prompted
to apply configuration.

14. Once again select console.

Enter the configuration method. (console/gui) ? console

15. The setup wizard detects the presence of Fabric Interconnect 1 and this switch (Fabric Interconnect 2) will be added to the cluster. Type y.

Installer has detected the presence of a peer Fabric interconnect. This Fabric interconnect will be added to the cluster. Continue (y/n) ? y

16. Enter in the admin password we created while setting up Fabric Interconnect 1. This switch will now connect into the fabric and retrieve the config from the Fabric Interconnect 1.

Enter the admin password of the peer Fabric interconnect:
Connecting to peer Fabric interconnect… done
Retrieving config from peer Fabric interconnect… done
Peer Fabric interconnect Mgmt0 IP Address: 192.168.1.10
Peer Fabric interconnect Mgmt0 IP Netmask: 255.255.255.0
Cluster IP address : 192.168.1.30

17. Enter in the physical ip address for this switch (Fabric Interconnect 2.)

Physical Switch Mgmt0 IPv4 address : 192.168.1.20

18. Type yes to apply and save the configuration. The Fabric Interconnect Cluster is now ready to manage.

Apply and save the configuration (select ‘no’ if you want to re-enter)? (yes/no): yes
Applying configuration. Please wait.

Configuration file – Ok

19. Log into your management station and open your web browser. Type in the Fabric Interconnect Cluster ip in the address bar and press enter. Click Launch once the page is displayed.

20. You can Always trust the content from this publisher or simply click Run.

21. Enter in the admin password you created earlier and click Login.

22. You are now connected to the Cisco UCS System Manager.