Resource Management Features of the Symmetrix VMAX for Cloud Providers

Pre-2018 post from old blog...please check support.emc.com for latest information.

As cloud computing continues to consolidate more and more multiple, disparate workloads into our compute, network and storage platforms there is now an ever growing need to control resource management at various different component layers within each stack.

Over the years the EMC has developed a portfolio of powerful, trusted features for the Symmetrix that enables cloud providers to dynamically manage resources both inside and outside the array whilst delivering a compelling total cost of ownership (TCO) that is required to compete in this utilities market.

• At the front end of the VMAX we have Auto Provisioning Groups and Host I/O Limits. Auto Provisioning Groups facilitate rapid change management for storage presentation and reclamation activities aka LUN masking through the use of groups of HBAs, targets and LUNs. Here providers have the ability to dynamically manage, scale and balance configurations for their subscribers. Host I/O Limits are a feature of Auto Provisioning Groups that provide the ability to place workload limits on particular services or subscribers.
• In the mid section of the array we have a powerful feature called Dynamic Cache Partitioning (DCP) that enables logical separation of cache.
• At the back end we have Fully Automated Storage Tiering for Virtual Pools (FAST VP) which is a truly awesome feature that leverages the difference in IO densities and capacities of flash and traditional spinning rust to provide a better response time at a lower cost by moving your most active data to the most appropriate performance tier and your inactive data to a bulk storage tier! Of all the features the Symmetrix has this is by far my favorite as it delivers so much operational efficiency coupled with numerous environmental and and cost benefits. For cloud or multi-tenant providers that have subscribers with varied workloads this is a must have.
• Federated Tiered Storage (FTS) enables providers to add an external storage tier from any supported EMC or 3rd party array that can avail of all the mature Symmetrix features like SRDF, TimeFinder etc.

Symmetrix VMAX Auto Provisioning Groups

Pre-2018 post from old blog...please check support.emc.com for latest information.

For those new to Symmetrix VMAX, Auto Provisioning is probably what you'll assume has been the norm for Symmetrix administrators for years but if you've experienced provisioning on the previous models with many devices to large clusters over many front end ports you'll recall either the endless hours you put in to ensure everything was mapped and masked properly.

Those days are gone!

With Auto Provisioning the entire provisioning mechanism has been vastly simplified and is now extremely rapid with some new SYMCLI commands and presentation wizards in Unisphere for VMAX.

Auto Provisioning Groups are a very welcomed paradigm shift for storage administrators during the presentation and reclamation of devices from the array and there are a few concepts to first understand and we'll cover each of them in turn before getting into Auto-Provisioning itself.

If you've been doing this on Symmetrix prior to VMAX then skip the first two sections as they will only tell you how to suck eggs!

Symmetrix Device Mapping and Masking 101

For a Symmetrix Logical Volume which I'll call a symvolid to be presented to a host it must first be mapped to a front-end port and given a channel address which we'll call a LUN number - this task is known as mapping.

Once the devices have been mapped up to a front-end port and given a LUN number an HBA must then be granted access to it - this task is known as masking and on DMX the front-end ports must be VCM-enabled and on VMAX ACLX-enabled. When a port is VCM or ACLX-enabled this means device masking must be carried out to grant access.

On Symmetrix it is recommended to map the symvolid to at least two front-end ports and then mask it to at least two HBAs.  Prior to Auto-Provisioning this procedure took a few commands each of which took up to 20 minutes to perform on the array.

With Auto-Provisioning - storage can be presented to even the largest of clusters in minutes.

Channel Addressing and Dynamic LUN Addressing

With Solutions Enabler 7.0 and Enginuity 5772 the Symmetrix gained a feature known as Dynamic LUN Addressing (DLA) which enables storage administrators to dynamically assign LUN addresses on a per HBA level - and now at the initiator group level on VMAX.  Previously you could only ever use the channel address assigned to the device on the front-end ports and this at times caused havoc for those operating systems like Windows and ESXi that are restricted to an address range of 00-FF (0-256).  If you had a Windows box and masked out a device with a LUN number higher than FF then you simply couldn't discover the device as the address range is beyond what Windows can natively address. DLA is optional on 5772 and 5773 but is default on Auto-Provisioning (5874 and higher) and empowers storage administrators with the ability to serve hosts with restricted address ranges on FA ports that have more than 256 devices. In a nutshell, with DLA it really doesn't matter what channel address is assigned to the device on the port as DLA will modify this at an initiator group level.


...enter Auto-Provisioning Groups

Auto-Provisioning introduces device management by joining groups of objects in a masking view. Lets take a look at the groups. In the examples shown we'll have a two node ESXi cluster - each host with its own unique boot from SAN device and a shared device accessible by both hosts. Host 1 will have four local devices and Host 2 one.


Initiator Groups contain the World Wide Port Name (WWPN) objects of the HBAs used by your host or cluster.

Auto-Provisioning: Initiator Group

Auto-Provisioning: Cascaded Initiator Group

• a VMAX can contain up to 8192 initiator groups
• cascaded initiator groups allow for groupings of initiator groups and can only go one deep - a real handy feature for clusters
• cascaded initiator groups have a maximum of 1024 initiators total
• a maximum of 32 Fibre Channel initiators or 8 iSCSI names per group or a combination of both per group
• an initiator can be a direct member of only one group
• an initiator group can be a member of more than one masking view
• port flags can be set at the initiator group level
• in general its more efficient to group all HBAs from one host in an initiator group
• the consistent_lun parameter can be used to ensure the same address is assigned to a device presented to an initiator group and is generally what I use
• akin to the Hostss folder on Navi/Unisphere Storage Groups
• suggested naming conventions: IG_HOSTNAME or CIG_CLUSTERNAME

We'll first create the initiator groups unique to each ESXi host for the local devices and another cascaded initiator group to use for the shared devices masking view that will be presented the cluster shared storage to both hosts.

symaccess -sid 1234 create -type initiator -name IG_ESXiNODE1 -consistent_lun <<< create the group and name it for the host
symaccess -sid 1234 -type initiator -name IG_ESXiNODE1 add -wwn 211100243581230d <<< add the WWN of the first HBA to the group for the host
symaccess -sid 1234 -wwn 211100243581230d set hba_flags on D -enable <<< toggle any port flags at the initiator level
symaccess -sid 1234 -type initiator -name IG_ESXiNODE1 add -wwn 211100243541230e
symaccess -sid 1234 -wwn 211100243541230e set hba_flags on D -enable
symaccess -sid 1234 create -type initiator -name IG_ESXiNODE2 -consistent_lun
symaccess -sid 1234 -type initiator -name IG_ESXiNODE2 add -wwn 211100243581123f
symaccess -sid 1234 -wwn 211100243581123f set hba_flags on D -enable
symaccess -sid 1234 -type initiator -name IG_ESXiNODE2 add -wwn 211100243541123e
symaccess -sid 1234 -wwn 211100243541123e set hba_flags on D -enable
symaccess -sid 1234 create -type initiator -name IG_ESXiCLUS -consistent_lun <<< create the cascaded initiator group and name (this is the parent object)
symaccess -sid 1234 -type initiator -name IG_ESXiCLUS add -ig IG_ESXiNODE1 <<< add the initiator groups (these are the child objects)
symaccess -sid 1234 -type initiator -name CIG_ESXiCLUS add -ig IG_ESXiNODE2


Port Groups contain front-end port objects from the array that are to be used by your host or cluster.

Auto-Provisioning: Port Group

There are generally two ways to implement port groups and I generally term these as rigid or elastic. Rigid being where you configure a fixed number of port groups with a fixed number of ports. Elastic being were you have a defined or modeled port group layout but create a new port group for every host based on these definitions or models and can grow or shrink these on a per host basis by adding or removing ports. Each have pro and cons from a day to day management perspective but my preference would be to go with the elastic approach.

• a VMAX can contain up to 8192 port groups
• the maximum number of ports equals maximum number of ports in the VMAX - currently 128 for an 8-engine box
• ports can be a member of more than one group which is handy if you decide to implement a port group per server or cluster
• port groups can be a member of more than one masking view
• for availability and performance port groups should contain a minimum of two ports from different directors and with thin provisioning a minimum of four ports should be considered - remember that for platforms like ESXi increasing the number of ports beyond four decreases the maximum number of LUNs the host can address as it 256 LUNs / 1024 paths.
• loosely akin to the advanced HBA registration in Navi/Unisphere Storage Groups
• suggested naming conventions: PG_NAME (hard) or PG_HOSTNAME (soft) or PG_CLUSTERNAME (soft)

In this example we will create three hard port groups on this VMAX - two we'll use for the local device masking (each with four ports) and the third (with eight ports) for the cluster shared device masking.

symaccess -sid 1234 create -type port -name PG_01_4MEM <<< create the 4 member port group
symaccess -sid 1234 -type port -name PG_01_4MEM add -dirport 8E:0 <<< add ports to the group
symaccess -sid 1234 -type port -name PG_01_4MEM add -dirport 7E:0
symaccess -sid 1234 -type port -name PG_01_4MEM add -dirport 9E:1
symaccess -sid 1234 -type port -name PG_01_4MEM add -dirport 10E:1
symaccess -sid 1234 create -type port -name PG_02_4MEM
symaccess -sid 1234 -type port -name PG_02_4MEM add -dirport 8F:0
symaccess -sid 1234 -type port -name PG_02_4MEM add -dirport 7F:0
symaccess -sid 1234 -type port -name PG_02_4MEM add -dirport 9F:1
symaccess -sid 1234 -type port -name PG_02_4MEM add -dirport 10F:1
symaccess -sid 1234 create -type port -name PG_07_8MEM <<< create the 8 member port group
symaccess -sid 1234 -type port -name PG_07_8MEM add -dirport 8E:0 <<< add ports to the group
symaccess -sid 1234 -type port -name PG_07_8MEM add -dirport 7E:0
symaccess -sid 1234 -type port -name PG_07_8MEM add -dirport 9E:1
symaccess -sid 1234 -type port -name PG_07_8MEM add -dirport 10E:1
symaccess -sid 1234 -type port -name PG_07_8MEM add -dirport 8F:0
symaccess -sid 1234 -type port -name PG_07_8MEM add -dirport 7F:0
symaccess -sid 1234 -type port -name PG_07_8MEM add -dirport 9F:1
symaccess -sid 1234 -type port -name PG_07_8MEM add -dirport 10F:1


Storage Groups contain device objects from the array that are to be presented to your host or cluster.

Auto-Provisioning: Storage Group

• a VMAX can contain up to 8192 storage groups
• the maximum number of devices per group is 4096
• a device can be a member of more than one storage group
• a storage group can be a member of more than one masking view
• devices are assigned dynamic LUN addresses using the Dynamic LUN Adressing (DLA) feature
• used by FAST DG and FAST VP to associate the content of a storage group with a tiering policy
• akin to the LUNs folder on Navi/Unisphere Storage Groups
• suggested naming conventions: SG_HOSTNAME or SG_CLUSTERNAME

In this example we have three storage groups: two for the unique local devices and one for the cluster shared devices.

symaccess -sid 1234 create -type storage -name SG_ESXiNODE1 <<< create the storage group for each host
symaccess -sid 1234 create -type storage -name SG_ESXiNODE2
symaccess -sid 1234 -type storage -name SG_ESXiNODE1 add dev 11A <<< add devices to the storage group
symaccess -sid 1234 -type storage -name SG_ESXiNODE1 add dev 11B
symaccess -sid 1234 -type storage -name SG_ESXiNODE1 add dev 11C
symaccess -sid 1234 -type storage -name SG_ESXiNODE1 add dev 11D
symaccess -sid 1234 -type storage -name SG_ESXiNODE2 add dev 222
symaccess -sid 1234 create -type storage -name SG_ESXiCLUS <<< create the storage group for the cluster shared devices
symaccess -sid 1234 -type storage -name SG_ESXiCLUS add dev 333 <<< add devices to the cluster shared devices storage group


Masking Views knit these groups of objects together on the array and automatically perform the mapping of the devices in the storage group to the front-end ports in the port group and the masking of the devices in the device group to the initiators in the initiator group.

Note this example shows example names used in the SYMCLI examples.

Auto-Provisioning: Masking View

• a single view can only contain one initiator group, one port group and one storage group
• akin to Navi/Unisphere Storage Groups
• suggested naming conventions: MV_HOSTNAME or MV_CLUSTERNAME

The final SYMCLI command we need to run is to create the view. Here we have three views: one for the local storage on each node and one for the cluster shared devices.

symaccess -sid 1234 create view -name MV_ESXiNODE1 -sg SG_ESXiNODE1 -ig IG_ESXiNODE1 -pg PG_01_4MEM <<< create the masking view for the host
symaccess -sid 1234 create view -name MV_ESXiNODE2 -sg SG_ESXiNODE2 -ig IG_ESXiNODE2 -pg PG_02_4MEM
symaccess -sid 1234 create view -name MV_ESXiCLUS -sg SG_ESXiCLUS -ig CIG_ESXiCLUS -pg PG_01_8MEM <<< create the masking view for the cluster shared devices

New Storage Presentation and Reclamation

So when you've configured your masking view and your cluster is now in production you get a request to present more storage. With Auto Provisioning to present additional devices all you have to do is add the device into the storage group and Auto Provisioning will automatically map and mask the device for you.

symaccess -sid 1234 -type storage -name SG_ESXiCLUS add dev 786  <<< new device presented to all cluster nodes in one command

Storage Reclamation is also fairly straightforward. If you need to remove a device or a number of devices from a host or cluster you simply remove the device from the storage group and you have the option to unmap in the same command.

symaccess -sid 1234 -type storage -name SG_ESXiCLUS remove dev 786 -unmap  <<< device removed from all cluster nodes and unmapped from all front-end end ports


Further information on Auto Provisioning and Dynamic LUN Addressing can be found @ http://support.emc.com/

Symmetrix Virtual Provisioning (VP) Migrations and Space Reclamation

Pre-2018 post from old blog...please check support.emc.com for latest information.

So I've put on a bit of weight over the last few years and when I talk about thick to thin in a storage context I wish I could apply a thin migration plan to myself but alas the only solution for that is exercise which I don't do enough of at the moment!

Anyhow - migrating to thin pools is something that comes up in discussion on a regular basis and over the last few months I've been privileged enough to author migration plans for and use in anger most of the great tools that EMC have to ensure customers get the very best out of that minuscule 768KB sub-extent size utilized by the Symmetrix VMAX!

Typical migrations usually consist of loads of traditionally provisioned storage arrays that are being consolidated into a new EMC Symmetrix VMAX which in most environments these days is being configured with VP pools were customers are leveraging other technologies such as Virtual-LUN and FAST VP or just simply VP itself.

For those who intend to or are leveraging virtual provisioning pool over-subscription the chances are that you're going to want to strip out the unused zeroes during any inbound data migration......and EMC provide a range of array and host level tools to facilitate this objective.

Array-based migrations to thin pools

On the VMAX itself there are currently two migration methods that you can use to strip out those unwanted zeroes and the products that facilitate this are SRDF and Open Replicator. For those of you who have been around Symms these products have been available for years and are probably the most tried and tested replication products in the enterprise storage market and in VMAX the developers have added the ability to reclaim zeroed space in flight with a few caveats. With SRDF, you must currently use adaptive copy mode and with Open Replicator the control volumes must exist on the Symmetrix VMAX and these methods are the perfect fit for migration purposes.

Host-based Migrations to thin pools

If your migration plan can't leverage an array based method then you're going to ask the host to move the data, which is not ideal as the array should do this for you but is sometimes a necessity for one reason or another. EMC have the tools in this space to help and these are currently Open Migrator/LM and PowerPath Migration Enabler (Host Copy).

Open Migrator/LM is a very powerful product that in its simplest form copies an existing volume to a new VMAX volume and once in sync you choose a time to swap the target device over to be the new source - simple as that. If you are migrating Windows Server 2003 or 2008 machines to your new VMAX and need to correct partition alignment and/or NTFS cluster size then this product can do both - with the applications online.

PowerPath Migration Enabler (Host Copy) is a great feature of PowerPath. Assuming you've PowerPath patched to supported levels you can leverage PowerPath to facilitate an online migration for any device managed by PowerPath irrespective of whether its a LUN from an HP, IBM, EMC or any supported array.

It's also worth noting that many third party vendors at this point including VMware and Symantec support thin reclamation initiatives and recent versions of Storage Foundations and ESXi support various thin-aware features like SmartMove and Storage vMotion that can remove unsed zeros during migrations.

Conclusion

Some of the benefits behind reclaiming contiguous zeroes in flight during a migration are that it minimizes capacity consumption and reduces the load on the array as only extents with data are actually being written to disk. All in all these migration methods can be used to reduce or eliminate the downtime usually associated with such migration activities and reclaim valuable unused capacity.

Those who can't take advantage of these thin friendly migration tools to Symmetrix VMAX you do have the option to run an online Zero Space Reclaim (ZSR) post traditional migration methods which releases blocks of contiguous unused extents back into the pool for reuse.

vSphere and EMC Symmetrix port flags

Pre-2018 post from old blog...please check support.emc.com for latest information.


My first experience of Symmetrix and VMware host connectivity and provisioning was back in the days of ESX 3.0.0 and DMX-3s running 5771 code and things have changed since then.

On the ESX side no more manual VMFS partition alignment and numerous other changes enabled by the vStorage APIs that are a great show of maturity and integration on both the VMware and storage vendor fronts. On the Symmetrix side we now have VP, FAST-DP, FAST-VP, Virtual LUN, Auto-Provisioning and other significant integration improvements made available via the excellent EMC Virtual Storage Integrator (VSI) plug-ins for vSphere.

What hasn't really changed that much (if you've been through the SPC-2 enablement with ESX 3.x) is the bit settings on the FA ports and with the release of vSphere 4.1 a few months back I revisited the Fibre Channel SAN Configuration Guide that notes the bit settings (or flags) that are required on Symmetrix.

If you are lucky enough to have VMAX and not DMX then the change you'll love is that device masking or 'fibrepath enabled' has changed to Auto-Provisioning groups.  Administrators of large clusters will love this feature as it will save so much time when presenting out new devices or introducing new nodes into the cluster.

My notes on the bit settings are below.

Notes on Symmetrix bit settings for vSphere

NOTE
Always check the EMC E-lab Navigator or EMC Support Matrix for up to date support info.

Requirement copied verbatim from the vSphere 4.1 Fibre Channel SAN Configuration Guide

EMC Symmetrix storage systems work with ESX/ESXi hosts in FC SAN configurations.  Generally, you use the EMC software to perform configurations.  The following settings are required on the Symmetrix networked storage system.  For more information, see the EMC documentation.

• Common serial number (C)
• Auto negotiation (EAN) enabled
• Fibrepath enabled on this port (VCM)
• SCSI 3 (SC3) set enabled
• Unique world wide name (UWN)
• SPC-2 (Decal) (SPC2) SPC-2 flag is required

The ESX/ESXi host considers any LUNs from a Symmetrix storage array with a capacity of 50MB or less as management LUNs. These LUNs are also known as pseudo or gatekeeper LUNs. These LUNs appear in the EMC Symmetrix Management Interface and should not be used to hold data.

The settings in the bullet points are FA port flags and are split into two categories - SCSI port flags and Fibre port flags.

Currently the VMware vSphere 4.1 Fibre Channel SAN Configuration Guide does not note that the ACLX flag is required to enable Auto-Provisioning groups on VMAX nor does not note that the OS2007 bit is optional.

SCSI port flags 

Common_Serial_Number [C] enabled

• Provides a unique serial number on multiple paths for the same Symmetrix device.
• Needs to be enabled if multipathing is being used.

SCSI_3 [SC3] enabled 

• Alters the INQUIRY data to to report that any device on the port supports ANSI version 'SCSI-3 SPC' (0x3).
• For WSFC guests the SCSI-3 persistent reservation (or [PER] bit) must still be set on the devices it uses that are under cluster control.
• Required to make use of SPC2.

SPC2_Protocol_Version [SPC2] enabled

• Alters the ID of the LUN to contain the network address authority (NAA) identifier.
• The LUN serial number is stored on page 0x80 and the NAA ID is stored on page 0x83.
• ESX/ESXi 3.x and higher read the INQUIRY data from page 0x83 (NAA id) as opposed to 0x80 (LUN serial number). 
• In ESX/ESXi, the NAA ID is persistent through reboots whereas the runtime name of the device is not.
• Required for advanced SAN-aware applications such as VMware Site Recovery Manager and the EMC Virtual Storage Integrator suite.

SCSI_Support1 [OS2007] enabled (optional for vSphere 4.0 and higher) 

• From the VMware documentation: "When enabled, this flag provides stricter compliance with SCSI standards for managing device identifiers, multiport targets, unit attention reports, and the absence of a device at LUN 0."

Fibre Protocol port flags

ACLX or Fibre path enabled on this port [ACLX] enabled (VMAX 5874 and higher only)

• Enables Auto-Provisioning Groups on the port.

VCM_State or Fibre patch enabled on this port [VCM] enabled (DMX3/4 577x only) 

• Enabled for device masking or volume logix.

Auto_Negotiate or Auto Negotiation [EAN] enabled

• Enables a handshake between two ports for optimal data transfer.

Unique_WWN or Unique world wide name [UWN] enabled

• Ensures unique World Wide Names in the fibre environment for the array components by using the Symmetrix serial number and port numbers.

References

• Solutions Enabler Documentation Set (Powerlink account required)
• vSphere 4.1 Fibre Channel SAN Configuration Guide