Low-Cost DR with SVC Stretched Cluster

Ever since IBM’s SAN Volume Controller was first delivered back in 2003 folks have been looking at the 2n cluster nodes and wanting to separate them physically to achieve a degree of disaster recovery. This is often referred to as  ‘split i/o group’ or ‘split cluster’ but as of April 2013 I have been told the preferred nomenclature is now “SVC Stretched Cluster”. For this to be a good idea, two additional features were needed. Vdisk mirroring, which was delivered with SVC 4.3 and the ability to manually place the quorum data wherever you want, which was delivered in SVC 5.1.

The key advantage of stretched cluster is that if you lose one of your computer rooms, the SVC will keep processing and is designed to automatically switch to using the secondary copy of any vdisks in less than 30 seconds. All this without any additional replication licensing being required.

There are some design rules you need to pay attention to if you want to maximise the DR value from stretched cluster. Firstly, SVC is a cluster and as such it has a quorum. SVC maintains 3 copies of the quorum. In the following examples I will assume we have two SVC nodes (SVC can include 2, 4, 6 or 8 nodes, but the vast majority are 2 node systems). The principles are the same for all cluster sizes.

To build a robust DR solution using SVC stretched cluster you ideally want 3 computer rooms – two of them up to 10Kms apart and a third within OM3 shortwave distance of the second. The first is to house the SVC node1, the second is to house SVC node2 and the third is to house your primary quorum. In reality you don’t need three computer rooms, but you do need 3 separate power sources as a minimum.

The SVC is an inherently HA device, but if you lose half the nodes (1 in this case) plus the primary quorum, the SVC will stop processing. However, with the quorum in a 3rd location, on separate power, you would have to be the victim of a campus-wide disaster to lose the system.

Just to clarify – this allows you to build a localized DR solution using ‘free’ features of SVC. You can of course do SVC to SVC replication over any distance you desire using the licensed  Global Mirror function, and that would give you a much more complete DR solution because of the greater physical separation.

There are a few different ways to skin the stretched-cluster cat, but the one that seems to be the most practical is something like this:

Location 1:

  • One SVC node
  • At least one disk system
  • At least 2 FC switches

Location 2:

  • One SVC node
  • At least one FC switch

Location 3:

  • At least one disk system. Must be from a special list of qualified systems, but can be very small e.g. DS3500 with 3 drives
  • At least one FC switch

The switches have been placed so as to avoid a fire in any one room taking out a second room. The fabric can of course be more complex with core/edge etc as required. You’ll note the longwave connections. The CF8 hardware nodes that have been shipping with SVC since 2009 have a standard option to add longwave SFPs. If you want to do stretched cluster on the 8A4 entry nodes you will need to source your own Finisar FTLF1424P2BCD longwave SFPs. You will need two supported longwave SFPs in each SVC node.

Remember the two key advantages of this solution:

  1. Lower cost than replication between two completely separate systems
  2. Automatic failover (<30 seconds) if a site goes down

Content revised for clarity 12th July 2010, 27th July 2010. “Stretched Cluster” terminology  clarified 23rd April 2013.

Advertisements

8 Responses

  1. Interesting, but as much as I can see you add the non-free third site and SVC uses per TB licensing and third site is nothing without SVC.
    In some way is similar to use 1 complete SVC pointing to original data and 2 groups of storage systems, 1 in main DC and the other in secondary. If something goes wrong you have the secondary site with all data in the other site.
    Dark fiber (10 km)? Maybe it’s better to use 2 SVCs :-)

    Like

    • CR1 and CR2 can of course be a lot closer than 10Kms. A lot of folks have two rooms across their campus, or across the street etc, that are linked already. ‘CR3’ can also be simply the far corner of CR2, provided you have it on separate power. That will still give you some localized DR capability e.g. fire in one end of the computer room.

      Like

  2. I want to implement 3 way DR solution for a Oracle Database solution. Can SVC help me achieving it and how effective and efficient this solution will be..??

    Response awaited.

    Like

  3. As always, it depends on your specific requirements, but yes one choice for three site mirroring is to use SVC split cluster and volume mirroring (which is synchronous) to create a two site high availability solution (across FC up to 10Km) and then SVC Global Mirror to replicate asynchronously to a third site (which could even be a Storwize V7000).

    One nice thing about split cluster with volume mirroring is that there is no manual volume failover required between site1 and site2 since SVC will automatically use the secondary volume. Also when you have a third site you can locate the quorum at the third site which makes split cluster more robust.

    There are other options of course, like DS8800 three site mirroring, or a combination of Oracle DataGuard log-shipping from site1 to site2 and then storage replication from site2 to site3 etc…

    Like

  4. Note that the max distance for SVC Split Cluster has now been increased to 300 Km. Check out the SVC and Storwize V7000 Replication Family Redbook, due for publication in July 2012 for more info.

    Like

  5. What happens if You loose 2 of 3 quorums? Is it possible to create a new quorum to get majority again on the second site?

    Like

  6. The three quorums are on 3 different sites (or power domains at least). If you have two nodes running in an iogroup and you lose the primary quorum, the second quorum gets promoted to be active. If you lose that quorum as well then the third one is promoted. There are a number of different scenarios for rolling failures, but essentially as long as you have either two nodes in an iogroup, or one node and an active quorum then you survive OK. If you lose multiple power domains simultaneously you are a) a very very unlucky person; and b) the owner of an SVC iogroup that has just stopped processing and needs help from L3 support to get it running again.

    Like

  7. Just for completeness, starting with code level 7.6 a third physical site is no more required. There is an ‘IP-quorum’ option based on a java app that can run at any cloud provider. Plus up to four standy instances of the app to fight paranoia. Just don’t place them in DC1 or DC2. SVC will select which app is the primary IP-quorum using Paxos-like algorithms.

    Besides, since code level 6.3 the maximum supported HA distance of a stretched cluster is 300 km using inter-switch links on extended fabrics (or 40km without).

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: