IBM XIV Gen3 and SPC-1

IBM has just published an SPC-1 benchmark result for XIV. The magic number is 180,020 low latency IOPS in a single rack. This part of my blog post was delayed by my waiting for the official SPC-1 published document so I could focus in on an aspect of SPC-1 that I find particularly interesting.

XIV has always been a work horse rather than a race horse, being fast enough, and beating other traditional systems by never going out of tune, but 180,020 is still a lot of IOPS in a single rack.

SPC-1 has been criticised occasionally as being a drive-centric benchmark, but it’s actually more true to observe that many modern disk systems are drive-centric (XIV is obviously not one of those). Things do change and there was a time in the early 2000’s when, as I recall, most disk systems were controller-bound, and as systems continue to evolve I would expect SPC-1 to continue to expose some architectural quirks, and some vendors will continue to avoid SPC-1 so that their quirks are not exposed.

For example, as some vendors try to scale their architectures, keeping latency low becomes a challenge, and SPC-1 reports give us a lot more detail than just the topline IOPS number if we care to look.

The SPC-1 rules allow average response times up to 30 milliseconds, but generally I would plan real-world solutions around an upper limit of 10 milliseconds average, and for tier1 systems you might sometimes even want to design for 5 milliseconds.

I find read latency interesting because not only does SPC-1 allow for a lot of variance, but different architectures do seem to give very different results. Write latency on the other hand seems to stay universally low right up until the end. Let’s use the SPC-1 reports to look at how some of these systems stack up to my 5 millisecond average read latency test:

DS8870 – this is my baseline as a low-latency, high-performance system

  • 1,536 x 15KRPM drives RAID10 in four frames
  • 451,000 SPC-1 IOPS
  • Read latency hits 4.27 milliseconds at 361,000 IOPS

HP 3PAR V800

  • 1,920 x 15KRPM drives RAID10 in seven frames [sorry for reporting this initially as 3,840 – I was counting the drives and also the drive support package for the same number of drives]
  • 450,000 SPC-1 IOPS
  • Average read latency hits 4.23 millsconds at only 45,000 IOPS

Pausing for a moment to compare DS8870 with 3PAR V800 you’d have to say DS8870 is clearly in a different class when it comes to read latency.

Hitachi VSP

  • 1,152 x 15KRPM drives RAID10 in four frames
  • 270,000 SPC-1 IOPS
  • Average read latency hits 3.76 ms at only 27,000 IOPS and is well above 5 ms at 135,000

Hitachi HUS-VM

  • 608 x 15KRPM drives RAID10 in two frames
  • 181,000 SPC-1 IOPS
  • Average read latency hits 3.72 ms at only 91,000 IOPS and is above 5 ms at 145,000

Netapp FAS3270A

  • 2 x 512GB Flash Cache
  • 120 x 15KRPM drives RAID-DP in a single frame
  • 68,034 SPC-1 IOPS
  • Average read latency hits 2.73 ms at 34,000 IOPS and is well over 6 ms at 54,000

So how does XIV stack up?

  • 15 x 400GB Flash Cache
  • 180 x 7200RPM drives RAID-X in a single frame
  • 180,020 SPC-1 IOPS
  • Average read latency hits 4.08 millseconds at 144,000 IOPS

And while I know that there are many ways to analyse and measure the value of things, it is interesting that the two large IBM disk systems seem to be the only ones that can keep read latency down below 5 ms when they are heavily loaded.

[SPC-1 capacity data removed on 130612 as it wasn’t adding anything, just clutter]

Update 130617: I have just found another comment from HP in my spam filter, pointing out that the DS8870 had 1,536 drives not 1,296. I will have to remember not to write in a such a rush next time. This post was really just an add-on to the more important  first half of the post on the new XIV features, and was intended to celebrate the long-awaited SPC-1 result from the XIV team.

Advertisements

9 Responses

  1. Hey Jim,

    Let me correct an error in your summary above, and add some factual color. Disclosure upfront: I work for HP Storage.

    The HP 3PAR V800 result you reference actually had 1,920 HDD; not the 3,840 you mention. Where did you get 3,840 from?

    The DS8870, and also the new XIV results *look* good for 2 simple reasons;
    – borderline short-stroking – large unused storage ratio; and
    – high system cache to application data ratio (ASU GB)

    DS8870 – ASU GB to cache ratio is 55 : 1
    XIV – ASU GB to cache ratio is 7:1
    3PAR V800 – ASU GB to cache ratio is 300 : 1

    Its easy to get high read rates and low latency with lots of cache and not much application data!! – Which all increases cost…

    DS8870 – $/ASU GB = $86.69
    XIV – $/ASU GB = $22
    HP 3PAR V800 – $/ASU GB = $12.87

    So; yes, good read latency is important – but so is efficient utilisation of resources. If a system cannot utilise the resources the customer has purchased – then that’s not a good investment.

    If latency is important and for the $976,071 the XIV config will cost me – I could get 6 HP 3PAR 7400 with SSD and get over 1.5 million SPC-1 IOPS at less than 1ms !!! – now that’s a good investment! :)

    regards,

    Paul Haverfield
    Storage Principal Technologist, HP Storage APJ.

    Like

    • Sorry about the drive count error – I was adding in the count I saw on the drive support line item as well, and needing 7 frames seemed to reinforce that count. Now corrected. Thanks for picking that up.
      As for the cache and short-stroking etc, while I agree with your facts, I’m not sure your protest is entirely valid. In my experience, every vendor tweaks the hell out of their config (within the bounds of the rules) and if they can make a substantial difference by adding more cache, or more free space I think it would be naive to assume they wouldn’t do that.

      Like

  2. What a refreshingly civil discourse on an SPC benchmark result!

    You’re right: it is totally reasonable for vendors to adjust their configurations to get the best results. But, as i read Paul’s comments above, I don’t think he was claiming foul. He was simply doing what the creators of the SPC benchmark intended for observers to do, i.e., to look at not just the performance numbers but at the pricing and configuration the vendors used to produce those numbers, and then determine which array would produce the best result in their environment.

    IBM had an objective with the XIV – to get the highest number and lowest latency possible – possibly because of all the flack the XIV has gotten over the years for not being particularly fast. And IBM achieved a very good result by bumping up against the SPC’s short stroking bar, and by managing to fit a high percentage of their data in cache – an unlikely scenario for most customers. On the other hand, HP filled the 3PAR up to its max number of disk drives, with a very high capacity utilization, basically saying ‘here’s our array.’

    I’m not sure what customers could derive from the XIV SPC results that would help them size their environment or make an informed storage choice. But the 3PAR result would be very helpful. Disclosure: I’m an HP Storage employee.

    Like

    • Again I think that is a bit naive. If the HP engineers running the benchmark were competent then they tweaked their box to give the best possible result. Personally I always halve any given SPC-1 result to arrive at an estimate of how things might perform in real life, and I would extend the same respect to the benchmark tuners from HP, who I assume are all very good at what they do.

      Like

  3. […] my last post I looked at the way some systems respond to the SPC-1 benchmark, often hitting 5 milliseconds read […]

    Like

  4. The XIV benchmark was not comparable to any of the others — it is not H/A (only meets the criteria for what SPC-calls “Protected 1”., which means a single-point hardware failure can take the system down.) This is not at all comparable to the other benchmark results which were all H/A. Cache coherency costs a lot in terms of performance — so when IBM “fudges” the benchmark like this, it means that what they’ve done is meaningless…

    Dunno why IBM keeps doing this — seems they are the only ones. They did exactly the same thing on their Power System SPC-1, and at least six others I recall…

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: