XIV & MS Exch 2007 Replicated Performance

Vendors typically only benchmark their fastest systems in any one class, which means that a bit of careful reflection is required to get a good understanding of things from a few results. The usual “nyah nyah ours is faster” kind of analysis and comment that seems to permeate the blogosphere doesn’t really achieve anything that’s for sure.

Let’s talk about benchmarking more generally…

1. Bragging Rights

There has been a lot of focus on storage product performance in recent years and now the enterprise storage benchmarks generally outstrip the performance that most customers require. I always saw benchmarks as being about transparency, but to many people they are about bragging rights. If the XIV motto is “fast enough” and benchmarking is about bragging rights, then I guess I can understand why benchmarking isn’t always a good fit.

In 2008 when Netapp benchmarked an EMC CX3-40 and published the SPC-1 result, they showed why EMC had avoided SPC-1. I guess benchmarks haven’t traditionally been considered good marketing unless you can somehow claim to be number 1 in a segment, or at least stick one to a key competitor.

2. Badly Balanced Systems

No vendor ever benchmarks a badly balanced volume layout yet most customers run just such systems because it’s just too hard to keep them balanced. What we see in the field is that when apps are migrated to XIV they go faster than when they were on competing systems. Some of that, but not all, must be better load balancing, because it happens out of proportion to any benchmark, and seems to be more noticeable the more jumbled up the workload is.

3. Breakage, Recovery & Other Admin Tasks

Benchmarks don’t test scenarios like what happens to performance when a controller fails or a drive is rebuilding. Netapp’s SPC-1 benchmarks in 2008 tried to show the difference in performance when snapshots were running, and highlighted the difference in efficiency between Redirect-on-write (Netapp & XIV) and copy-on-write (most others).

So while I still like benchmarks, I do have to concede that they are a bit of a blunt instrument.

The only benchmark that has a decent smattering of results including XIV is a Microsoft Exchange 2007 ESRP. What follows is my summary of these results. It’s worth noting that these numbers are way lower than most vendors claim for generic IOPs, way lower than equivalent SPC-1 numbers, and way lower than the real-life results most customers see in the field.

Worth noting when reading these is that vendors often stop adding drives when the bottleneck moves to someplace else on their system.

Update: Also worth noting is that vendors usually benchmark with end to end RAID10, but seldom install with end to end RAID10 – I wonder what the benchmark results would be if they used RAID5/6 like they do in real life? Perhaps only the XIV and Netapp results are realistic in that respect.

  • DS8300 33,600 IOPS with replication. 640 drives RAID10
  • DMX4-4500 28,800 IOPSwith replication. 480 drives RAID10
  • DS5300 26,400 IOPswith replication. 252 drives RAID10
  • FAS3170 23,000 estimated IOPs (28,800 w/o repl). 184 drives RAID6
  • CX4-480 21,000 IOPS with replication. 186 drives RAID10
  • SVC Entry Edition 2 nodes w/3xDS3400 16,128 IOPS (20,160 w/o repl). 144 drives RAID10
  • XIV 16,000 IOPS w/repl & 15 nodes each with 8GB cache & 12 drives RAID10
  • DS5020 11,200 est. IOPs (14,000 actual w/o repl). 112 drives RAID10
  • FAS2050 3,200 est. IOPs (4,000 actual w/o repl). 44 drives RAID6
  • EVA4400 2,460 est. IOPs (3,072 actual w/o repl). 46 drives RAID10
  • Sun 7310 1,536 est. IOPs (1,920 w/o repl). 7xSSD/66xSATA RAID10
Advertisements

18 Responses

  1. This is REALLY interesting. I think that this info combined with some statistic data from DDF are good points to defend XIV product.

    Like

  2. So, you’re basically saying that an XIV with 180 drives (I love how you did not specified it saying 15 nodes :) is worth 16000 IOPS and a NetApp FAS3170 with only 4 drives more almost double that number.

    Considering the fact that IBM position the XIV near DS8x00 as Enterprise Storage, meanwhile NetApp use the 6000 series to compete against DS8x00 is perfectly clear that XIV is NOT an enterprise storage solution.

    Or am I missing the point here?

    Like

    • Fabio, I think you missed the point :-)
      I think you compare to Cache IOPS. This values are even worst than disk IOPs, than in NETAPP 3170 with RAID-DP and 4 disks would round 350 IOPS.

      Then, you said that a big NAS with steroids (sorry N-Series from IBM) is compared to DS8K ? It’s a good system, agree, but a big midrange (NetApp uses same OS and similar features in all systems) is not like a high end, that’s why most financial services use DS8K, DMX or USP.

      Again,I agree with you, numbers don’t look well, it’s below DS5300, a mid-range system.

      Like

  3. Fabio, I’m not sure where you get from 16000 IOPS with replication to 23000 IOPS with replication by doubling, unless I’m missing something? :)

    Either way, the key piece of information that is missing is a cost comparison, I know roughly how much I’d pay for an XIV but not for the tested FAS3170 with the various software licences. If the FAS3170 comes in at a comparable cost over 3-5 years to the XIV, then it’s definitely a positive sign for NetApp.

    Like

  4. @Hector
    a FAS6080 is a big and mean machine, I can assure you that a LOT of financial institutions use it in production on mission critical systems, actually they use it for everything except for z/OS systems that are tied to the old TLA vendors.

    @Ewan
    Ok, my math was a bit optimistic, but we’re talking almost 50% IOPS increment with a machine that’s considered midrange.

    Like

    • Just be careful assuming that the FAS6000 (IBM N7000) products are a lot faster than the FAS31xx (N6000) range. We generally look at the larger N machines for their increased I/O slot count rather than their increased performance.

      Like

      • I’m surprised that you don’t know your own (OEM) products: the FAS6000 line, apart from doubling the IO slots, also doubles the core count and RAM (16 core and 64GB against 8 core and 32GB).
        Take a look at the spec sheets.

        Like

        • I’m familiar with the core counts and RAM. What the spec sheet may not mention is that the result of doubling the core count is dependent on ONTAP’s ability to scale. I expect scalability will improve with time, perhaps with ONTAP8. Also FAS6000 uses NV6 while FAS31xx uses NV7 (onboard memory controller).

          Like

  5. This is the type of information I like to see when designing a solution. I also think that you have presented an, if you will pardon the inevitable pun, ‘enlightened’ view of benchmarks in general.

    Vendor provided performance metrics tend to represent the maximum theoretical performance of the device. They also bear little resemblance to real-world performance.

    “Standard benchmarks” like SPC, at least make a token effort to represent real-world conditions, but the hardware configurations used to achieve the published numbers are often well outside what is commonly implemented. As you have pointed out, these benchmarks are more often used for bragging rights than for any practical effect. While better than vendor provided metrics, these benchmarks should still be taken with a stiff dose of salt.

    The next logical points of comparison would be cost of acquisition and cost of ownership. Once the technical teams have determined that a given device will meet their requirements, the final decision tends to be made on financial grounds.

    Like

  6. Even though NetApp has a higher ESRP result than XIV I want to emphasize that ESRP is not a benchmark and should not be used as such.

    It’s an example of a validated config for a certain # mailboxes but it’s not as if the boxes are highly tuned to run it (I’ve seen the NetApp settings for ESRP and they’re far from tuned, I could increase the number substantially with a bit of tuning).

    Look for real customers running real Exchange workloads – NetApp has one with over 1.3 million users, for instance…

    And, last but not least:

    Running exchange is nice and can be done on almost any system, properly configured.

    What about doing stuff like application-aware backups and restores, restoring individual emails, application-aware replication?

    I wonder, does the XIV do any of that?

    Start thinking outside just the “I can provision a LUN for you” box. That’s the easy part :)

    D

    Like

    • Fine box that I’m sure it is, reality is that a FAS3170 doesn’t really do app aware stuff – that’s done by a piece of software called SnapManager running on a Windows Server. In XIV land it’s done by an app running on a Windows server also, only we call it Tivoli rather than make out it’s part of the storage box. Check out Tivoli Storage FlashCopy Manager for example. http://www-01.ibm.com/software/tivoli/products/storage-flashcopy-mgr/

      Like

      • And flashcopy manager is licensed per server? What if you have 50 Exchange servers?

        Does it also control app-consistent replication to another array or just local snaps? :)

        Snapmanager products are effectively the handshake between the NetApp back-end and the app and are licensed once per controller to enable them.

        They control the snaps and clones.

        They actually provide the pretty GUI and just talk to the SnapDrive agent already installed in the client systems, and that’s what takes the VSS-aware snaps. That client is NetApp-only and is as part of the solution as the NaviCLI agent would be on an EMC box. It’s not add-on software in the same sense as the Tivoli Flashcopy Manager is.

        Since you work at IBM and resell our stuff :) you can try those out easily, look at SMVI, RCU (free), SME, SMSQL, SMO, SMSAP, SMOSS…

        D

        Like

  7. Most customers do not run the workloads defined in the benchmarks. Most customers also run in unbalanced systems and even if ESRP does provide some data on best practice for x number of mailboxes, the customer rarely gets that config. XIV is unique where data is always balanced and providing application test results (like ESRP) does actually reflect what customers experience in their data centers.

    By the way, XIV was the first to publish ESRP for Exchange 2010 due to no need to try and figure out what config to run (system is optimized by nature), where to put the logs, etc.

    If you’re looking into the future of where MS is going with Exchange and how well it plays with XIV go to Exchange 2010 resutls :
    http://technet.microsoft.com/en-us/exchange/ff182054.aspx

    Like

  8. Wow, I’m stunned.

    Do any of the commenters here actually work in the storage industry?

    With 7,200 RPM drives, the XIV here is doing about 88 IOPS per disk. That’s amazing.

    Most of the competition here is struggling to get to 100 IOPS per disk — with 15,000 RPM disks!!!. Many are struggling to hit 60 IOPS per 15K spindle!

    Read again — 88 IOPS per spindle using 7,200 RPM spindles,

    If you’re jaw hasn’t hit the floor yet, you’re either not reading or you’re not comprehending.

    Like

    • @CMG – if you want to see a benchmark which blends cache and disk IOPS – more like real world – go look at http://www.spec.org/sfs2008/results/, where some get iops numbers that are still very close to what the raw disk will provide, and others many times more than that – I recollect seeing numbers approaching 1000 IOPS per disk – that gives a better idea of what an array can do with a given load

      so NO 88 IOPS for a SATA disk array is NOT amazing – it should be a lot better

      Like

  9. @CMG

    If you’re stunned by 88 IOPS for a SATA disk there’s probably something wrong with your perception. It’s not a DROBO.

    If you found a storage array struggling to do 60 IOPS from a 15K spindle there’s something wrong with its configuration, can you name one?

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: