XIV relies on lots of Intel cores and distributed caches to deliver performance. People who get hung up on disk systems being all about disk drives have trouble understanding XIV in so many ways, including power consumption.
If you’re going to compare XIV power consumption you need to be clear what you are comparing it to. XIV is positioned primarily against EMC VMAX-SE and HDS USP-V USP-VM. If you are comparing equivalent performance and capacity, you probably need to compare a XIV to a two-frame VMAX-SE and a two-frame USP-V USP-VM. The XIV has an immediate footprint advantage.
Whenever I have done comparisons, what I find is that no matter how you cut it, you will end up with something in the range of 7-10 KVA and 20-30 KBTU/Hr.
Most vendors’ marketing charts will show an advantage for themselves over others, and that is usually based on things like RAID selection, or comparing to an older system using smaller drives etc. Sure I have had customers tell me on occasion that we were much lower on power and cooling than another vendor, but I think that is generally sweet-spot dependent.
So my personal view on power and cooling for tier1 disk systems is that on average there is not much difference between vendors.
———————————————–//————————————————————
p.s. I have never been quite happy with this post on XIV power and cooling. Originally I intended to do a more quantitive analysis, but after several goes at it I gave up and settled for the unsatisfactory “not much difference” conclusion. A more accurate way of putting it would actually have been “it depends”, but that’s not much more satisfactory.
I feel similarly about price comparisons – they are two things that I think are best compared by customers in their specific situation.
I will try to stay away from areas like this in future and I apologise for the lack of usefulness of this original power & cooling post.
I feel better now.
Filed under: XIV |
We're all only here temporarily... 
[…] But let me get a look to the latest one: “Comparing XIV Power Consumption”. […]
LikeLike
Welcome to the blog-o-sphere.
I’d love to see the facts behind your assertions. Back in August 2008 I presented the facts about power consumption based on then-current platforms, and the XIV array was far and away the WORST when compared to other products.
And positioning the XIV as competing with V-Max and USP-V is delusional – IBM’s own XIV engineers now tell customers that for mission-critical, enterprise applications they should choose the DS8700, and not an XIV. Any array that will corrupt 79TB of user data because of a simultaneous two-drive failure in two separate drive bays is barely appropriate for home hobbyist use, much less “mission critical”.
LikeLike
Which assertion do you want a fact for? You are famous for your factoids. The delusional comment is pretty typical. The IBM XIV engineers comment is another piece of your spin – DS8700 and XIV both have their place.
LikeLike
Let’s just start with the actual power utilization requirements behind this assertion:
“Whenever I have done comparisons, what I find is that no matter how you cut it, you will end up with something in the range of 8-12 KVA and 26-37 KBTU/Hr, and XIV will fall into the bottom of that range. These numbers seem to be the same even if you include mid-range systems that probably can’t deliver the same real-world performance as theses three systems.”
And I never said XIV and DS8700 didn’t have a place – just that XIV enigneers have specifically told customers that XIV isn’t the best choice for mission-critical applications where absolute data integrity protection is more important than “cheap.”
LikeLike
Sorry, but if you tell:
“If you’re going to compare XIV power consumption you need to be clear what you are comparing it to. XIV is positioned primarily against EMC VMAX-SE and HDS USP-V. Not only that but if you are comparing equivalent performance and capacity, you need to compare a full XIV to a full VMAX-SE and a two-frame USP-V. The XIV has an immediate footprint advantage.”
1) You’re talking about USP-VM, not USP-V . This system can also virtualize external storage like SVC. From your point of view, is this comparison fair?
2) You’re saying that XIV is comparable in performance to VMAX-SE and USP-VM? Any performance data? In my calculations it’s near DS4700 from sustained I/O, but only calculations, not benchmarking. Can you provide with some performance information?
3) You talk about Cache:
a) 8 Gb is memory in each node, are you telling me it’s all cache? nothing of this is used by OS memory?
b) IBM use to say it is not the amount of cache, but the algorithm you use
4) Mid-range performance: again, sorry, but yesterday one customer mentions he tested AMS2500 (mid-range) & V-MAX (high-end) and AMS2500 outperforms V-MAX. Are you sure? Have any performance data?
LikeLike
You are right inasmuch as audited XIV performance data is very hard to come by. The approach that benchmarks aren’t important seems to be something that Moshe brought with him from EMC. I have seen it perform extremely well in the field and it seems to go better the greater the mix of workload you throw at it, which may support the ‘benchmarks don’t do it justice’ position. The lack of hard benchmark data is a point of frustration for us all.
LikeLike
Maybe the concern is because IBM use to release benchmarks and I think this gives some respect in market. Power gains it’s respect with benchmarks and opinions.
I think it’s some kind of EMC-like commercial strategy that I don’t really like, because as a technician I need some facts so I can recommend a customer which system to choose.
It’s like SUN servers policy: They said “V490 with sparc IV+ is faster than 550Q”. This is the only data you had, so you sized a 550Q (now 750 with Power7) , customer asked for a discount , you agreed and 6 months later you realized you’re system was 3 times faster than V490 comparative sizing (Fact, this really happened).
The same applies in the opposite way: If you say XIV is as fast as V-MAX SE, then you could over or under size, and customer could sue you, or you could loose respect from the market.
LikeLike
Interesting post. If you believe that tier 1 systems consume at least 8 KVA per rack, you are mistaken. Full rack of Xiotech ISE consumes 6.63 KVA, and can be as low as 5.0 KVA _running at full workload_ as per SPC-1/E. 6,000 SPC-1 IOPS for 363 watts, per element, according to published, audited report from SPC.
LikeLike
To be honest I don’t know much about Xiotech, but you are of course free to make any claims you want about the supremacy of your product.
LikeLike
Thanks, and I know you don’t know much about Xiotech; only people who truly understand the entire storage industry do. I make no claims, actually, because I have facts and audited, public performance data. There are no claims needed when you have facts. The ISE scored 6,000 IOPS for 363 watts according to SPC-1/E.
I look forward to seeing your hard facts and audited data. IBM has posted SPC results in the past for various models of the DS series; it is, shall we say, interesting that they have not for the XIV, not even SPC-2.
So before you post more about power consumption, publish your SPC-1/E results, please.
storageperformance.org
LikeLike
Those of us familiar with SPC-1 for many years now (and the recent 1/E energy extension) are well aware that the results are meaningless until we all get to see your SPC mandated Full Disclosure Report, which has been missing from your submission for more than six months.
The core of legitimacy of the SPC benchmarks rests on public availability of the Full Disclosure Reports.
Suggest you begin “winding down” the jaw flapping and finger wagging until you post your FDR.
LikeLike
Welcome to the wonderful world of marketing Jim, where if you can’t win it, you spin it. You’re doing fine.
LikeLike
I work for IBM but I hope I don’t feel the need to win everything or resort to conscious spin. Part of my beef is that the competitive nature of the blogosphere undermines intelligent exchange. One of the reasons for blogging about XIV is I have seen it work really well in the field and it seems to get unfair criticism out in the blogosphere. So I make no excuse for defending XIV in this environment. I could start blogging about the weaknesses in other people’s products, but I have no interest in that. I am sure most of them work fine. It would be nice if others acknowledged that XIV does also.
LikeLike
I think we’ll all cut you slack, so long as you stick to the facts.
‘Tis a bit hypocritical then to come out swinging in your second post with a baseless assertion that XIV arrays use significantly less power than competitive kit. I’d say that your assertion is not only abjectly false, it is a rather blatant example of “blogging about the weaknesses in other people’s products.”
On the other hand, you’ve already admitted many of the shortcomings that IBM refused to communicate about the product until the storage blogospehere collective exposed the realities to unsuspecting customers, so you can’t be all bad.
Good luck to you on your blogging mission!
LikeLike
See there you go again, making stuff up and attributing it to me.
LikeLike
Jim, forgive me, but you’re asking people to give you money for an XIV rather than give me money for a NetApp system. I owe it to those people that evaluate where to spend their money to give them — as far as I can — the facts about my product, and to help them understand your product as compared to mine.
Yes, it’s competitive, but that doesn’t stop intelligent exchange; as long as you have at least *some* evidence that the stuff you are saying stands up to scrutiny. If it doesn’t, you will and should be mercilessly examined.
This and your previous blog are both cases where you have been fast and loose with the evidence. Look carefully at this latest blog; there is nothing of substance in what you claim. No numbers, no examples, no justifications, no comparisons, no reasoning, just a stream of personal observations capped with “So my personal view on power and cooling…”
You want storage buyers to give you their hard cash on the basis of your personal view? Get real.
LikeLike
Some of what you say is true. That post on power and cooling was not my best work : ) I started out wanting to make it quantitive, but I found that the results could be whatever I wanted them to be depending on the assumptions I applied re performance and capacity. I should have walked away, but I was already committed, having promised to post on that topic. I will try to stay away from areas like this in future (pricing is another example) and leave it to customers to assess in their individual cases.
LikeLike
Jim,
comparing the XIV to the USP and V-Max? How?
Both those systems are highly scalable and functional, not to mention immensely reliable and proven in the real world.
You also don’t mention that IBM has internal docs showing 17K IOPS for the XIV yet it’s sold as a 100K IOPS system.
Even if we admit that using cache only you can get 100K IOPS, a USP or V-Max can go way higher with cache-only theoretical numbers (the USP Theoretical max cache-only performance is ridiculous).
Moshe may think benchmarks are unimportant but customers don’t.
Customers like to know what happens to the performance once they run out of cache, for example. Or should I say, informed customers.
There’s a multitude of systems out there that have published performance numbers using audited tests. All of them scale past 73TB base2 (let’s get real about the size as well).
NetApp systems with the large cache (can go to 4TB cache) have shown multiple results of superior power efficiency due to reduced disk counts. Other vendors can do the same with a combo of cache and SSD, for workloads that would destroy the cache of the XIV.
So, pardon everyone’s scepticism when you claim decent power efficiency.
Let’s look at the logic:
by design you cannot be more than about 44% efficient (right?)
Plus you need a UPS in every rack.
I can build you a NetApp system with less drives, tons of cache and less footprint in that same space. Or, use the full rack and provide much higher density and vailable space. If you do the math, the power/space requirements are less.
Same for most other vendors.
So – and this is what I’ve been struggling with since XIV was announced – exactly what is the point of this system beyond the amazingly nice GUI?
It’s not more reliable, it’s not faster than enterprise systems from IBM, NetApp, EMC and HDS, and it’s not more space- or power-efficient.
Maybe you should focus your posts on that ;)
D
LikeLike
Dimitris, actually I’ve found a document in XIV page where it says 20K disk IOPS and 100K cache IOPS:
Click to access taneja_brief_nov08.pdf
Talking about UPS, XIV comes with 3, don’t need to add one.
I think NetApp has great products, but FAS only uses RAID-DP, so I don’t think it’s a fair comparison in terms of power efficiency.
I think XIV is a different product, ease of use and GUI is just a part of it. It’s about self-healing, obviously with IBM engineering support, reducing engineer arrival risk window.
About creating a FASxxxx machine big enough to compete, why not N-Series gateway with 2 DS5020 in mirror? ;-)
LikeLike
And I have seen an internal IBM doc showing 17K IOPS and much more besides that. Anyway…
Datacenters have their own UPSes, typically high-end, therefore I don’t see the need to add the 3 XIV comes with that force you to waste even MORE space.
Sure, you could put N-series gateways in front of whatever IBM has (incl XIV) to get the extra features, but what does that give you vs buying the NetApp disk?
Gateways are typically for cases where you already own the storage and want to make it do something more.
And one of the major points of RAID-DP IS power and space efficiency so I think that’s absolutely a valid point to make, why do you want to minimize the strength of the argument?
If I can have speed and higher reliability than mirroring while wasting less space, why on earth not go for it??? That’s what RAID-DP offer in a mathematically and empirically unassailable fashion.
All decent arrays do self-healing anyway, again I fail to see the point here.
Rebuilding the blocks of a failed drive in XIV rapidly is an undeniable benefit but also something the XIV architecture needs anyway. And, BTW, 3Par can do something similar but at least they offer truly scalable and highly-performing boxes that will run rings around an XIV.
So – if I can get something from other vendors that provides the goodness of XIV plus higher scalability, reliability and proven high-end performance and space and power efficiency, I ask again: what’s the point of XIV, in its current incarnation?
D
LikeLike
Dimitris,
“Sure, you could put N-series gateways in front of whatever IBM has (incl XIV) to get the extra features, but what does that give you vs buying the NetApp disk?”
NetApp disks are SO expensive !!!! And FC performance is not the best, so I’d prefer a pass-through, that’s why.
“Gateways are typically for cases where you already own the storage and want to make it do something more. ”
From other vendors, not NetApp. Sorry, but with Hitachi’s USP-V/VM and with IBM’s SVC you can map existing LUNs, but with NetApp V-Series you need to format with WAFL. Of course, if customers agree to format their disks it works.
“If I can have speed and higher reliability than mirroring while wasting less space, why on earth not go for it??? That’s what RAID-DP offer in a mathematically and empirically unassailable fashion. ”
Always there’s an alternative, of course. Problem with traditional RAID is that you cannot “self-heal” (if that kind of thing exist in english), you just drop redundancy and then just wait for engineering to repair it. It’s a different approach with different implications.
XIV is a system with self-healing capabilities , 3 UPS so you get independence from DC (never get a power outage in a Datacenter?), fast and AUTOMATIC Rebuilds (yes other vendors claim same thing,perfect for them, but not all of them fire the rebuild event with no human intervention), more disks than many systems, etc.
I think the most disturbing points are no disk redundancy, but data redundancy, and disk IO performance range in Mid-Range arenas. There’s a cache workload preference, yes, but it doesn’t mean it’s a bad system.
LikeLike
Jus noticed that my previous reply went in as a new post ‘doh (John Martin)
LikeLike
HS. – NetApp disks are SO expensive !!!!
If you compare our capacy shipped by our revenue figures as published by the likes of IDC and compare them to IBM or EMC or pretty much anybody else, you’ll see that on a cost/raw GB basis, Netapp is the price leader.
Also even if we were 50% more on a cost/raw TB, we’d still be cheaper than XIV just by virtue of RAID-DP vs mirroring.
Finally on a cost/spindle IOP we’re generally better than the competitition (more on that later)
HS – And FC performance is not the best.
Again, look at the benchmarks, pretty much every one shows that on a IOPS/spindle basis we lead the industry. I’d be interested in knowing who you think gets more performance per spindle at an equivalent price point and whether you’ve got an audited benchmark to prove the point ? I’m not saying that there are workloads where SVC or USP virtualisation isnt going to be better from a performance point of view, but for most workloads, especially as they get more random and more write intensive, you’re better off with NetApp technology. e.g.
IBM SVC SPC-1 benchmark achieved 380489 IOPs with 2304 15K spindles – 165 IOPS/spindle
NetApp 3170 SPC-1 benchmark acheived 60,515 IOPS with 64 15k spindles – 270 IOPS/spindle
HS – with Hitachi’s USP-V/VM and with IBM’s SVC you can map existing LUNs
Good point, but the corresponding amount of value that is added is incremental – you get more cache, lun migration, mirroring, and arguably simplified managment. All of this is good, but considerably less than you get from v-series which offers fully unified storage, deduplication, thin provisioning, metrocluster, vfilers, and a number of other goodies. If you’re going to put a “bump in the wire”, you have to make sure the value you get out of it makes the exercise worthwhile. IMHO USP-V and SVC dont add sufficient extra value.
HS – 3 UPS so you get independence from DC – this is only useful if the networking and servers also have additional UPS protection. Its cool, but what’s the point ? Most small UPS’s I’m familiar with are power-hogs, the ones in the XIV might not be, but how about the other ones that would be required to get the benefit your talking about. if you’re asking a DC designer to put UPS’s in front of all the other critical kit, he’ll probably do this in a centralised way in any case.
HS – Problem with traditional RAID is that you cannot “self-heal” … you just drop redundancy …
Um … I’d love to see some elaboration on that point, because if you’re using up “hot spare” capacity then that aint so, unless you think that using a dedicated hot spare disk somehow reduces redundancy more than using distributed hot spare capacity
HS – but not all of them fire the rebuild event with no human intervention
Really ? Who ? I’d love to see some hard references, I didnt think there was a single reputable vendor who didnt automatically rebuild raid sets from hot spares.
John Martin
Consuting Systems Engineer
ANZ
LikeLike
John Martin…wow…what a load!
re:
“look at the benchmarks, pretty much every one shows that on a IOPS/spindle basis we (Netapp) lead the industry.”
Not even close.
First off, you are way off in left field comparing a 3170 to a 2,000 spindle system.
Secondly, the SPC-1 on the Netapp box used 244 spindles, not 64.
Thirdly, there are are at least a half dozen systems (starting with the DS5000) that beat the 3170 on >>all three<>79TB<< (usable) XIV.
Go check your math, dude!
Ok now…lastly…you said:
"but for most workloads, especially as they get more random and more write intensive, you’re better off with NetApp technology."
John, have you looked at your own SPC results? Your 14ms response time for writes to cache is among the worst reported on an SPC-1. Don't you remember? All flavors of parity-based RAID suffer from a small block write penalty — especially Netapp's. 'DP'. Gotta say, this is the first time I've ever heard anyone assert that Netapp is good at random writes! Even their marketing people don't spout such nonsense.
Been drinking a little too much of that Kool Aid???
LikeLike
Just pointing out the IOPS/spindle measure, you’re right of courseee that the 3170 had 224 spindles not 64 (not sure how that happened), but the IOPS/spindle count of 270 still works, so I stand by the assertion that based on that measure, our IOPS/spindle efficiency is still better than the most of the other results.
I’m not entirly clear on what you mean by
“on >>all three79TB<< (usable) XIV.",
As far as I can tell there is no XIV SPC-1 benchmark, so I assume this is a cut and paste glitch/typo like my "64 spindles" one above
In any case I did a quick review of the the DS5000 resuls and came up with the following.
DS5300 gives you 227 IOPS/Spindle
DS5300 with FDE gives you 243 IOPS/Spindle (better result with FDE … go figure)
DS5020 gives you 326 – A truly excellent result.
So I'll grant you that the 5020 came up with something surprisingly (almost unbelievably) good which bears more investigation, but most of the reviews I've done on various benchmarks shows that on an IOPS/Spindle basis NetApp leads the industry.
As far as your explanation for the write latency, you're a little off. This isnt the place for a tutorial on how WAFL optimises random writes (basically we turn them into sequential writes, eliminating the parity issue), but small block random writes really arent an issue. I'll blog about this some other time, or feel free to send me an email and I'll point you to some excellent material that's already been written on this subject.
As far as the latency goes, I've always been surprised at why most other benchmarks seem to stop before the systems are really pushed hard. I suspect that littles law kicks in and theres a scary looking vertical spike in latency after a particular threshold is reached. In any case we designed the solution to sit under the 30ms threshold which it did. If we'd decided to throw in a bunch more spindles and had the same RAW:Usable ratio as the arrays which use RAID-10 (which would have given the write allocator more room to do optimisation), the latency graph probably would have looked different, but we didnt so its a moot point.
As far as the NetApp Kool-Aid goes, its refreshing and delicious, but not nearly as funky as the stuff they guys at http://www.thirdio.com seem to have on tap, you gotta love a swirly mandala flash image on a storage vendors site :-)
LikeLike