Prior to joining the SPC, EMC were vocal about such benchmarks being based on artificial workloads that do not accurately match any one customers profile, so what’s the point! I see where they’re coming from, but don’t personally feel such a fact renders such benchmarking entirely useless. Here’s why –
In my experience, many large enterprises have their own in-house simulated workloads that they use to bench storage arrays. Even these customer-created benchmarks don’t exactly mimic their respective production environments. However, they do provide a common baseline against which multiple arrays can be measured. Therefore, they are of some value. Ditto for benchmarks such as the various SPC and SPECsfs benchmarks. They’re not perfect, but they’re not useless either.
In my opinion, arrays like VMAX, VSP, DS8, 3PAR, XIV, CLARiiON… seem very often to find themselves on the receiving end of just about every I/O profile conceived by man. Such arrays are often deployed to meet ~80% of an organisations block requirements. The other 20% often finding its way to more niche architectures such as TMS RamSan, Isilon, FusionIO, Exadata… Would love to know if you disagree.
I’d be willing to bet that most of the above mentioned arrays tackle a mix ‘n’ match of all things structured and semi-structured (MS Exchange, Oracle, SQL Server, Siebel, SharePoint etc..) on a daily basis at countless organisations across the planet.
If I am right, then I postulate that just about any artificial workload can form the basis of a half-decent benchmarking workload – If an array going to get a bit of everything in production (and that bit of everything today may potentially be different to the bit of everything tomorrow) then why not give it a bit of everything in test – right?
With the above in mind, the problem with SPC, SPECsfs, and any other benchmark for that matter, lies not so much the artificial nature of the workload, as it does in the
hilarious ridiculous configurations the vendors submit! One only needs to look at the VNX configuration EMC submitted for its February 2011 NFS SPECafs2008 benchmark – 4 x somewhat combined VNX5700 with over 400 SSD drives! Amazing results and shows the capabilities of the architecture at one extreme, but pleeeaaze!
Storage Formula 1
Much like Formula 1 racing, where the Renault car and engine that is submitted for the race bears absolutely zero resemblance to the Renault that my wife drives, the configurations submitted by the vendors bear zero resemblance to the configurations they sell to customers. One configuration is for pure sport, whereas the other is for customers to buy.
However, a vital difference between Formula 1 racing and storage vendor benchmarking is that one of the two is interesting and exciting, whereas the other is…… well ….. not. I’ll leave it to you to decide which is which 😉
So…….. good on EMC For joining and starting to play ball. But howz about having some balls and submitting realistic configurations that your customers regularly buy. While the same goes for everyone else, I’d expect the 800lbs Gorilla to have the biggest balls, hence why I call out to EMC to set the tone – even the top EMC marketing folks agree that things need to be changed. At least that way, it will be of some value to your customers and not just be a stage for donning your Speedos and flexing your steroid induced IOPs and latencies.
UPDATE 29/02/2011 – If vendors feel they absolutely must submit ridiculous configurations to the likes of SPC and SEPCsfs in order to acheive large numbers, why not also submit realistic every day configurations as well? That way you get to brag about your big numbers, but also provide customers with something useful. Obviously, if you're not confident about your realistic configurations and have something to hide, then we understand why you don't do this 😉 See here for where I got the idea.
On a different but somewhat similar note, HDS recently announced that its flagship product, the Hitachi VSP, had received a supposedly prestigious iF Product Design Award. Personally I’d never heard of the award, but don’t let that detract from it.
According to Hu Yoshida, CTO of HDS, the award takes in to consideration things such as “… environmental sustainability, architectural design quality, and overall functionality”.
All well and good, but was the VSP up against any of its peers? All well and good if it was, but If not, then what is the real value?
When it comes to meaningful benchmarking and valuable design awards, we are so close, yet so far.
Just my thoughts, appreciate any comments.