XtremIO Craps on EMC Badge

By | September 18, 2014

Let me say… as a person who’s been involved with storage for a long time, I did not enjoy writing this. But it had to be written.

Right. This is a rant. But let me tell you…. it’s worth a read.

I can’t quite believe I’m writing this. I mean the title just doesn’t make sense…. EMC are famed in the storage industry for their enterprise class products. Surely they, of all people, wouldn’t ship a tier 1 product that’s such a mess! But they have.

There’s just no doubting that EMC have shipped a tier 1 storage array (XtremIO) with code that was a long long way off being ready for the enterprise.

The Problem

The big issue at the moment is that existing EMC XtremIO customers are staring down the barrel of a destructive firmware upgrade (code version 2.4 –> 3.x). Yes destructive! Meaning if you leave your data on the system while EMC perform the upgrade, you get a free data wipe as part of the service!

Clearly not what you want for your data. So the alternative is to evacuate all of your apps and data off the XtremIO, upgrade the firmware, and then copy all of your data and apps back. And if you just have your techie hat on and say “well….. there are products and services to do that”…. yes I agree. But put your business hat on for a second…… You’ve just deployed your mission critical business app(s) on the XtremIO, and now you have to go to the app owner (these guys all already think the IT department costs too much money and is a waste of space) and ask him/her for an outage window. Have fun with that!

And if you say “well….. mission critical Tier 1 apps would surely have in-built HA, just take one half down, upgrade it, and then upgrade the other half….”. Fair enough. But that HA architecture is there to protect the app. Asking to take half of it down – for a lot of hours – is gonna result in some strong language from your app owner. Let’s face it, an upgrade like this is huge co-ordinated effort, potentially involving – storage, DBAs, OS guys, app owners, the business, business partners, and EMC. That’s a sizeable piece of work, and in many cases will require a Project Manager. And it’s almost certainly a piece of work that you’ll kick off at 11:01pm on Friday night with the whole of Saturday as the Change Window, and all of Sunday as contingency. And don’t forget the testing once its all back up on the new code.

And if you’ve got HA, do run on the new code one one array for a week, and then repeat everything again the following weekend for the other array? Or do you risk doing it all in one weekend – I really miss decisions like those.

Oh, did we mention it’s a major rewrite. So that automatically triggers huge app testing, huge risk, and huge scepticism form the business – on a platform you’ve only just convinced them to migrate to. And in their opinion, anything that even remotely goes wrong after the upgrade is the fault of the XtremIO. Nightmare!

The Why

Now the issue behind all of this is that EMC were desperate to bring a product to market in the AFA space. They couldn’t risk others gaining any market share on them. So, they took a half-baked product, employed their best-of-breed sales force, and relied on the respected EMC badge to lure customers. And shame on them. In the name of market share they have crapped on the EMC badge and kicked their customers in the crotch.

I challenge anyone to tell me it isn’t so!

The Smoke and Mirrors Response

And I don’t buy any of this rhetoric from Chad Sakac. This is pure smoke and mirrors. Chad goes on about not listening to “people who go negative“, but instead listen to people “who disclose warts“. Well…I could be wrong on this one…. but…. I believe the wart was disclosed by a customer. Good on EMC for owning up once it was brought to light (not that they had a choice). But I honestly don’t think they disclosed the wart in the way he is insinuating. Could be wrong on that one.

Also, all of Chad’s talk about disruptive change being required to implement new features and the likes. Sorry, but that disruptive change was buying and implementing the XtremIO! Not two subsequent disruptive code upgrades!

And then saying “if customers want to sit tight and avoid the upgrade, they are entitled to do exactly that – they will continue to enjoy all the XtremIO awesome they are loving“. What? As soon as the guy next door has the same product as you, but his/hers goes twice as fast and stores more than twice as much data as yours (coz he/she is running 3.0 of the code and your stuck on 2.4)… well…. all of a sudden what you’ve got is anything but awesome. If that wasn’t true then we’d be calling all reliable old and outdated technology “awesome” – but we don’t , we call it “old and outdated”.

Why for some EMCers is everything so facemeltingly awesome? Reminds me of the story of Peter and the Wolf…. when everything is always facemeltingly awesome, we end up not believing them when something actually is.

And comparing this situation to Data ONTAP 7-mode and c-mode. That’s comparing apples and oranges if I’ve ever seen it. ONTAP c-mode was like 10,000 years in the making and was signalled way ahead of time. In fact 7-mode and c-mode are pretty much two different products, and NetApp ran them in parallel and continued to feature develop both for a long time. And early on, 7-mode was more feature rich than the newer c-mode. And I’d punt that most NTAP customers moved to c-mode with new HW. None of that is true with XtremIO, none of it. It’s a pointless comparison.

Changes like this should simply not be happening on a brand new product, that is of course, unless that product is pre-GA.

It Gets Worse

What do people who are currently thinking of buying XtremIO do?

On the one hand you don’t want to be running mission critical Tier 1 on a .0 release of code that has more critical rewrites than most .0 releases! But on the other hand you don’t want to deploy on 2.4 and be planning a major upgrade project before the PO’s out of the door. We call that a rock and a hard place!

But the rock gets even bigger and harder………

XtremIO on code 2.4 is stable and fast (but lacking some fundamental “enterprise” capabilities). But 3.x is a major rewrite. I think Chad said the on-disk layer and the metadata layer were both getting major overhauls. So what’s to say it even works the same after the upgrade? May be one of the reasons it was so reliable and fast is because it didn’t really do anything clever, or was dependant on the old design. With 3.x of the code we’ve a got a big re-write of critical components, and we’re feature adding. Suddenly it doesn’t even look like the same product any more – remember, it’s all about the software these days – and the XtremIO software is changing changing changing!

The Huge Shame

Chad mentions in his blog that XtremIO is the “fastest growing revenue storage product EVER“. Fair play. This is thanks to the genuinely market leading EMC sales, and the genuinely excellent EMC badge. It’s just a shame it’s that same badge that this product has just crapped all over!

I know the storage world is changing, but EMC shipping product like this!!!!! Please no!

Don’t forget to check out the new In Tech We Trust Podcast over at http://intechwetrustpodcast.com/

30 thoughts on “XtremIO Craps on EMC Badge

  1. pannu

    true true true..I am glad we are just in POC mode and we broke the Array like 2 times already

  2. Joe

    Is It this worth your time?? Is your time that worthless? Or do you really get paid to vomit your frustrations on others?

    Anyhow, a very subjective point of ones product with a narrow minded thinking and perspective.

    Keep writing and you’ll succeed in life.

  3. John Martin

    – NetApp Employee Expressing Personal Opinions –

    “ONTAP c-mode was like 10,000 years in the making” … it certainly felt like it, now that the feature parity stuff is all done, it will be nice to move back to innovation on new foundations. For what it’s worth the official nomenclature is “Clustered Data ONTAP” (aka cDOT), there are no “Modes” (that nomenclature came from a time when we planned to run both 7G vFilers and cDOT vServers on the same controllers at the same time)

    “I’d punt that most NTAP customers moved to c-mode with new HW.”,

    that’s a very safe bet, in fact, I understand that’s what most of NetApp’s large customers wanted. When NetApp talked to our customer base about how they wanted us to get onto cDOT, they told NetApp that they felt their ONTAP 7G deployments were mission critical, and they couldn’t run the risk that the change to cDOT was a one way street. I heard it was also those large customers who told us that the “running both at the same time” was something they didn’t feel comfortable to put into production, unless there was a very easy, and rapid way of rolling back.

    Those customers were planning on moving to cDOT when they refreshed or purchased new hardware. That is the underlying rationale for the current migration based of moving workloads to cDOT. Doing a non-disruptive onbox upgrade to cDOT (even in a mixed mode) was actually a reasonably straightforward engineering project, what made it hard was that those large customers also expected a non-disruptive revert in case they decided they didn’t like the change after it went into production (which has been possible with ONTAP 7G upgrades for a while now). Providing a non-disruptive revert that was guaranteed to work required a really large investment for a feature that most of those large clients said they probably wouldn’t use.

    It’s worth noting that a core design principal of cDOT is 100% non-disruptive upgrades along with the ability to non-disruptively revert if the customer identifies anything they don’t like in the new version, clusters can even run multiple software versions at the same time for extended periods of time, which minimises upgrade risks even further. We think non-disruptive on-box upgrades are really important, and cDOT is better at doing that than any other storage product I’m aware of.

    I personally feel that Having Chad S. comparing the current XtremeIO situation with 7G-cDOT migration is more than a little unfair, but regardless of the reasons I’ve outlined above, I think pretty much everyone in NetApp wants the transition to be smoother than it is today. We’ve had non disruptive software upgrades and reverts in ONTAP 7G for years now, and it causes us a lot of internal angst that our customers have to _any_ disruption to get to cDOT, though that too will improve (cant say more unfortunately outside of NDA until after our Insight event in Las Vegas at the end of October).

    I get why EMC is doing disruptive upgrades, designing storage platforms with rich data services is a lot harder than it looks, and to get rapid innovation sometimes you have to trade off stuff to get your time to market. Having said that I’m honestly surprised that anyone would design a new shared storage product that didn’t include non-disruptive on-box upgrades, to me that still seems like table-stakes for an enterprise class product.

    I actually feel a lot of compassion for Chad S, right now, I’d hate to be turning that sows ear of a situation into a silk purse.

  4. Mark Kulacz

    No XtremIO code was written until 2009. No paying customers until 2011 (or was it 2012?). It takes many years to build enterprise storage. ONTAP/FAS, Symmetrix, 3PAR, etc.. the big systems have taken years (decades) to get to where they are. People think Isilon is some fancy new product but it is 13+ years in the going now.

    You cannot just hack code into these systems. This is infrastructure, not an iPad app.

    Either XtremIO just happens to have the most capable storage SW engineers in the entire world (and the rest of em at all companies, including NetApp and HP and IBM and (what the hay lets put Pure in there too even though Vaughn wont follow me on Twitter) … etc. … and including my old friends at EMC, are just a bunch of uncreative lazy stiffs), or perhaps XtremIO is just an XtremAlpha product that really needed another 2-3 years in the oven. The choice was made – Oversell XtremIO as far more complete than it was, to defend lost VMAX revenue caused by the explosion of all-flash systems.

    The frustrations and disappointment here is not against the engineers of XtremIO, but very specifically against the marketing of XtremIO, and in the situation that the EMC executive team must have placed product management into. It is an embarrassment. Running support for VDI at a trade show on an XtremIO proves nothing – You can get a killer workload on a re-purposed generic server with SSDs and FreeNAS and some 10Gb ports. But #AskWahatElse ? And even right now, honestly, if EMC just was exceptionally clear that this is a product that will require workarounds to be sorta-NDO-ish, all of the steam is out of the discussion.

    I am an employee of NetApp Corp. Proudly. (Ex-EMC too… 13 years.. so was my wife…). My comments are my own and no not necessarily represent those of my employer.

  5. Jimbo

    Hello NetApp guy. The blog post wasn’t about NetApp. Gee, somethings never change. If anyone mentions NetApp in a blog post with anything less than gushing praise, then some NetApp guy is sure to jump all over themselves feeling they have to defend it. Do you have any idea how old that gets to the rest of us in the industry? In a dog it would be called ‘fear biting.’ Stop it for crying out loud!.

    Now, to Nigel’s piece. Of course Nigels’ right. And the EMC guys all know it. Everything he said about EMC’s experience and customer knowledge is true. And the EMC guys know exactly what they’re giving the customers with XtremIO – and I have no doubt they don’t like it either. XtremIO forces them to contradict the very messages they use to sell their other products.

    Sometimes I’m ashamed of our industry. There is nothing wrong with XtremIO that honest marketing couldn’t solve. We are livid about XtremIO, not because of the product itself, but because of the lies and the ‘up is down, down is up’ rhetoric surrounding it. .

    But then if EMC just marketed it sanely, it wouldn’t be ‘the fastest selling array in history’ or some other such nonsense. It’s called lying and it’s called greed. it’s a sickening, embarrassing reflection on our industry that the most respected company in our industry has to resort to those tactics. And it’s sickening how many analysts go along with it. They too know better, but greed wins.

  6. Ed Saipetch

    The world is changing. This stuff is bound to happen more and more as things change. What has lagged the most in the IT industry? Storage. Nigel, your issue may be with EMC marketing this as awesome-but-old-skool storage but many legacy players in the storage industry have taken a beating. Netapp for the un-unified ontap, XIV for dual disk failures, and the list probably goes on and on. We can go piss on Amazon’s EBS or Joyent’s DC failure but the bottom line is again, tough things or even bad things happen to customers. I believe that this is just an example of changing times of the storage industry needing to continue to innovate at a break-neck pace.

  7. Amirul Islam

    A great rant and absolutely agree that EMC have put some of their customers in a very difficult position right now. No doubt, the likes of Pure and newer vendors are lapping this up, and why shouldn’t they? I thought the days of a destructive upgrade on a Tier 1 storage system had long gone. In fact, I’ve worked on many different vendor storage systems and don’t recall ever having to perform an upgrade that destroyed the data. Always took a backup or replica just in case, but never had to restore it to the Primary array because a software upgrade wiped it.
    Some very valid points about the major changes in the new code and underlying filesystem and metadata, and the risks associated with being an early adopter of these changes. Anyway, would love to hear from a customer who is currently being affected by this debacle.

  8. Mike Shea

    Mike at NetApp here – I think that the essence of the wrong doing is simple: XtremIO was sold to customers with a ‘non disruptive upgrade’ promise. And they knew that was false. KNEW IT.

    I’ve been working with Engineers, Product Managers and Sales people for a long time. An engineers job is to keep customers from getting burned, and they are good at this when the company culture supports this.

    You can be absolutely certain that EMC engineering told the marketing and sales teams that such an NDU statement was patently false. There is no way that EMC engineering did not positively know that this upgrade would be disruptive and destructive, and no way did they ignore communicating that to sales and marketing. No chance. It is what they do.

    The problem is a culture that allows this kind of behavior.

    Blogging after fact is not transparent. It is not even repentant.

  9. Jimbo

    Sorry Ed. Yes, the world is changing, and yes, that means that products are being rushed out the door. But that does not give any marketing department an excuse to lie and deceive customers. And that is exactly what EMC is doing with XtremIO.

    Again, I don’t have a problem with XtremIO not being fully baked. What I do have a problem with is EMC selling me a VMAX in Tier-1 because of XYZ and then trying to sell me an XtremIO in Tier-1 and telling me that XYZ doesn’t really matter. Or worse, telling me XtremIO has XYZ when it doesn’t. It’s an insult to everyone’s intelligence and it’s shameful. And no one should try to excuse it.

  10. Dan Stone

    Has anyone that commented on this rubbish ever used an XtremIO array ? Well I have and you will be blown away if you get your hands on one. The folks that are buying these arrays did not just fall off the turnip truck. Any IT person that enjoys his job would have done a little homework and known it was a new product from a startup within EMC. The fact that it does not have VNX or Vmax in it name could offer a clue that it was not a legacy product. Try it you’ll like it.

  11. Nancy Cason

    Hello, Nancy the EMC Technical consultant here. Of course you all will think I am biased, however I have several XtremIO customers whom I fully disclosed the disruptive upgrade to. My customers have also had OUTSTANDING results when testing XtremIO (more details below).

    Let’s talk upgrade first. We worked the disruptive upgrade into customers’ project plans and timelines (can you say Storage VMotion). Also, VPLEX, tech refresh migrations, and other goodies that can be provided by EMC will get this done as well.

    Now let’s talk XtremIO testing. The joint testing of XtremIO, which involved my customers and a major healthcare Electronic Healthcare Record vendor, produced outstanding results in all areas – performance, dedup, snaps, ease of management, so on. Testing showed with a VERY mixed workload of reads and writes, IO sizes ranging from 4K to well over 64K, write IO bursts, multi-snaps running concurrently – latency remained below 1ms even when XtremIO was doing over 90K IOPs. The test involved two XtremIO encrypted “bricks”, 10TB per. I am a long time EMCer and have always taken a VERY cautious approach with my customers when it comes to new technology. My job is to be the customers’ trusted adviser, so I have to admit when XtremIO was new to EMC, I was very hesitant to recommend it to my customers until it was more “proven”. I have always been a VMAX-kinda-gal. Fast forward a few months. Now I am a true believer and very confidently recommend XtremIO to my customers. The technology is simply outstanding in all ways – mega easy to use/manage, mega fast, unbelievable dedup, instant non-impactful snaps, (no garbage collection – sorry, just had to say it).

    The world is changing everyone. You can either throw stones at the change agents leading the way or……….

  12. JohnM

    Hello Nigel and thanks for blogging on this. I was reading about this at a VMUG event and could not believe it when I was reading it from Register.

    I would agree Jimbo and kind of surprised by the response from Dan on it’s fast and try it you’ll like it… I don’t think I want to risk losing data and no I don’t need to try it. In all fairness we did look at XtremIO at VMworld and after it was purchased by EMC and we decided it needed a bit more time to integrate with EMC. I guess we dodged that bullet.

    In full disclosure we did purchase PureStorage and last week completed our second non disruptive upgrade without incident. We have been very happy with our solution so far but non disruptive was always at the forefront with Pure. From the engineers to the sales team.

    I don’t think XtremIO/EMC set out to do this but felt XtremIO was a bolt on solution to a customers flash problem for EMC. This needed more time to mature and develop but EMC had to address sales in the flash race. Time will tell how the storage community will look upon EMC in the future.

    John

  13. Jimbo

    Yeah, great, Nancy. Too bad your marketing wasn’t as cautious as your engineering. Here’s an idea. Next time test your marketing before it ships.

  14. Nigel Poulton Post author

    Great comments guys, and I’ll get around to replying shortly…

    @jim and @johnm jsut so you know…. it seems Dan works for EMC (according to his IP address) and just forgot to mention that (easily done). Might explain his stance though.

  15. Nigel Poulton Post author

    Thanks everyone for the cracking input into the discussion. A few quick replies….

    @Mike Shea, I’m also pretty sure this is marketing doing what it usually does – distort the truth – and that the engineers will have cringed. As and when they leave EMC we’ll probably hear their stories. Sadly the IT industry, and storage especially, seems to have this culture of distorting the truth.

    @Ed I know we’ve had our debate on Twitter, but for the record here…. I totally get that the world is changing. But putting a half-baked product out to market isn’t the way forward IMHO. Like releasing a car and and then recalling it for a new engine installation (and providing a courtesy car while they put the new engine in). Anyway, we think differently on this and good, it’d be a boring world if we all agreed.

    @Dan Stone. First up are you an EMCer? Your IP address is registered to EMC and if you are you should please disclose. No excuses on this as the NetApp guys that posted before you all disclosed their employer. I wouldn’t buy a car – no matter how good it was – if I had to take it back for two engine refits in the first year. Too much of a pain.

    @Nancy. Thanks for disclosing your emplyer… If youre listening and don’t mind me asking… when did you disclose the disruptive upgrade to your customers? And I’ve no issue with change, and XtremIO from 3.x sounds like it might be getting ready for the enterprise. But to me, it’s been sent to market not fully baked (more than one major architectural change having to made disruptively). And despite the go-faster stripes, that’s sure gotta grind on a VMAX-kinda-gal?

    @JonhM Agree it needed more time. It’s getting their now though…. and soon everyone will have forgotten about this debacle. Except for the odd IT guy who got roasted by his boss over this 😉

  16. Nancy Cason

    Me again, yes the biased, EMC badged gal named Nancy (LOL!). As I continue to keep my trusted-advisor hat on, I would like to say that I DO understand the frustration out there. However, I would like to ask that we all (1) keep things professional, and most of you have, (2) keep the bigger picture in mind, (3) know that Gartner’s first ever Magic Quadrant for All Flash Arrays has XtremIO not only in the leader quadrat, but has XtremIO as THE leader (even with the disruptive upgrade coming, which is JUST around the corner), and lastly (4) know that XtremIO V3 is bringing even greater performance gains (!!!) as well as additional space savings via compression (!!!), and all without garbage collection (sorry, had to say it again, LOL, it is Friday!). I suspect when V3.0 is released and we are hopefully all over the anger of the V3.0 upgrade, XtremIO will move more into the upper right hand Gartner Magic Quadrant and distance itself even more from the one close competitor.

    Now, for the bigger picture. Flash drives and All Flash Arrays are certainly disruptive technologies. We (all people and companies IT) are certainly in a transitional period!! This roller coaster has lots of loops and turns. As the industry goes through this transition, EMC will obviously not want to keep recommending the use of a screw driver if in fact a new hammer is the best tool for the job, faster, easier to use, more affordable, better support the needs of the business, so on. EMC will be evolving as the market evolves as it has always done, otherwise…… (Digital Equipment who?). I have in fact recommended XtremIO to accounts for some of their workloads that have historically been on more traditional arrays (both EMC and non-EMC arrays). I don’t think of this as a bad thing – I see this as an advisory recommendation. I don’t want the companies that I support to nor the company I work for to end up like………Digital Equipment who?

    Nigel, back to the mega irritation over the disruptive upgrade and “half baked” products. You may remember when EMC acquired Kashya which became the wildly successful RecoverPoint product line (which continues to evolve by the way – RecoverPoint for VMs just around the corner). Kashya originally had a host-based-splitter as well as other splitter solutions. A few months after EMC acquired Kashya, EMC decided to no longer offer the host-based-splitter and instead focus on array based and SAN based splitters (and now has a VPLEX splitter). I know some customers were angry regarding the pull of support for host-based-splitter. I had one of those angry customers to deal with. )-; However, it was the right thing to do for the overall growth and improvement of the product line. I feel the same is true for this V3.0 upgrade. It is the right thing to do (even though it will be painful for some in the short term).

    I believe keeping the bigger picture in mind, which is essential given the mega transition going on in IT right now, is more important than ever! Just sayin…..and TGIF to all.

  17. Spikes

    Nice nancy this is a sales opportunity for vplex vmax and a profitable discount on a second slightly used xtremio box. But of course you don’t need it, this won’t happen again see the HA video.

    FYI you better not talk about customer issues in here emc lawyers may get wind, there are agreements that customers sign.

  18. Pingback: XtremIO Craps on EMC Badge | IT and Leadership ...

  19. Chris M Evans

    It’s certainly interesting reading these comments and seeing remarkably polarised and in some case short sighted views.

    1. The storage industry is changing. Of course it is. So is every part of the IT industry. But for enterprise customers with valuable data assets (like your bank account details), science projects like Ceph and Red Hat Storage Server won’t cut it. That’s why banks continue to buy VMAX & VSP – because the hardware is ultra-reliable and more importantly the sale is backed by significant engineering and support skills when things go wrong. You get what you pay for.
    2. We should expect new storage platforms to be immature. Wrong. We should expect vendors to bring products to market with a solid view of their roadmap 3-5 years ahead and when there are others in the industry already offering a wide range of features and functionality, those new players should already have planned out how their infrastructure will support those features. Not everything has to be delivered in v1.0, but a company should have the foresight to know how they will implement a new feature 12 months ahead – without a total rewrite.
    3. Many IT users have never stepped outside the EMC boundary. I’ve been in large and small organisations, delivering technology. I’ve been the customer on many occasions. EMC love to control an account (as do NetApp) and everything is directed towards that. EMC’s success has been as a sales company. When you start to buy other technology, then the game changes and you feel more exposed. There are lots of IT teams just not that confident about looking at other solutions. Staying with EMC (or NetApp) is a nice comfortable scenario. Ny the way, I also advise a large EMC & NetApp VAR, so I see this stuff EVERY DAY.
    4. It’s a question of honesty. EMC/XtremIO have known for some time this issue would be a problem for customers. It’s not the disruption per-se that becomes an issue but the fact that EMC marketing shows everything in the garden as rosy is a clear deception. Take this video from last year – https://www.youtube.com/watch?v=-uTkO758Wxw – Point 3 shows non-disruptive upgrades. This is not true. EMC should take down this video but of course it’s in the public domain and in people’s consciousness now. Watch these videos from Tech Field Day; http://techfieldday.com/appearance/emc-presents-at-tech-field-day-extra-at-vmworld-us-2014/ Josh Goldstein is asked twice how big the dedupe blocks were – he responds “just a few kilobytes” – why not give a more accurate answer here? Because this is right in the middle of the change from 4K to 8K and the delegates would have picked that up.

    One final thought. I’ve seen two EMC’ers on different threads claim that their XtremIO customer was OK with disruptive upgrades. Both were VDI. Can EMC provide ONE reference for a mission critical customer using XtremIO 24/7 where they also were OK with a disruptive/destructive upgrade?

    For disclosure to everyone, I’m an independent consultant (www.langtonblue.com) – I also authored the recent post on XtremIO at http://blog.architecting.it

  20. SimonD

    Full disclosure – Pure Storage employee here…

    Nancy, please don’t fall for your own marketing. The Gartner Magic Quadrant is NOT all about XtremIO when they talk about EMC. The position you have been given is based on XtremIO, the VNX-F and maybe a bit of credit for DSSD. Remember the MQ is about the companies ability to execute and its vision, not about the actual arrays.

    For a real evaluation of the actual XtremIO product you should look to Gartner’s Critical Capabilities report that was issues at the same time as the MQ. In every one of the criteria looked at the XtremIO was beaten by a minimum of 2 ‘start-up’ arrays…

  21. notavendoremployee

    I was told about this issue by an EMC Engineer several weeks ago and we are only running a POC. My question is why are there not blogs about all the XtremIO problems that already existed, burning up the internet. The fact that you have to evacuate the array to expand the X-bricks, you have to evacuate the array to encrypt the X-brick, you can’t expand a 10TB X-brick with a 20TB X-brick. Has anyone heard about the “logical limit”? On the 20TB X-brick you can’t provision more than 40TB of disk, which guarantees a terrible dedupe percentage despite compression add-on version 3.0 or no compression. So they are adding compression, but isn’t changing the block size effectively reducing the dedupe capabilities that already exist on the array? All the IOPs statistics for the XtremIO are based on a 4k block size running an 8K test. What will happen to the 250k IOPs result at 8k block size? What about misalignment? Won’t the larger 8k size cause additional IO misalignment that will eat up IO from the array? Non-disruptive controller failure, non-concurrent drive failure capabilities. There are so many issues and potential issues.

    The destructive upgrade is a big deal, but there are many other big deals that existed before this was ever disclosed.

  22. Chris M Evans

    Notavendoremployee,

    EMC have very rigorous contracts that forbid customers talking about their hardware. These have been in place for years on the VMAX platforms. Remember also that EMC have never provided performance figures on VMAX. EMC likes to control the message.

    Have a look at the terms of your POC and see if you are restricted from discussing the results.

    Now you make a good point about the difference between 4K & 8K and how it affects compression. It would be useful if EMC published figures on how things will change because theoretically all of the figures they have quoted so far have been based on a 4K architecture, including IOPS and latency. With a potentially big change to the architecture we should expect to see revised measurements.

    As for the expansion scenario, I have a feeling that’s another issue waiting to boil over. XtremIO uses an algorithm to determine which of the available nodes is responsible for a subset of the hashed address space from each block of data. One of the reasons the 4K -> 8K change could not have been done in place was presumably that if the hashes were regenerated across an 8K block, the result would have been new hash that would have meant the data was being stored by the wrong node. This is the same issue if a cluster is expanded.

    So, for example in simple terms, imagine a 2 node configuration where the division is based on the first hex value of the hash. Node 1 manages 0-7 and node 2 manages 8-F. We expand to 4 nodes and now node 1 does 0-3 and node 3 does 4-7 (node 2 does 8-B, node 4 C-F). What happens if we want to read a hashed block starting with the number 4? Previously node 1 would provide it. Now node 3 must, but unless a physical data migration has taken place, the data won’t be found. Not good.

    There are two scenarios here; either all the data is moved during downtime when adding extra nodes (painful because this is a move not a copy; can’t go back if it goes wrong), second, the system could keep before and after tracking of every hash block in a separate table until everything rebalances – this is a nightmare, would have performance and memory use implications.

    So, expect to see expansions being disruptive (or not offered) for some time, or another re-write needed to add extra metadata for block tracking….

    Chris

  23. Pingback: XtremIO Upgrade Kit: Pure offers free swing gear for destructive EMC upgrades - Pure Storage Guy

  24. Pingback: XtremIO Upgrade Kit: Pure offers free swing gear for destructive EMC upgrades | Pure Storage Blog

  25. Pingback: EMC XtremIO Upgrade is Non-Disruptive to Customers

  26. Pingback: penguinpunk.net » Pure Storage – “Architecture Matters”

  27. PissedofXtreme Customer

    We moved forward with EMC’s XtremIO and to date we are still not on our production cluster. We were moving VMs and our VDI pools over to it and one horrible Friday it crashed. We were down for over 8 hours. We have had issue after issue since. This last weekend we were going to upgrade our production cluster from the buggy 3.0 code to the stable 3.0.1 code. I spent almost 3 weeks, mostly nights, moving systems off the production cluster to the POC XtremIO that is on the 3.0.1 code, yes we still have that, to prepare for the upgrade. 2am Saturday morning EMC called off the path due to a bug in the 3.0.1 code that would cause a disruptive upgrade and possible patch-rollback in a 4 brick cluster. Now we have to wait once again for a new patch, 3.0.2, that has no official release date. Needless to say, I have had it. We haven’t told the business yet that the update didn’t happen.

  28. Pingback: Why Nutanix isn’t singing the VSPEX BLUEs |

  29. End

    Excellent post. Object storage is still in it’s inacfny and as you point out, there’s not a dominant standard, but the successful vendors will build solutions that combine applications and infrastructure through efficient APIs that deliver needed functionality.

Leave a Reply

Your email address will not be published. Required fields are marked *


*

You can add images to your comment by clicking here.