So a few weeks ago I was at Nimble Storage HQ for a briefing with the Storage Field Day crew. Nice site, nice breakfast, and thanks for contributing towards my travel and hotel costs (that’s the disclaimer out of the way).
Anyway, as part of the presentation, Rod Bagg (VP of Customer Support) made the claim that ~60% of Nimble storage arrays get upgraded during business prime time hours. Seriously! During the middle of the business day! And Rod clarified that these are production systems, not just test systems in a lab.
Now….. call me old fashioned, but I couldn’t swallow that!
The evidence is here 12:00 – 15:00 minutes on the video….
A Great Product or Bad Admin Practices?
The Nimble angle is that this is a huge testament to the quality of their product. To me….. it’s either incorrect data, or testament to dangerous administration practices.
All I can say is…. with what I know about IT infrastructure, that would not happen on my shift!
Don’t get me wrong, I’m not saying that the Nimble product isn’t good – to be honest I actually like it. But to me, no matter how good a critical shared infrastructure product is, it’s still too risky to upgrade during business hours.
I know that the world is changing, but has it changed that much?!?!?!?
You see…. no matter how reliable a product is, it only takes one thing to go wrong to bring it to its knees.
Put another way… if I signed off on an upgrade to a shared Nimble storage array in the middle of the business day, and that upgrade *went south*, I’d expect to be marched off site and asked never to step foot in the building again.
My Experience With Upgrades
Now I know things can go wrong in the middle of the business day even when a system isn’t being upgraded. But in my experience, the risk is a lot higher when performing an upgrade. I’m used to doing things like –
- making sure we have spare drives on site during upgrades
- making sure we have the vendor support duty managers mobile phone number
- making sure my technical staff don’t have any other plans for the day….
…all as preparation for the worst.
And I’ve had bad things happen during upgrade. And every time, I was damn grateful that we’d started our upgrades early Saturday morning and had all day Saturday and Sunday to mop up if things went wrong.
I had one time where I was at the cinema watching the new Star Trek film when one of my team called me to tell me about a certain storage array that had gone down. That was at about 8pm on a Saturday evening and it only got fully back up and running at about 09:30am Monday morning – after core business hours had started! It wasn’t great.
Am I Wrong… Do I Need To Get My Butt Out Of the 1990’s?
I’m not doubting the Nimble product here. But I am doubting the practice of upgrading core infrastructure in the middle of the business day.
Now….. am I living in the past? Nimble aside…. is it safe to be upgrading core infrastructure components in the middle of the business day?