If you haven’t heard yet, HDS have just announced their next generation enterprise storage array.
WARNING! This article is a technical deep dive. It’s about 3.5K words and is not recommended for light-weights. If you want to know what its all about from technical perspective I suggest you grab a hot brew and read on. If a technical deep dive is not what you want, then scurry on somewhere else.
Still here? Nice one! I think you’ll find this the best source of technical info on the VSP available anywhere.
So lets start with a picture. This is what a VSP looks like –
Pretty ugly right! However, my mother always told me it’s more important for something to be beautiful on the inside. So don’t let the ugly front doors put you off your brew 😉
For those who may not know, HP also sell the VSP via an OEM agreement with Hitachi Ltd of Japan. So the product has two names, an HDS name and an HP name – but its the same product under the hood.
The HDS name – I suppose with the current trend of everything needing a V at the beginning of its name, or the need to include the words virtual or cloud, it’s not surprise that HDS have called it the Virtual Storage Platform, or VSP for short.
A couple of points on the HDS name –
- It’s a whole lot simpler than USP V, or is that USPV or USP-V or USPv, or may be even USP-v…..
- I suppose its better than CSP, where C would be for Cloud 😉
Internal Code Names: Internally at HDS the project was known as Victoria, and prior to that as project Mother Goose (guess it's been that long in coming that it needed two internal project names). The Mother Goose name I'm not sure about, but the Victoria I have a fair idea….. The guys at HDS tell me the obvious, they chose the name Victoria as it starts with a V, and that it has it’s beginnings in the Latin for Victory. However, I’m fairly certain the real reason is that somebody in the team probably had a childhood crush on a girl named Victoria and they never quite got over her 😉
The HP name – HP have previously named the product XP, with previous versions including XP1024, XP12000 etc. So one may have guessed that HP might have named it something like – XPV, vXP, XP2048.. But No. HP are marketing the product as the P9500, bringing it in line with the current HP naming standards – MSA is now P2000, Lefthand is P4000, XP is now P9000 series.
Hitachi (日立)Factory Name: The internal Hitachi factory names for previous generations of this family of product were –
- Lightning 9900 = RAID400
- Lightning 9980V = RAID450
- USP = RAID500
- USP V = RAID600
The VSP follows suit and is internally referred to as the 日立 RAID700. If you view the results of a SCSI Inquiry, or SCSI codepage 0x83, you will see this as the product ID.
High Level Overview
The VSP is a Fibre Channel only (oh, and FICON) enterprise storage array that scales from a single bay up to a maximum of 6 bays. The two central bays house what Hitachi is calling Control Chassis (containing processors, memory, front and backend connectivity…) as well as drive enclosures. The remaining 4 bays can contain only drive enclosures. It can scale from zero (0) drives to a maximum of 2,048 drives, making it the biggest (Update: 29/09/2010 – biggeest from a drive slot perspective) storage array from Hitachi, but still not as big as some of the competition – but size is not everything to all people 😉
NOTE: There is no minimum entry model like with previous versions. Also the smallest config can scale to become the largest with no forklift upgrade required. This is in stark contrast to the previous generation where a USP VM could not be upgraded to become a USP V.
Now to the REAL technical stuff…….
Today’s Holy Grail for all decent storage arrays is the ability to tier at the sub-LUN level. That is, the ability to have the most active parts of a LUN reside on fast (most expensive) media, and the less frequently accessed parts reside on slow (cheaper) media. The VSP has this and it calls it Hitachi Dynamic Tiering (HDT).
Sparing you a lecture on Sub-LUN tiering I’ll just pick the interesting technical points for the VSP implementation –
- All drives can be placed into a single tiering pool. You just throw your SSD, SAS and SATA into an SLMP (Single Large Melting Pot – I made that up). Not what I was expecting given existing HDP Pool best practices, but I suppose it works from a simplicity perspective.
- New data is staged to the highest tier available, and then, as it becomes less active it is migrated down the tiers. Most folks I speak to seem to initially think it will be best to stage data at the lowest tier and then promote it up through the tiers as it is used. I guess staging to the highest tier means that you will always be fully utilising your expensive Tier 0. Time will tell if this approach works.
- The Sub-LUN extent size (which Hitachi calls a Page) is the infamous chubby chunk… the 42MB HDP Page. Basically the VSP will move data up and down the tiers in units of 42MB (contiguous space). If you have 1MB of a file that is hot, the VSP will migrate that 1MB, plus the remaining 41MB of that Page up a tier. The same for demotions.
- On the policy side of things, you can set gathering windows, exclusion windows, and the movement cycle is between 1-24 hours.
So a tick in the Sub-LUN tiering box for the VSP. All that remains to be seen is whether it does what it says on the tin.
SAS backend and 2.5 inch drives
The writing has been on the wall for a while now, Fibre Channel as a drive interface is on its way out. Don’t panic though, it won’t happen overnight, but by the same token, don’t be blind to the facts, it is happening. The new protocol, interface and backend architecture is SAS (Serial Attached SCSI).
NOTE: Enterprise class SSD drives come with SAS interfaces, and you can plug Enterprise class SATA II drives on to SAS backends.
As far as I’m aware, the VSP is the first of the enterprise arrays to adopt a SAS backend, but it’s unlikely to be long before the rest follow suit. So is there any value in the SAS backend, or is it just a tick in the box to be future proof?
There is an argument that the 6Gbps full duplex SAS backend will be able to sustain more IOPs and throughput to SSD drives, certainly more than a 4Gbps FC-AL (or switched FC-AL) backend. However, don’t expect to be able to drive the ~45,000 IOPs that the spec sheet of an STEC Zeus IOPS claims. Also note, that the spec sheet I’m reading from the STEC website only shows 3Gbps SAS interface, making it slower than 4Gbps FC at the time of writing :-S However, when a 6Gbps interface comes along, the plumbing is already there in the VSP.
As far as I’m aware, the only manufacturer supplying SSD drives for the VSP is STEC Inc and I believe the drives to be the same ZEUS IOPS drives (200GB and 400GB varieties) that the competition use.
The pictures below are close-ups of a drive enclosure. The picture on the left shows the enclosure with the fans (on hinges) opened, the picture on the right shows with the fans closed into position –
2.5 inch drives. For me, 2.5 inch drives are the future of disk drives, and allow for greater capacity in the same footprint. For example, a single frame VSP with a Control Chassis 0 can have 384 drives, each of which can be 2TB. That’s a lot of capacity in a relatively small footprint.
It’s also strange that once you’ve seen one system with 2.5 inch drives, all systems that still deploy 3.5 inch drives look clunky and dated. Bring on the 1cm drives! J/K
UPDATE 29/09/2010 : Readers should note that capacities of 2.5 inch drives currently lag behind those of 3.5 inch drives. For latest drive info see relevant drive manufacturer websites.
Front End Director (FED) Design
So the front end design of the VSP is significantly different to the design of previous generations and one of the more technically interesting changes.
At a high level, Hitachi have taken the wide-striping concept that is now commonplace in backend architectures, and are bringing it to the front end.
Basically, the FEDs and BEDs (Front End Directors and Back End Directors) have custom I/O routing ASICs that are specialised for I/O traffic management – Hitachi are calling these data accelerator ASICs. These ASICs have an affinity with ports. However, the more general purpose CPUs are no longer locked to particular ports, they have been moved to a processor complex called the Virtual Storage Director (VSD) where they are pooled and can have their resources dynamically assigned and un-assigned from any front or back end port.
Example: In legacy or more traditional architectures (I’m going to keep this high level), Processor 1 would be tied to Port 1 and Processor 2 would be tied to Port 2. If Port 1 was running like the clappers and maxing out its Processor, while Port 2 and Processor 2 were sitting idle, there would be no way to assign the resources of Processor 2 to Port 1. Now, with the VSP, the CPUs are pooled and can be assigned or unassigned from any Port, so a port that is running flat out can have the resources of multiple CPUs assigned to it.
Of course the proof is in the pudding. BUT, this could potentially do to the front end, what drive pooling and wide-striping have done to the backend. Look at a heat map of a traditional backend where specific spindles are assigned to specific LUNs and applications, and compare it with a heat map of a system that employs drive pooling and wide-striping. The difference is amazing.
This could also have potential heat and power benefits, turning off processors when they are not required – although I’m speculating now.
ASIC vs Commodity: The general purpose CPUs, where the microcode, including copy services, replication services, HDP etc execute are Intel Core Duo Xeon quad-core processors. The ASICs on the FEDs and BEDs are Hitachi designed. Hitachi, like 3PAR, obviously feel there is still value in custom silicon. This is in stark contrast to the likes of EMC VMAX and IBM XIV which have taken the commodity route, driving value out of software.
The custom ASICs take care of latency sensitive data such as user data, whereas less latency sensitive data, such as asynchronous replication traffic, is processed by the general purpose CPUs on the VSDs.
While on the topic of the front end, I should point out that this is a FC and FICON only array. No iSCSI (no plans for it) and no FCoE. However, FCoE is apparently not far off, may be the end of Q4 or early Q1 2011.
Control Chassis (Logic Boxes)
As mentioned earlier, and seen in the images below, the centre two cabinets, in a six frame VSP, house the Control Chassis’ – Control Chassis 0 and Control Chassis 1. The picture below is of the front and rear of a frame containing a Control Chassis. The Control Chassis is in the bottom half on both pictures, the top half contains the drive enclosures hidden by fans.
And a close-up shot of each – note the slapdash fibre cabling – tut tut 😉
As will be show in the diagrams below. the Control Chassis contain the following –
At the front there are 4 x Virtual Storage Directors (VSD), and 8 x Data Cache Directors.
At the rear there are 8 x FEDs, 4 x BEDs, and 8 x what Hitachi are calling Grid Switches, or GSW if you like.
Data cache is backed up to onboard flash drives, which I believe will mean no requirement for large batteries etc. Pretty cool, assuming I’m correct. It has been confirmed that the previous statement of not requiring large batteries to hold cache in a power loss scenario is correct.
Each BED has 8 x 6Gbps SAS paths. That adds up to 32 backend SAS links per Control Chassis, and 64 x 6Gbps links in a fully configured unit. SAS runs at full duplex.
Control Memory (CM) is located on the Virtual Storage Directors alongside the general purpose CPUs making it very quickly accessible to the CPUs, like an L2 cache. As with previous architectures, Control Memory stores the usual metadata and system state such as LDEV mappings, DIF tables, run tables etc.
Grid Switches – The Grid Switch boards provide the, 4-lane PCIe gen. 1, paths that cross connect all of the other boards. Each Control Chassis can have either two or four GSW boards. Each GSW board has 24 unidirectional ports, each port having a send and receive path, with each operating at 1024MB/s.
Hitachi refers to the Control Chassis as being tightly coupled, and the interconnect between the two is PCIe gen 1 over copper. The Control Chassi are able to communicate by mapping Hi-Star over PCIe.
Microcode and Microcode Updates (interesting)
As you would hope and expect, a large part of the VSP microcode is inherited from the USP V, which in turn inherited it from the USP, which inherited it from the Lightning 9980V, which inherited it from …… The code from the USP V was recompiled to run natively on, and exploit the advantages of the new Intel processors. All new features to the VSP, such as Dynamic Tiering, were developed to run natively on the VSP hardware.
Interestingly, the microcode now runs in fewer places – basically the Virtual Storage Directors. This has the effect of hugely simplifying the microcode update process. On that topic, and it’s not very often that I get excited about the microcode update process (in fact I’ve never gotten excited about it before), but this time I’m excited! Stay with me on this…..
If we remember back to when we talked about the general purpose CPUs (where the ucode runs), we mentioned that the processors are not tied to front end ports. Well as it turns out that this has a huge impact on microcode updates. Basically, because the processors that run the microcode are not tied to ports, when you update the microcode, no ports ever have to go offline! Take a second to let that settle…. once again, front end ports do not go offline during microcode updates! There are other architectures out there that require paths to go offline during code upgrades, these are an outage waiting to happen. I’m not saying there wont be a reduction in performance, but this is still HUGE!
Real world experience: Anybody who works in large environments knows that microcode updates can cause issues. Path failover and multi-pathing in general are often the cause. Common examples include servers that incorrectly have both paths cabled to the same SAN fabric, not noticing that a path is already in a failed state on a server, applications that don’t work well with path-failover… So to have a solution where paths are not taken offline during a microcode update is an absolute god-send for large organisations – IMO.
Data Centre Friendly
This might seem strange, but for products that are designed to spend their entire lives in data centres, enterprise storage arrays are often like a wart on the nose of a well designed Data Centre.
In the past, it has almost been a pre-req for enterprise storage arrays have features like bespoke custom sized cabinets and blowing hot air in whatever direction they felt like – sometimes towards the ceiling 😉
Well finally VSP has made some steps in the right direction –
- It comes in a standard sized 42u 19 inch rack
- Has a hot and cold isle airflow design
- Can take power feeds from above or below – you can now install one in your garage without installing a raised floor 😉
Not quite at the point where customers can install one in their own pre-provisioned racks, but it’s moving in the right direction.
Hitachi are also claiming ~40% power reduction, which, if true, will go down well with every Data Centre I know.
But it Looks So Ugly
Let’s face it, most of us have a shallow side to us and like stuff that looks good.
But before I get lambasted over this, let me state that I know that appearances are not that important when it comes to kit that spends it’s life in a data centre and never sees the light of day. However, I think there are a couple of areas where it does matter (but way way less than how well it works etc…). Those couple of areas are –
- First impressions count.
- It’s about brand. Hitachi are no doubt billing this as an exciting product and trying to generate interest – at the end of the day they need to ship it in large quantities. In my opinion, the outward design does not give a good first impression and certainly doesn’t capture the imagination.
Although HP didn’t actually put a blue neon light on the front door, their marketing guys seem to be getting the message with the photo-shopped image on the HP StorageWorks P9500 below –
In a day when the competition are sporting kit with dual-power-fed blue neon lights and others with bezels designed by Armani, I was hoping for more from Hitachi. But its not the end of the world and I think I’ll live, just about.
However, on the inside it certainly looks neater and tidier than previous generations. This, again, is important as it speak of good engineering.
Moving on from something that’s not all that important, to something that absolutely is…management software.
Apologies for mentioning this so far down the post, but I haven’t actually tested the software, and I have never been a fan of Hitachi storage software, so I find it hard to get excited. Having said that, it looks for once like they might have come good on management software …
A previous poll on this website about how good or bad Hitachi Storage Management software is showed the following results –
33% said it was “Poor”
25% said it was “Average”
14% said it was “Good”
15% said they would “rather pull their teeth out than use it”
8% said it was “excellent”. These voters no doubt worked for Hitachi 😉
The Victoria launch is a joint Hardware and Software launch. Device Manager is now at version 7, and it appears that Hitachi might have been listening to feedback.
From what I've seen, aesthetically speaking, it's a huge improvement on previous versions. Like other decent management GUIs it’s based on Adobe Flex technology and it actually works how you would expect a decent management app to work (I’ve seen a demo). You know, like being able to resize the screen, move columns, re-order columns…
Let's hope that it also scales and doesn't need "occasional" reboots.
Hitachi are positioning it as a unified (there’s a word that is finding it’s place alongside virtualisation and cloud) management platform across their storage products – VSP, HCP, HNAS, AMS… Hitachi have also spent a lot of effort on building in SMI-S.
My personal opinion is that the management interface looks a lot better, in the same ball park as the likes of SMC, but not in yet the same league as XIV, 3PAR or UniSphere.
Still Legacy RAID Architecture
As with most things in life, you don’t get everything you hoped for. And for me, the big thing that is missing is a revamp of the underlying RAID architecture. With the long wait for the product I was hoping they were working on this.
In today’s world of 2TB and larger drives, an average of around 10^14 Bit Error Rate and therefore risk of Unrecoverable Read Error being something like one in every 12TB, elongated rebuild times, larger capacities to try and background scrub…… I’ll kill the list there so as not to keep you awake at night worrying about your RAID. Point being, RAID architectures that were written 10, 15, 20+ years ago are showing clear signs of creaking at the seams.
In my personal opinion, distributed parallel RAID architectures that offer ultra fast but ultra low impact rebuilds (or re-protection) are needed. I personally don’t want to see triple parity RAID or other band-aid solutions. A redesign is needed. Technologies like some of those sported by 3PAR, XIV, Xiotech etc need to make their way into enterprise storage arrays. Rant over.
Misc – VAAI and Encryption
VAAI. Unfortunately there is no VMware VAAI goodness on day one. However, this will apparently come in the first code rev planned for either the end of this year or very early next year.
Encryption. The Back End Directors are capable of encrypting data, internal to the VSP, while at rest, with XTS-AES 256 bit encryption. Encryption is done in hardware with apparently no overhead, so wont impact your performance.
I’m also interested at the overlap with the AMS line. VSP will scale as small, or smaller than AMS but with much richer (pun intended) features. It seems an industry trend for midrange architectures to be reaching in to the enterprise space, whereas this seems the opposite way around. Hmmmm…
Well, I’ve about worn my keyboard out so will bring this to a close. Hope it has been useful.
Feel free to leave comments, thoughts, questions… but please, please, please, if you work for a vendor, please disclose this. Thanks!
You can talk to me online about technology on Twitter, I’m @nigelpoulton, or http://twitter.com/nigelpoulton
Disclaimer. I do not work for Hitachi, HDS, HP or anyone involved in the product. I have, however, been contracted in professional services and implementation architect type roles by both HDS and HP in the past. Opinions expressed in this article are my own and do not represent those of any of my employers, past, present, or future. I have obviously had exposure to the product and people prior to product release, see previous posts for info. I have not received a penny in relation to this product.