Today Pure Storage announced a pretty cool new all flash storage array, the Pure Storage FlashArray.  An interesting startup playing in an interesting and massively disruptive area of technology.  This post guts under the hood of the Pure Storage FlashArray and explains exactly why solid state storage technologies are literally changing the rules when it comes to storage array design.

Warning: This is a ~2,000 word deep dive article and not for technical lightweights. We get under the hood of the Pure Storage FlashArray, including technologies like inline deduplication and how flash storage is literally changing the rules of enterprise storage array design.

BTW: I recorded a Deep Dive podcast with Matt Kixmoeller, Rick Vanover and Stephen Foskett where discuss the features of the Pure Stoarge FlashArray and get down in the weeds of some of the cool stuff that flash technology enables in storage arrays. You can listen, or download the MP3 from the link below - 

DON'T MISS AN EPISODE – SUBSCRIBE FOR FREE!

• Subscribe through iTunes
• Subscribe through Google (Android users should also install Listen)

…or click to download this podcast

FlashArray Architecture Primer

Still here…..good.  Let’s start with just a few quick basics to set the scene, and then we’ll get knee deep in some pretty interesting stuff….

The Pure Storage FlashArray is commodity based dual active/active, stateless controller architecture.  There is nothing custom about the hardware, its all off-the-shelf stuff with the secret sauce being the in the software, which Pure are calling Purity Operating Environment.

The two controllers are connected via dual redundant 40Gbps QDR Infiniband.  Cutting straight to the chase, I have to say I am not a fan of dual controller architectures. However, I will say two things about this -

1. As we will find out throughout this post, solid state storage changes just about all of the rules.  So I will hold out on giving them a hard time over this (for the time being).
2. The low latency high speed interconnect (QDR Infiniband) lends itself to a more scale-out future. So I expect scale-out will come some time in the future.
3. I should add that I’m a fan of stateless controller architectures.  They make life so much easier when it comes to upgrading and replacing failed components…

The controllers contain a small amount of DRAM which is used primarily for caching the hottest metadata as well as being a working area for write data as it passes through Pure’s data reduction technologies.

All storage within the FlashArray forms a single large pool of storage and is comprised of consumer grade MLC flash memory from Samsung Electronics.  All volumes created on the FlashArray are automatically presented to all front end ports and the intent is that hosts will round robin I/O across all front end ports (effectively wide striping on the front end).

Today’s controllers support 8Gbps Fibre Channel only, with other protocols, most notably 10Gbps iSCSI on the horizon.

The dual controllers are connected to expansion shelves that house the solid state drives and the NVRAM (actually STEC ZeusIOPS SLC flash drives) via redundant 6Gbps SAS.  Every expansion shelf houses 2 x NVRAM drives in the outer two drive bays.  Housing NVRAM outside of the controllers contributes significantly to the stateless design of the controllers. 

As well as NVRAM being outside of the controllers, system config data is also stored outside of the controllers, in the expansion shelves.  System config data and other metadata is stored right alongside user data in the flash drives of the expansion shelves.

Stateless controllers (separating the configuration and user data from the compute power) is something I’m a fan of and this is a good move by Pure.  For one thing, it makes swapping out the controllers for later generations a lot simpler.

The products announced today are the FA-310 single controller system (use at your own risk!) and the FA-320 dual controller HA system.  Check the Pure website for more details as we wont be talking marketing, we;re just going to talk tech!

That’s the high level primer done and dusted. Now lets talk about the cool stuff….

Inline Deduplication of Primary Data

Pure are making a lot of noise about being able to dedupe primary data inline without taking a performance hit.  If you’re like me, you’ll need a truck worth of salt to help you swallow a statement like that.  Such things don’t seem possible, right!?  Well bear with me because I think this is very interesting….

BTW, this is a cracking example of where solid state changes the rules.

Deduplication, a la Pure Storage FlashArray, is inline (post ACK to host) at a variable block size down to as small as 512 bytes.  And it’s global across the entire namespace of the FlashArray.  Good stuff. 

As a write comes in to the FlashArray, a relatively weak hash is applied tot he data to give the FlashArray a hint as to whether or not the data has been seen before.  The hash is performed by built-in hashing functions available on the Intel chips running the FlasArray.  There is no custom silicon here (no ASIC, no FPGA and not even any GPUs). 

Of critical importance is the fact that utilising such simple built-in hashing functions carries very little overhead for the controllers.

That’s all well and good, but weak hashing functions will require byte-for-byte compares to make sure that we don’t make a mistake and think we have a match when we actually don’t.  And bit-for-bit compares are expensive as hell when it comes to performance (you have to read data from disk).  But therein lies the secret.  The FlashArray has no disks in it, at least not the spinning kind.  Reads from NAND flash are so fast they are practically free!

Soooooo….. using built-in simple hashing functions generate very low overhead for the controllers, and byte-for-byte compares from flash are lightning fast.  Pretty cool stuff!

Lets quickly step through an incoming write to show how it fits together -

1. A write enters the Pure FlashArray (where it is checksummed and copied between controllers over the IB backend)
2. Basic pattern removal and a quick compression is performed on the write.
3. The data is copied to the NVRAM in the expansion shelves. At this point the host receives and ACK, meaning all subsequent operations are asynchronous.
4. Next the hash is applied to the data and a byte-for-byte compare is carried out to determine whether the data is unique or actually a duplicate of data already seen. Interestingly this manipulation of data is performed against a copy of the write in DRAM to make it even faster.
5. Once the data has been deduplicated, all remaining data is unique and is ran through a compression algorithm to squeeze it down even further.  Remember, all of this is after the host has received and ACK but before the data is written to flash.
6. From hereon in, writes are coalesced into structures called segments (usually 56MB), parity is calculated, the data is committed to flash and the copies in NVRAM and DRAM are released.

It is worth pointing out here that all writes to flash are append operations.  At no point, under normal operating circumstances, does the FlashArray update data in-place.  This avoids write amplification, improving lash wear and tear and write performance.

Net net, all writes to flash are deduped, compressed and written in append mode. Perfect for writing to flash.

Pretty cool in my opinion, and something that just isn’t possible with architectures designed around spinning disk.  Dedupe of primary data, inline, in spinning disk architectures just doesn’t exist. At least not as elegantly (I nearly said as pure) as this.

Actually, another reason why deduplication of primary data is rarely seen in spinning disk architectures is that it has the side effect of scattering your data all over the back-end, or more accurately leaving pointers that point to all sorts of random locations on the back end.  And it’s spinning disk 101 that that orderly (sequential) placement of data on the backend helps improve performance.  If your data is laid out in nice contiguous chunks, reading it back will be a pleasant experience, but as soon as you start asking those read/write heads to jump here there and everywhere, all performance bets are off! 

None of the above applies to flash.  Flash is great at random reads, and the speed of random reads makes the reconstitution of deduplicated data fast!  Another thing that spinning disk can struggle with, at least without the help of a large cache.

Interestingly, deduplication is not integral to the process of writing data to flash in the FlashArray.  If the array is is processing huge amounts of write data, the system will dynamically turn off deduplication so that write segments are flushed through the system even faster.  Once the burst of high write activity is complete, deduplication is dynamically turned back on.  However, users are not able to administratively control whether or not data is deduplicated.  Interestingly and importantly though, data that was not deduplicated on ingestion will be deduplicated later (more on this shortly).

Also on the topic of compression, if the unique data being written to flash does not compress well, it will be laid out in it’s uncompressed form. This way there will be no decompress penalty (~50µs).

So, what kind of crazy vendor marketing dedupe figures can we expect to see?

Well….. according to Pure, the data back from the first 100 or so units shipped to customers suggests an average deduplication ratio of 5.8:1.  Pretty good, and pretty believable.  Oh and that’s with the array still operating at sub 1ms latency.  As mentioned on the podcast, this will go a long way to making the purchase of an all flash array a reality. The guys at Pure are talking about $5-10/GB.  Starts to make it compelling.

So with all of that in mind…..why aren’t all flash arrays doing dedupe?

Continuous Background Optimization

The Pure Storage FlashArray never rests, it is always trying to optimise the way that data is laid out on flash.  There are background processes running (that Pure refer to as Continuous Background Optimization) that are constantly scouring the array looking for ways to keep it lean and improve the way data is protected and laid out.

As an example, the Continuous Background Optimization processes will try and group homogeneous data types such as -

• Segments of unique non-changing read heavy data.  Such segments can be grouped together to minimise how often they are re-written, keeping flash wear and tear to a minimum.

• Segments of highly referenced dedupe data that can be made smaller and afforded extra protection.

Also of interest is that each time data is picked up and laid out elsewhere on flash, it is passed through the data reduction engine, meaning that it is checked again for deduplication opportunities.  After all, more data may have been written to the array, adding potentially new dedupe opportunities.  By continually examining the backend and re-laying it out via the data reduction engine -  

• Data stored on flash is kept fresh.  This is increasingly important as we move more and more toward consumer grade flash memory, and eventually TLC flash.
• There is no need for a dedicated rebalance operation to be used when new drives are added to the array.  As a natural by-product of the data optimisation processes picking up and re optimising the backend layout, data will eventually find itself evenly balanced across the larger back-end

RAID-3D

RAID.  Hmmmmmmm a technology from the 80’s that is fundamental to protecting data in every data center across the globe. But also a technology that get’s a lot of stick for being dated and unable to cope with today’s demands.  Should be interesting…..

Well, for starters, RAID-3D is based on write segments, not disks, making it more of an object based RAID approach.  This is a much more modern approach than creating RAID sets based around disks, and potentially works better with the type of failure modes seen in flash.  Not only does it tend to make rebuilds faster and smarter (you don’t have to rebuild zero data that has no user data on it) it also lends itself to parallel RAID operations where multiple drives are involved in protecting and re-protecting data. Net net, fast rebuilds and re-protect operations.

It is also worth pointing out that during re-protect operations on the FlashArray, when new parity is calculated on segments that have reduced parity due to the failure, the data is again passed through the data reduction engine and therefore deduped again and re laid out.

When write data comes in to a FlashArray, write segments are filled up, and parity is calculated against the write segment (segments are actually complex structures made up of smaller structures with their own parity and checksums etc). 

All data is stored with at least dual parity, with certain segments having a 3rd copy based on the importance of the data in that segment. 

Data is then striped across the backend in a pseudo random fashion so that data is evenly spread across the backend, lending itself to nice self-balanced backend layouts.

In the event of flash failures, the FlashArray prioritises the re-protection of data rather than the rebuilding of the failed drive.  In fact, there are no dedicated spare drives, instead spare capacity is reserved within the the pool on all drives in the pool.  According to Pure, re-protect operations should take approximately 20 minutes.  This is fast, and the fact that all data is at least dual parity protected should help owners of Pure Storage FlashArrays sleep well at night.

While on the topic of protecting data, it is also worth noting that all data is checksummed on arriving in the array. These checksums are checked every time data is read back from flash.  This checksumming not only ensures that the data read back is error free (free from dreaded bit errors), it also ensures that the data being read back is the intended data.

Final Few Bits and Wrap-up

Full Array Encryption.  Errr yeh…. not a great deal to say on this other than it is always on with zero key management.  And apparently no overhead either.  I don’t know so much about this at the moment.

Caching.  The FlashArray reserves a small amount of DRAM as a read cache.  This read cache will be generally be populated with the most highly referenced dedupe data, as this is likely to be the most frequently accessed data.  I personally like the logic behind this as its not the usual guess work of read-ahead – instead it’s the guesswork that highly deduped data will be frequently accessed .  And if it doesn’t prove overly useful for your I/O proile, not the end of the world as reads from flash are amazingly fast anyway.

All flash Arrays: The market for all flash arrays is maturing at a rate of knots. There are already a ton of interesting startups with innovative technologies.  But the big traditional vendors are now starting to wake up to this market with the likes of EMC acquiring XtremIO – a huge validation of long term viability the market itself.  One has to wonder how long before more of the startups get acquired.

Pure Storage:  The way I see it, Pure Storage are an interesting company with some disruptive technology playing in a potentially new and large market.  They seem to have a cracking product underpinned with solid technology.  They are aiming at the traditional Tier 1 storage market of FC attached Symmetrix VMAX, HDS VSP, HP 3PAR etc.  This is a big market, tons bigger than the so called Tier 0 market.  However, this large Tier 1 market is dominated by big hairy Gorilla called EMC.  And they don’t like other people coming and setting up camp in their territory.

If you like listening to technical podcasts, or have a commute to work where you’d like to listen to engaging technical discussion, Rick Vanover and I recorded a Deep Dive podcast on the Pure Storage FlashArray and all of the cool technology it employs.  The podcast can be listened to or downloaded in MP3 format from the links below. It’s a cracking technical discussion -

DON'T MISS AN EPISODE – SUBSCRIBE FOR FREE!

• Subscribe through iTunes
• Subscribe through Google (Android users should also install Listen)

…or click to download this podcast

1. What is the point of hanging solid state storage off the end of a network wire, surely doing so cripples its performance?
2. How is solid state storage best used in a storage array?  As a cache extension, as a tier, as a metadata accelerator, for everything?
3. How can data reduction technologies like deduplication be implemented without sacrificing performance?
4. Build it yourself or pick it form the shelf. Are the best solid state storage arrays custom built form the flash up, or can they be assembled form off the shelf parts?

The discussion was lively at times with each of the vendors (Nimbus Data, Nimble Storage, Violin-Memory, Solid Fire) represented having strong and different opinions.  If you're interested in solid state storage arrays its definitely worth watching.

If the link below doesn’t work, try this link.

But….. looking closely at version 3.0 of Microsoft’s Server Message Block (SMB) protocol has me excited.  It looks the bizzo!

A year or two ago I wondered why Microsoft bothered developing SMB in the face of NFS. After all, Microsoft had a decent NFS stack, NFS was marching forward, and Microsoft was bleeding cash like never before. What was the point in throwing R&D at the development of “just another file sharing protocol” when they could just ride the wave of NFS…

Anyway, fast forward to today, and we’re on the verge of seeing major uptick in the SMB protocol.  An uptick that is really really interesting.

Windows Server. The Best File Serving Option

It’s my personal opinion that 64-bit Windows servers, with appropriately priced storage, make for a pretty compelling file server solution.  Usually better than the SMB based file server solutions offered by the likes of NetApp and EMC!

Microsoft tends to be ahead of the above vendors in the implementation of SMB features, as well as integrating better (or even at all) with other useful file based technologies such as DFS, Anti-Virus etc..

Many vendor based solutions ship either limited or zero support for things like DFS and are a lot later to market with features like BranchCache.  And when the do come to market, they often only implement a limited subset of features.  An example with BranchCache might be implementing only the Content Server role, but not Hosted Cache Server.

BTW BranchCache has also been improved in SMB 3.0!

As the author of the protocol, Microsoft will always be ahead of the EMC’s and NetApp’s of the world when it comes to implementation of the SMB features we want.

I remember one vendor telling me that even after asking SMB developers at Microsoft, they could not get a solid answer for exactly how a directory listing should be displayed.  Should the listing return contents in alphabetical order…  While those days are long gone, 3rd party vendors will surely always be behind MS.

SMB Moving Hyper-V Forward

Just about anyone, except may be for the most ardent fibre channel bigots, will tell you that VMware works like magic over NFS.  Its so simple to configure and has historically had a bunch of advantages over block (as well as some disadvantages).

Previous versions of Hyper-V were not ideal for running virtual environments over SMB, and previous versions of SMB weren’t as ideal as SMB 3.0 for Hyper-V.  Now, both Hyper-V 3 and SMB 3.0 are great for each other.

Hyper-V will now support SMB a la VMware and NFS. VHD files and virtual machine config files can be hosted on SMB 3.0 shares. Snapshots too as well as VSS and SMB 3.0 being tightly integrated.  All goodness.

This will be cracking or Hyper-V.  Bringing the simplicity and cost advantages that file protocols tend to have over block, without sacrificing much in the way of performance and reliability etc.

A large number of VMware customers love VMware over NFS and previously Hyper-V was lagging.  Hyper-V 3 with SMB 3.0 could well close that gap chasm.  Pretty exciting IMO.

SQL Server over SMB

SQL Server 2008 R2 over SMB 2.1 looked interesting.  User databases could be stored on/accessed over SMB shares.  SQL Server 2012 with SMB 3.0 promises to be even better.  Not only can the SQL system database now be stored on an SMB share, the performance improvements will make such configurations all the more compelling.

I speak to more and more Oracle customers doing Oracle over NFS. I expect to see the same in the SQL Server and SMB world.

SMB Encryption

Security is becoming more and more important, and more and more of a pain in my rear!  Secops guys are being handed bigger and bigger sticks to beat us with and encryption is showing its ugly face in more and more places in the stack and the data center.

Try kerberizing NFS 4 for privacy (encryption) and you will find that its complicated and has limited support from a lot of vendors.

According to the Windows Server Blog, enabling encryption on SMB 3.0 is as easy as checking a check box.  That’s great………. although I confess that I did enjoy the almost countless hours I spent in the lab kerberising NFS 4.0 in privacy mode  

SMB Direct

SMB Direct is SMB over RDMA.  Seriously, SMB over Remote Direct Memory Access via RDMA adapters like InfiniBand and RoCEE.

Now this is one that Im a little more sceptical about and not sure it’s a great place for SMB.  It sounds cool, but am unsure about the use cases and appetite folks will have for it.  I’m not expecting to see huge pick-up for a while.  But I’m open to being totally wrong.

Summary

As I said at the very beginning…. I’m not exactly enamoured by Windows Server 2012.  I’m not sold on the UI and have not yet seen any killer features to make me, or customers, desperate to have it.  However, dont let that take away from the promise of SMB 3.0.  SMB is maturing and maturing at pace!

While SMB 3.0 might not be a headline grabber like a fancy touch-screen friendly UI or Hyper-V, it looks like SMB 3.0 will be absolutely fundamental to many improvements in Windows Server going forward.  And I kind of like it when a low level, non-headline-grabbing technology like SMB steals the show.

The big emphasis on using SMB for server applications like SQL Server, as well as Hyper-V show that SMB is way more than just a protocol for file servers.  SMB 3.0 sheds light on the future of networked storage in the Microsoft world.

Back in the day… File server protocols were for boys.  Block protocols were for men. If you wanted low latency, decent IOPS, or remote direct memory access you need block.  May be the world is changing….

The who’s who of the solid-state storage world will be in attendance and I’m expecting it to be cracking technical event discussing what is easily the most important and interesting topic in the storage world! In fact one of the attending vendors has recently implied that the solid-state revolution may be as pivotal as the transistor revolution.  Interesting take, considering the massive influence on life and society that the transistor has had.

This inaugural Solid State Storage Symposium is a one day event be guaranteed to be packed with high quality information and discussion. If you;re in the area and want to attend, free tickets are available from here.

In the afternoon I will be running the following panel –

What’s the best solid state storage array architecture?

synopsis:  The 20+ year old array architectures that are served up, day-in day-out, by the traditional vendors are struggling to cope with todays I/O demands. How they will cope with tomorrow’s is anyone’s guess. Looking to plug the hole and steal their business is an increasing crop of solid-state array vendors. Yet they’re all going about it entirely differently. This panel will get to the bottom of how to design great SSD arrays and why they’re architected like that.

I’ll be doing my best to make sure the questions I put to the panel are the best possible questions on the topic.

Following on from this I’ll be sharing more of my thoughts around solid state storage through posts and podcasts.

Chris Evans, Howard Marks, Robin Harris and Stephen Foskett will be in attendance and leading the panels and discussions. These are guys that I respect a lot and recommend you follow them and their opinions. They have some great insights and opinions, despite the fact that I don’t always agree with them.

Vendors in Attendance

The following vendors will be in attendance –

NOTE: Although the discussion Hu and I had was over Twitter, I should point out that I have met Hu many times in person and he is an absolute legend and despite being over 70 years old would kick my rear in a fist fight!

Tier 1 Shmear 1 Zzzzzzzz

We’ve been here a million times before and I wont drag it out. Although there is no strict definition of what Tier 1 is, most people have a good idea…. usually VMAX, VSP (and P9500 OEM’d by HP) and occasionally DS8×00 from IBM. Enough on that.

My point is that traditional definitions of Tier 1, and hence the platforms that cling to this accolade, are becoming less and less relevant by the day. And the reason is clear….. SSD!

SSD Has Changed Everything

Before the rise of SSD, it was accepted that Tier 1 storage sucked in performance compared to RAM in a server, often by several orders of magnitude. Those were hideous days!

Fortunately those days are drawing to an end. Application owners, and even some business folks, are becoming savvy to the delights of SSD.  The cat is out o the bag so to speak, the horse has bolted.  And once the horse has bolted there is no reigning it back. Beleive me this is a good thing!

While I’m a fan of much of the architecture behind VMAX and VSP (especially VSP/P9500 as I’m a sucker for customised designs) there is no getting away from the fact that the way VSP/P9500 and VMAX implement SSD is a bit of a fudge.  They have basically lashed 3.5-inch and 2.5-inch SSD drives onto their existing back-ends. Back ends that were designed for rotating rust, not SSD.

And its not just back ends, these so called intelligent controllers have been designed and honed over the years for spinning media, again, not SSD.  What am I saying….?

Well…Take the spec sheet numbers for a ZeusIOPS SSD from STEC.  We’re talking 120,000 read IOPS and 75,000 write IOPS. Say what you will about vendor spec sheet numbers, these are crazy figures compared to 15K FC/SAS spinning disk. 

The question then begs…. how many of these bad-boys can a traditional controller architecture like VSP/P9500 or VMAX drive across its back end?  Let me cut the suspense, not many!  In fact I might challenge whether a VSP/P9500 or VMAX can even drive a single one of these drives across its back end!  Correct me if I’m wrong, but I wont be holding my breath for a corrective response. 

And of course, none of the above stops any of the vendors from selling them to you, at a cracking price too.

Do They Need to be Driven?

While I appreciate that there is some foundation for a counter argument that traditional controllers do not necessarily need to drive SSD hard due to auto-tiering technologies such as FAST VP, Hitachi Dynamic Provisioning/HP Smart Tiers.  However, then why not sell something more suited to the controller and back end architecture. Say something more like Mach16, Mach8IOPS etc also from STEC.  They seem cheaper and more suited toward the specks of traditional T1 controllers.

So What’s the Answer

I’m certainly not a planet sized brain guy that works for a vendor, but I’ve seen some stuff that looks like it will fit the bill nicely….

Technologies such as SCSI Express and NVM Express look like cracking candidates. Technologies/protocols designed especially for SSD.

Keep the SSD closer to the controller.

There is absolutely place for SSD in the server, but also in the shared storage array… just not hidden away on the back end loop of a shared storage array like some unwanted child.

Keep it closer to the controller where it can be accessed faster and driven better.  From here it can be used as a 2nd level cache (as in FAST Cache, only not starved on a back end loop like existing FAST Cache implementations) or as a LUN or extent of a LUN, as in FAST VP or HDP/Smart Tiers.

Hell ,we wouldn’t dream of exiling VSP or VMAX DRAM on a back end loop. Why do the same with SSD!? 

Sure I appreciate that it was done as a quick fix, “get SSD in their no matter how you do it”, and disk drive form factor on a back end loop was the simplest and path of least resistance. But it’s ugly!

Todays Gen1 Implementations Are Not All Bad

While I admit that todays SSD implementations in VSP/P9500 and VMAX do serve some customer requirements. They do not do it very efficiently.  And I also admit that the vendors have done some work to smooth the road to SSD in their existing T1 architectures such as tuning caching algorithms and making sure that SSD access doesn’t hog the loop (pun intended). 

However, existing implementations are somewhat like driving a Ferrari around central London. Build a freeway! 

If things don’t get better, the rising generation of arrays bearing the mark “Designed for SSD” such as WhipTail, Kaminario, Violin Memory etc will start to crop up in traditional Tier 1 accounts.  These guys are working hard at implementing many of the traditional Tier 1 features into their products, features like NDU code upgrades, replication…. oh and cloud integration.

Yes, Tier 1 is about robust replication, caching, N+1 or higher…. but its also about performance. High performance storage is still a bit nichey today, but it wont be tomorrow. Oh and it’s getting late!

UPDATE: (3rd April 2012) It seems I need to add a clarification to this post.

I am absolutely not saying that performance is the numero uno characteristic of Tier 1 storage systems. What I am saying is it is one characteristic. No high performance, no Tier 1 badge.

My point being that the rise of SSD is redefining everybody's performance expectations – massively. To the point where if the traditional Tier 1 vendors dont keep pace, they will no longer be considered performant. No high performance, no Tier 1 badge.

Also. Im not saying the current crop of SSD vendors are better than the current crop of Tier 1 vendors. I am saying that the current crop of SSD vendors are disrupting the industry and that disruption will only increase as each day passes. The current crop of tier 1 guys need to up the ante or risk becoming dinosaurs of the industry.

Oh and Im not bashing Hitachi or EMC, hopefully that came across early on in the post when I stated that Im a fan of the architectures, especially VSP/P9500

Last week, Rickatron and I had the privilege of having Dr Garth Gibson, the co-inventor of RAID technology, on the Deep Dive podcast.

To say that Garth is a legend of the technology world is an understatement. 

I’m posting this here in case anybody reading this site doesn’t know about the podcasts that I do over at http://infosmackpodcasts.com.  Hopefully the show is a great listen and may be something that you can listen to on a drive/flight/train home for the Christmas holidays (for those of us who celebrate Christmas).

I’ve not had a chance to listen to the show myself yet, but we touched on some cracking technology topics when recording.  Garth’s insight is tremendous.  This guy is thinking way way ahead and is extremely switched on.

Some of the topics we covered included -

• Future of RAID (obviously)
• Future of solid state storage
• Future of magnetic recording media – rotating rust
• Sub-LUN tiering
• Commodity Hardware with all of the smarts in software
• The rise of less and less reliable technology
• Parallel NFS (pNFS)
• Securing storage
• Technology predictions for 2012

The show is about 55 minutes long, and Rick and I did our best to raid this guys massive brain.  No small task.

Enjoy listening.

DON'T MISS AN EPISODE – SUBSCRIBE FOR FREE!

• Subscribe through iTunes
• Subscribe through Google (Android users should also install Listen)

…or click to download this podcast

Well…. while there I had a quick Twitter exchange with Scott Lowe and Amnon Izhar about fabrics.  This is another topic that gets the old juices flowing.

Cisco UCS Kicking Some @rse

Not that long ago I recorded a podcast on the topic of Cisco UCS.  At the beginning of the podcast, Rick and I did a goofy intro making fun of former HP CEO Leo Apotheker’s apparent comment that if Cisco were selling UCS they must be selling it on planet Zircon, because HP weren’t coming up against it in the real world. 

Well…since then I know of loads of people taking up Cisco UCS, whether it be straight UCS or as part of a Vblock.  So it’s pretty clear that Cisco have really taken the game to HP in the compute market.  And clearly, in some people’s opinions, it has advantages over and above HP’s offerings.

Yes, UCS scores points on management, but they also score technical points too.  UCS has good tech.

HP have traditionally taken a steady approach to pushing server based technologies forward.  Things like Virtual Connect FlexFabric are ….. well….. OK.  Kind of average.  But with Cisco in this market, average won’t cut it any more!

Software

No doubt that HP has to improve their software.  Now that is a task that I do not envy.  If describing HP’s software I’d have to use words like clunky, bloated, slow, cumbersome, flaky…  Making significant strides in this area will be long a hard.  Good luck HP.

The Cool Stuff – Fabrics

It’s my opinion that HP need to up their game on the server technology front.  They need to be bolder and more innovative (even if via acquisition).  HP needs to get back to being best of breed and not rely on the fact that they are HP.

One way to do this, and at the same time take the game right back to Cisco, would be to buy Xsigo Systems.

NOTE: I am in no way suggesting HP are even remotely thinking of buying Xsigo – how the hell would I know! I'm just saying I think it's a cracking technology and they would make a cracking combo!

Xsigo’s Server Fabric would really cloud enable HP solutions and put them ahead of the 10Gbps curve, as well as simplify and add flexibility to HP solutions.  And all of this to an extent not possible with average technologies like VC FlexFabric.

One of the things I like about Xsigo solutions is similar to what VMware have been doing with vMotion technology.  VM movement operations (using the fs3dm datamover with hardware offload) can now be done within the storage controller without the data ever having travel up the stack from the array, through the fabric to the ESXi host and then back down again.  As a result not only are storage vMotions faster but the load is taken off the fabric and ESXi hosts.  It just makes good sense to do things this way!

Similarly, Xsigo allows us to do things in networking and server-to-server or VM-to-VM connectivity that just make sense.  Things like server to server traffic not having to travel up and over the network -

Now I’m no stupid, yes I expect the network guys to come out with their pitchforks and tell me how this doesn’t fit with their legacy backwards current models and I expect the network security card to be played.  But hey, the best technologies are always disruptive and force us to rethink the way we have traditionally done things.  And that’s only one of the advantages Xsigo brings.     

In a nutshell Xsigo would leapfrog HP servers past Cisco UCS (yes I think UCS is probably ahead o HP on a technology front right now).

What About A PCIe Fabric

So I like the idea of that and yes I know that the number of PCIe nodes deployed makes Ethernet look niche, but when I looked into this in the past (~2 years ago) PCIe wasn;t so scalable lacked a lot of networking capability and scalability.  Server fabrics dont necessarily have have all of the features of networks, but when I looked in to it back then I didn't feel that PCIe was a great fit.  Mental note: Go and revisit this.

Anyway, may be a blog and a podcast on Xsigo is required…

Setting the Scene

I’m not a fan of taking a legacy array technologies and shoe-horning them full of SSD. Frankenstorage springs to mind!

I am, however, slightly more of a fan of technologies like Violin Memory and Kaminario (to name just a couple). 

However, what I’ve seen at HP Discover has the potential, in my opinion, to make even the likes of todays Violin Memory and Kaminario arrays look legacy, very soon.

Let me take a really quick minute to set the scene…… it wont take long….

In many ways I like Violin Memory.  It’s been designed, almost from the ground up with SSD in mind – definitely not a technology designed for rotating rust and then fudged or butchered for SSD.  Take the lid off one and you’ll see exactly what I mean. 

It looks like a lot of thought and design effort has gone in to it.  And the technologist inside of me likes that.  However,………. I wonder if it’s too custom and too proprietary to live and thrive in todays market?  Todays market demanding commodity and all that jazz!

On the other hand there is Kaminario.  These guys take standard off-the-shelf Dell blade systems, off-the-shelf Fusion-io cards, layer some clever software on the top and out pops a high performance SSD array.  Ticks the commodity-is-king and software-is-everything checkboxes, but has its drawbacks.  Servicing the Fusion-io cards is clunky and requires you to crack the lid of the blade server open (never good in Tier 1 production data centres).

Doing things properly

So what I saw at HP Discover could well be the future of SSD and SSD arrays, and it goes by the name of SCSI Express.

SCSI Express is a protocol that is currently being independently standardised under part of INCITS T10 by the SOP-PQI Working Group and the SCSI Trade Association, with involvement from SFF Committee and PCI-SIG.  Quite a crew and quite a project, but it was suggested to me that it might be standardised in six or so months.

SCSI Express enables SCSI over PCIe (SOP), and under the hood it is a SCSI initiator talking to a SCSI target over PCIe via PQI. 

PQI stands for PCIe architecture Queueing Interface which is a flexible and extensible transport layer that is very fast and lightweight.  I’m told that it leverages the best from some of the existing proprietary SCSI over PCIe solutions available from companies such as PMC, LSI, Marvell and even HP.  The difference being that PQI and SCSI Express are being developed as open standards rather than being proprietary to the above mentioned companies.  Existing SCSI over PCI protocols such as MPI from PMC are found in silicon in most of the array controllers we see in the world today including EMC, NetApp, Hitachi…

Blah blah blah…. but what does this all mean?

Well, here we have a protocol and interface that is being designed especially for high speed SSD, not spinning rust!

Might not sound much, but when you look at some of the legacy architectures that we are currently bolting SSD drives to (it’s not uncommon to stick an SSD capable of 40,000 IOPs in an array with a backend that can support only a fraction of that) it starts to bring this into perspective.  Todays SSD drives are severely hamstrung by the legacy architectures we bolt them to and it verges on a crime to do such.

SCSI Express will release the shackles.  It will allow you to take a hot-plug 2.5-inch form factor SSD and install it into a 2.5-inch form factor drive bay on the front of an industry standard server, just like we do with hot-plug drives today.  The major difference being that the SSD won’t be hamstrung by SAS or SATA.  The drive will mate with a specially designed, but industry standard, interface that will talk a specially designed, but again industry standard, protocol (the protocol enhances the SCSI command set for SSD) with standard drivers that will ship with future versions of major Operating Systems like Windows, Linux and ESXi.

On the topic of hot-pluggable, I’m led to believe that this is not very robust on PCIe as we know it today.  However, it’s doable and the guys at HP Discover told me that this should be standardised and available by the time this is productised (somewhere around the end of 2012.

The Future of SSD Arrays

So, again, in my humble opinion, the future of SSD arrays is unlikely to look like a VMAX, VNX or even VSP……  Nor is it going to look like a Violin Memory array. 

In my opinion it is going to look like an HP Proliant, Cisco UCS…. name your industry standard off the shelf x86 server, crammed full of industry standard form-factor hot-pluggable SSD drives, running SCSI over PCIe with all of the smarts and clevers in software on top o VMware (sorry couldn’t resist throwing the VMware comment in). 

Seriously though, I can see it.  While I love SSD and some of the SSD arrays out there, I’ve has always felt like there is something not quite right about them.  I think SCSI Express/SOP is the missing magic!

And when these products ship and change the world, I plan on putting my feet up and retiring, as this will clearly solve every problem that the storage world has or ever will have!

Prototype at HP Discover

The concept box on display at Discover is an early prototype, but was an HP ProLiant server with an early Fusion-io 2.5-inch SSD drive connected to the PCIe bus (I know its not really a bus) via an SFF 8639 backplane connector (PCIe 12Gbps 6 lane).  However, this is also doable over PCIe cable implementations. 

In the prototype unit at HP Discover, the SCSI Express drives connected to the PCIe bus and bypassed the HP RAID controller similar to the picture below. 

This kind of implementation leaves a at least a couple of options when it comes to RAID - 

1. Have RAID implemented higher up the stack and utilising CPU cycles (the dreaded software RAID).
2. develop newer RAID controllers with SCSI Express and SCSI over PCIe in mind

RAID………….. now there’s another technology that could do with being brought in to the 21st Century!  but that’s another conversation.

The plug for mating with the server is SFF 8639 and the current board connects to the server via the SAS cables.

Footnote: To SCSI or not to SCSI

Interestingly, the somewhat competing standard of NVM Express is looking to do its magic without SCSI.  A bit bolder, but I have to wonder how much harder?

Half of me would love to get rid of SCSI and the legacy that it brings.  But then again three quarters of me would have liked to see Ethernet replaced with something like Infiniband.  Ethernet clearly isn’t going anywhere and I my head tells me SCSI isn’t either.  Too deeply entrenched.

On the positive side though, SCSI is battle hardened and well understood.

Footnote: Competing Standard NVM Express and EMC

No storage futures story is complete without mentioning EMC. Sorry HP and the rest of the storage industry but I like to be honest.

Interestingly EMC are not on the list of companies behind SCSI Express.  But they are on the list of those behind NVMe!

As we know, EMC are one of the biggest families in the storage Mafia, and they have significant influence over VMware, one of the biggest families in the technology Mafia.  Now, in mind (where 2+2=33) having the daddy of the storage industry behind NVMe, coupled with the interesting noises that VMware has been making about the future of storage, I can’t stop my mind running wild with what they might be up to…. I would it even put it beyond then to be planning the death of SCSI!

Anyway, enough for now.  Thoughts and comments mandatory

Anyway, now that that is out of the way….  It’s old news that HP recently announced a major refresh of it’s 3PAR line, the P10000 or 3PAR V-Class.  I despise HP’s naming conventions so I will call it V-Class.

One potentially interesting thing to note from the product branding P10000 (that’s ten thousand) is that HP have given the 3PAR a higher number than the rebadged Hitachi VSP, the HP P9500.  Generally speaking with HP storage products, the higher the number the bigger and more enterprise the array (that’s my interpretation anyway) -

P2000 = MSA.  Aimed at small businesses

P4000 = LeftHand iSCSI array.  Aimed at SMB

P6000 = EVA.  Aimed at SMB

P9500 = OEM’d Hitachi VSP.  Aimed squarely at high end enterprises

So one would naturally assume that something with a higher number than the P9500 to be… well….. more “enterprise”.

Anyway lets get to the good stuff……

The Good Stuff

For me, the major improvements that came with the V-Class are the following -

• Moving from PCI-X to PCIe
• 4th generation ASIC
• Peer Motion

Aside from the innovations listed above, the V-Class just feels more enterprise, more high performance.  Things like 2 x 4th gen ASIC per controller node, compared to the previous generation T-Class only having 1 x 3rd gen ASIC per controller.  Similarly there is 2 x quad core CPU’s per controller node versus 2 x dual core in previous generations.  Then there’s more than doubling the available control and data cache per node as well as the inclusion of T10 DIF support.  All in all, it feels like it now has the muscle to stand it’s ground against VMAX and VSP.

The Stuff Thats Missing

Of course it’s not perfect.  On the Negative side the following are disappointments -

• No 2.5-inch drive form factor. Only option remain 3.5-inch. This is behind the curve and a disappointment.
• No SAS backend.  The backend remains switched FC-AL and while I appreciate that this allows large distances between the controllers and disk cabinets I’m unsure how enterprise this is.  Don’t get me wrong, I like the idea, but I’m not sure I’d want to bet my entire array on a single FC cable routed under the floor and across to the other side of the data centre hall!?  Also, the switch to SAS as the backend of choice is well under way.
• No de-duplication or compression. Not that anybody else does this either, but with the knowing that the 4th gen ASIC was on its way I wondered whether we might have seen these features. If HP/3PAR had come to market with this then that would have seen them widely recognised as a leader again.

PCIe

I won’t spend long on this.  I think it speaks for itself.

While I’ve been a long time fan of the 3PAR architecture, I’ve always been a little embarrassed of the PCI-X architecture.  It’s old technology and has no place in a modern and innovative high performance storage array.  Anyway, its gone, so let’s never speak of it again.

4th Gen ASIC

Why do I care about ASICs, FPGAs and things like that?

While I agree that I shouldn’t really care as long it performs well and stays on its feet, knowing what is under the hood and how something is put together helps you in all kinds of ways.

For me, custom silicon (ASIC) has its place in high end of storage arrays for at least another 5 years.  Offloading certain functions to ASICs is more efficient and allows for higher performance.  Not too dissimilar to the approach VMware has taken of late where it offloads functions through VAAI to the storage array.  The concept is simple, offload specialised tasks to the expert – VMware offloads storage tasks to the storage array, similarly within a 3PAR array, InForm offloads certain storage related tasks to the specialised silicon (ASIC).

Hitachi agree with this approach and the VSP (P9500 in HP parlance) has taken a very similar approach.

For deep technical discussion on the use of custom ASICs in storage arrays listen to the recent “Odds and SODs” episode of the Infosmack Deep Dive podcast that I host.

Peer Motion: More Than a Get Out of Jail Card

At a high level, Peer Motion promises to simplify migrating data between arrays, and potentially federate those arrays into teams of loosely coupled arrays. 

The hope from customers will be that Peer Motion will allow them to dynamically and non-disruptively move workloads between arrays – similar to the way that auto-tiering products dynamically move extents between different tiers of storage…. 

One immediate use case may be as a get out of jail card that allows for more aggressive overprovisioning.  The worry about overprovisioning has always been “what do I do when I can no longer add capacity to my overprovisioned array?”.  I for one would feel a lot more comfortable aggressively overprovisioning if I knew I had a technology their that could migrate apps to another array and thus free us space on the old array…..

Another no-brainer use case will be tech refresh.  Many large organisations struggle hugely with tech refresh. They buy technology and cant get off it.  And this isn’t always vendors trying to lock them in. Many times organisations can’t even seamlessly move to the next gen architectures from the same vendor!

If HP/3PAR manage to nail this in true 3PAR style (simple and efficient) then I will be extremely impressed and HP can expect many happy customers.

At day 1 it looks like Peer Motion will allow non-disruptive migrations between any 3PAR systems.  Roadmapped, no doubt, will be heterogeneous Peer Motion.

Summing It Up

If 3PAR didn’t have a truly enterprise class high performance model before, they do now!

If VMAX and VSP didn’t see 3PAR as competition at the high end before, they will now!

If you wouldn’t consider 3PAR in the past, may be you should now.

All in all this is a good move forward.  While like the move to PCIe and the 4th Gen ASIC, the real killer feature is Peer Motion.  If HP/3PAR nail this they are on to a winner.

However, most of the work that went in to this refresh will have been well under way when HP bought 3PAR.  So this should be seen as a true 3PAR array with probably very little HP influence (good or bad). 

Now the onus is on HP to invest and give 3PAR the space to continue to innovate.  On the other hand, the worry is that HP may starve and strangle 3PAR.

Comments welcome, and don’t forget to tune in to the Deep Dive podcast I do over at infosmackpodcasts.com.  We try and generate high quality technical discussion on a broad range or enterprise tech topics.

Nigel

With that in mind, the Infosmack Deep Dive team have worked extremely hard to put together 1 hour and 15 minutes of no nonsense deep dive podcasting with some of the most recognised experts on the topics.  We’ve done our best to cut through the crap and deliver technical content that you will find hard to find anywhere else.

If you want to know the craic with vSphere 5, this is the best place to find it…

In the show we cover what we feel are the hottest topics, and we spend time on them.  Topics such as -

• Virtual Hardware version 8
• Storage DRS
• The new HA engine
• VMFS 5
• 64TB LUNs
• ESXi
• Virtual Center 5

Joining us on the show this week we have some of most recognised professionals and experts on the topics -

Frank Denneman – Consultant Architect at VMware

Duncan Epping – Principal Architect at VMware

It’s fair to say that Duncan and Frank wrote the bible on VMware HA, DRS and Clustering.  There first book VMware vSphere 4.1 HA and DRS Technical Deepdive is honestly one of the best tech books I own.  And I’ve ordered my copy of their new book – VMware vSphere 5 Clustering Technical Deepdive.

Michael Heffernan – Chief Technologist at HDS.  Heff is responsible for HDS integration with hypervisor products and has some deep hooks in to VMware.

Add to the above, Rick Vanover and myself and we have a team of people doing our absolute best to give the information that you want!

I hope you enjoy the show and learn something. I did.

Listen to the show via the player below, or visit Infosmackpodcasts.com

If you enjoy the show, we have plenty more planned and plenty more already on the shelf an waiting to be listened to.  Go see for yourself at Infosmackpodcasts.com.