3PAR ASIC–Two-edged Sword?

I caught up with Howard Marks last week when he was in town in London.  We went out for a quick bite to eat one night and talked shop for two or three hours.  I know my place when in the company of a grey beard like Howard, so for the most part I kept quiet and listened.  However at one point I vented my frustration that 3PAR arrays still don’t support flash as a cache – despite being based on a modern innovative architecture supposedly better suited to todays demands and requirements than something like, let’s say an apparently donkey architecture like EMC VNX.

But hang on a minute.  EMC VNX has supported flash as cache for ages now.  I cant be bothered to look it up, but I reckon VNX has supported flash as a cache (in the form of FAST Cache) for at least 2 years, probably more. 

Seriously, I thought the uber-modern architecture of 3PAR was supposed to make adding innovative technologies easier.  Could it actually be that the architecture of the box is hindering the adoption of important technologies like flash as a cache!  I mean seriously, how long does it take to catch-up to a 20 year old legacy technology like VNX?

Howard suggested that the problem might lie with the 3PAR ASIC.

We know that ASIC design can elongate the innovation cycle when compared to implementing on commodity Intel type architectures, but I honestly never thought that the ASIC might be behind the sloooooow uptake on flash as a cache.

I know that 3PAR support flash as a tier, but I also know that that isn’t always enough.  VNX supports flash as a tier and/or as a cache.  How can 3PAR be behind?

I have to admit that I’m a fan of the way 3PAR implements thin technologies, and I do  believe that implementing thin through the ASIC gives them an edge.  But in this case, assuming Howard is correct in assuming that the ASIC is the stumbling block to implementing flash as a cache, it seems the ASIC really is a two-edged sword – the ASIC giveth and the ASIC taketh!

Oh, and since I said Im a fan if how 3PAR implements thin technologies via the ASIC, it is only fair to say that when it comes to flash as a cache implementations, VNX clearly wipes the floor with 3PAR.

QUICK UPDATE: I'm certain that there will be tons of existing, and potential, 3PAR customers with legitimate use cases and requirements for flash as a cache.  So surely this will be an engineering priority within HP, and as such would have been implemented by now if it could!? 

21 comments for “3PAR ASIC–Two-edged Sword?

  1. April 24, 2013 at 12:56 am

    ASIC are really good for the core stuff like like
      – RAID (though I'd argue that a cheap intel chip's ability to do bazillions of XOR calcs per second makes that kind of unnecesary),
      – inter-box communication (though again RAPID-IO in VMax and iWARP enabled NIC's do that pretty well, though they have their own ASICs to help there,),
     - inline analysis of information for thin provisioning detection. (No competition here)
    Having said that, I don't think it's a matter of whether ASICs are a hindrance, or even whether ASICs deserve to be at the very core of the solution. The problem is that the  architectural elegance of the 3PAR hardware came from a cleansheet design for a Tier-1 disk array. That design was built around  disk+DRAM and their inherent limitations, flash and ubercaches weren't part of that design.
    The older, architectures like the VNX you mention are a litlte looser and less single minded (it happens with age I suppose), so it arguably easier to add things like flash and megacaches, without breaking the architectural purity of the 3PAR model,  and that (rather than the ASIC issue), is where I believe the core problem lies.
    It may also be that in the case of companies like  EMC and NetApp, the R&D budgets and commitment for storage innovations dwarfs the investments that HP has made in 3PAR. ASIC centric approaches probably dont help, but they're not insurmountable either … (as a parallel case in point BluArc did some pretty cool things with FPGAs up until they ran out of money and got bought by HDS ,, I havent seen much out of them since then).
    On the other hand, if the whole software defined storage thing pans out though, then 3PAR with its ASIC accelerated hardware that does simple fast and reliable things in hardware for disks might make a an ideal platform for the flexible smart stuff like caching done up in the server layer where CPU and software is cheap.  That will of course require HP to spend a chunk of money and R&D in building that software based feature set outside of the array, and o date, I've not seen them do much around that. It also begs the question, of whether that might expose 3PAR to (from HP's point of view) undesirable commoditisation trends ? Will they choose to disrupt themselves before someone else does it for them ?
    It may just be a matter of timing, either way for all of the vendors in the storage industry there are Interesting times ahead ..

    Regards
    John Martin
    (Disclosure – NetApp employee, though not posting on behalf of my employer, nor are these views necessarily representative of those of my employer)

  2. April 25, 2013 at 4:48 pm

    Hey Nigel,

    You and GreyBeard (Howard) guessed wrong!  The 3PAR ASIC is not a limitation for us in having flash cache on HP 3PAR StoreServ.  You seem to be assuming that 3PAR and VNX are an "apples to apples" architectural comparison and that we need flash cache as badly as VNX.  HP 3PAR doesn't have to overcome a 20+ year old architecture and the performance baggage that it carries.  You said "VNX is wiping the floor with 3PAR".  That's just rubbish and here's why.

    1). With the HP 3PAR architecture, we spread everything everywhere over a low-latency interconnect (an HP 3PAR unique).  We are leveraging massive distribution and massive randomization - as a result performance is high and consistent, even under failure/maintenance conditions. And we do this without trading off $$/IOPS (big cache monolithic), $$/GB (short-strokers) or complexity. I remember in our recent SPC-1 submission, an industry pundit said that he heard we short-stroked HP 3PAR to get what were the best results to date.  If he had looked at the submission, he would have seen that we were using around 80% of the 3PAR capacity.  Repeat after me: Massive distribution and massive randomization resulting in high and consistent performance (NOTE: if you find some VNX SPC-1 results that we can compare, let me know).

    2). This HP 3PAR architectural advantage drives a different set of customer priorities.  We've focused on other areas of innovation versus flash cache.  As example, if you didn't notice, we introduced the 3PAR StoreServ 7000 – we think this was an important innovation so we could have one architecture and one common set of tier 1 data services that span from the mid-range to the high-end.

    3). EMC VNX is a traditional, two-node modular architecture with active-passive controllers and narrowly striped volumes. Performance is neither high nor consistent – so the need to boost performance was high.  EMC investing in flash cache early is no surprise. But Fast Cache is only a temporary fix – it's effective up until the point you max out your VNX and your only options is to buy another one.

    I can't publicly comment on whether we'll have flash cache in the near future but I'm happy to connect you and Howard up with our team to answer your questions.  And hopefully we can get you to HP Discover in Barcelona in December.

     

  3. April 25, 2013 at 10:04 pm

    @John.
    Thanks for the comments. I see what you mean about the tight architecture potentially be the bump in the road to implementing flash as a cache. Oh and an interesting assumption that the likes of EMC and NTAPs R&D budgets for storage dwarfing HP/3PAR. :-S

    @Calvin.
    Calvin thanks for the feedback. Dont get me wrong, the 3P architectural advantages over the competition are not lost on me! But I stand by my statement that a flash cache would be valuable weapon in the 3PAR arsenal and plenty of customers would like it. In certain scenarios, VNX with a flash cache WILL beat 3PAR – despite the 3PAR architectural advantages. The 3P architecture is great, but would benefit from flash as a cache as well.

    I’m sure it will come, and when it comes it will be hailed as a great addition to the 3P portfolio.

    BTW I’ll take a closer look at your comments about SPC-1.

    Oh and let’s not get like politicians either…. I didnt say “VNX is wiping the floor with 3PAR”, I actually said “.. when it comes to flash as a cache implementations, VNX clearly wipes the floor with 3PAR”.

  4. April 25, 2013 at 11:12 pm

    LOL – I actually wasn't trying to pull the politician ploy of quoting something out of context but can see where you might think I was.  Sorry about that.  

    Be sure to take me up on the offer to talk to guys that are a lot smarter than me on this stuff! They can better explain why they made the decisions they did – tons of data that supports the mantra of massive distribution and massive randomization resulting in high and consistent performance.

  5. john
    April 26, 2013 at 11:10 am

    Having seen a number of POC's vs VNX I can say that the opposite has been true in my experience. VNX with flashcache doesn't beat 3PAR except in some very niche areas. The problem with flashcache, like all caches really, is that the performance benefit is difficult to predict up front, so your mileage will vary,

    From what I've seen flashcache is incapable of providing consistent guaranteed performance across a wide range of workloads, unless you backfill with more disk (more expense) to smooth out the peaks and troughs, which defeats the object of flashcache in many cases. That's why it regularly loses out to 3PAR in bake offs. BTW the ASIC is not a blocker to any of this stuff it's just about different architectures having different priorities, yes it would be nice to have every feature in the book, but do you do it first, or do you do it right. 

    Like Calvin said VNX needs flashcache.

    HP employee :-)

  6. April 29, 2013 at 7:07 pm

    @John (John H) Very long time, no see.

    I agree that 3PAR will beat VNX in most instances, especially when you start to scale!  And this is down to the architecture and the ASIC….

    Its just a frustration that it doesnt support a flash cache, as flash cache tech would be useful.  Im sure it will come, and when it does Im sure it will have its use-cases advocated by HP…  

    By not having it, VNX, NetApp et al can rock to a POC and make it look like VNX performs better.  Of course POCs tend not to represent scale.  Not having this feature leaves the door open to other vendors.  And to my main point, Im fairly certain the ASIC is behind why we arent seeing lfash utilised as much as it could, and will be in the future, in 3PAR systems.. 

  7. John
    April 29, 2013 at 9:00 pm

    Nigel, think about where the ASIC sits in the data path, also remember it’s interaction with that data is designed to be completely transparent. Now why would such an implementation inhibit adding features. Like I said priorities, priorities, priorities, the 3PAR architecture had very long legs vs the soon to be end of life traditional architectures.

  8. April 29, 2013 at 9:06 pm

    John, now we're talking. Thats the kind of thing Im looking for. I'll take a look at the architecture diagrams later and see what I think about your point.  Appreciate your input.

  9. Jim Haberkorn
    April 30, 2013 at 11:05 am

    EMC has a Fast Cache white paper that explains the limitations of Fast Cache, and makes it clear that Fast Cache should not even be considered unless there has been a thorough analysis of the workload and it is determined that the backend is the bottleneck. Too many customers have implemented Fast Cache thinking it makes a VNX faster – it does not. VNX performance is ultimately limited by the controllers, and Fast Cache won't help that.

    The white paper also says Fast Cache performance falls between that of a HDD and an SSD and that any new data has a warm-up period during which the performance will be roughly that of a HDD. Because of that, SSDs with good tiering software will be much more effective than Fast Cache. The problem with the VNX is that its FAST VP tiering software promotes data in a highly inefficient 1GB chunk, and therefore for the VNX Fast Cache probably is necessary. Even so, EMC will admit that data on one of the VNX's SSDs will be faster than data in Flash Cache because of Fast Cache overhead.

    But 3PAR has blazing fast controllers in whch many controller tasks are offloaded by the ASIC, and also has excellent tiering software. So, Calvin's point is accurate:  – 3PAR customers don't need Fast Cache right now. 3PAR with its tiering software will still be much faster than a VNX.

     

  10. April 30, 2013 at 10:57 pm

    Do you have a link to this EMC paper Jim?

  11. Jim Haberkorn
    May 2, 2013 at 11:08 am

    Hi Nigel,

    Sorry for the delay – May 1st was a holiday here in Switzerland! Here are two links to the same EMC paper I referenced in my comment: the 2nd link is to the specific paper I referenced. However, I quickly scanned this updated paper and it looks like all the caveats I mention in my comment above are still there.   http://www.emc.com/collateral/software/white-papers/h8046-clariion-celerra-unified-fast-cache-wp.pdf

    Here’s the link to the older paper just as an FYI (October 2011) http://uk.emc.com/collateral/software/white-papers/h8046-clariion-celerra-unified-fast-cache-wp.pdf

  12. May 14, 2013 at 3:02 am

    I love this idea! For a long time I have liked the tech behind the EMC potrucds but have hated their sales cycle. When I was a an EMC VAR it was hard to try to learn the product and that left us relying on the EMC sales and engineering teams that never seemed to be in sync. Since going back to be a customer I have simply avoided EMC because its not worth the hassle of the broken sales process.However if our team had access to to cool technology and the ability to review it in our labs and play with it then it would go a long way towards making me comfortable engaging the EMC sales channel. Because I would be coming from a position of knowledge and power now just a dumb customer.

  13. John
    May 23, 2013 at 11:51 pm
  14. john
    May 24, 2013 at 12:08 am

    Bit of a Freudian slip there ;-) or more likely just phone typing.

  15. Billy
    June 11, 2013 at 3:46 pm

    I'm not going to say who, where or how I know… but you will see Flash Cache on 3PAR in the near future.. along with some other VERY sought after features around data unification and data reduction.

  16. Glenn W
    July 10, 2013 at 4:05 pm

    Note from a guy that currently works at EMC but spent twelve years at HP/Compaq.

    Having worked closely with both Jim H and Calvin they know I tell it like I see it and don't fudge.

    My personal testing shows VNX has advantages over 3PAR (I've only tested the smaller configurations <100 drives).  That was without FAST Cache.  FAST Cache can improve many situations not just a few corner cases.

    But that really isn't the point these days is it?  What is of paramount importance to customers is ease of use and reliability oh and that little thing call money.

    VNX has a unified GUI.  This makes things easier than the 3PAR approach of a gateway to provide the service.  VNX can always change the backend hardware without affecting the user experience.  3PAR can't.  But this is a different discussion.

    I would point out one more thing about spreading all your data across chunklets.  It can cause a lack of consistency because a rogue server can affect every other server using those drives.  The latency is less than predictable.

    NOTE:  These are my personal opinions not those of my employers past or present.

    Glenn

  17. billy bathgates
    August 20, 2013 at 10:29 pm

    are any of the people commenting here NOT emplyees of stroage vendors?

  18. Jared Bizzy
    August 21, 2013 at 1:00 am

    I actually work in an environment where we have a mix of VNX, Clarion arrays and other assorted EMC solutions. I also, personally, run a separate test environment that is backed by a 3Par F400. As Mr. Calvin Zito said, you can't make apples to apples comparisons between the two. 

    EMC is beautiful at creating tuned, custom performance storage for tier 1 enterprise applications in a dedicated environment. The EMC FAST cache is an extension of the FAST (Fully Automated Storage Tiring) system. It does not act as a direct cache for the storage processors. The cache must warm up and determine which blocks to promote to flash cache for peak performance. In a lot of dedicated IT shops this is great. You have the same ebb and flow of data day to day and once the FAST cache learns this you'll have great performance gains.

    However, in the case of an IaaS organization, like my test environment, I think 3Pars wide stripping is a great advantage. The full potential of the underlying disks is always available to the work load. And, that performance is augmented at a dependable rate by the node cache (actual cache, not flash). This works really well for heterogeneous workloads like when virtual applications are built and torn down regularly. In my experience FAST cache can't adapt to these demands in a way that justifies its expense. If you have a good idea of your peak performance demands you can build your 3Par array(s) to handle that with a percent buffer. As long as you keep an eye your environments growing demands and react accordingly you'll always have the performance you need. 

    I feel like FAST cache lets you cheat on your back-end disks. If you have a stable, predictable enterprise, it can make your slightly overburdened FC and SATA disks run like champs. But, if you want always available, predicable service levels on heterogeneous workloads 3Par can give it to you with an easy to understand provisioning platform. For me, 3Par's zero detect feature is also pretty sweet.

    So, from a tech guy who uses this stuff every day, I don't think it's fair to draw apples to apples comparisons. FAST cache is a cool feature but, I can't imagine 3Par is in a rush to make this happen. They do offer flash arrays with wide striping.

    - Jared

  19. Gary Anderson
    September 18, 2013 at 8:34 pm

    I worked as a storage implementing specialist for many years. The company I worked for sold 3PAR/EVA, VNX/Clariion, AMS (HDS), Storwize/XIV.  I will not dwell too much on performance, but want to write a few words about the VNX product.  And believe me, I have nothing against EMC or their VNX platform, but here are some of the things that I experienced working with EMC’s midrange system.

    If you buy a unified system, you get basically a block unit with nas gateway and tons of cables. The cable spaghetti looks pretty much like a prototype and still does more than 2 &frac12; years later. The first batch of VNX systems sold, had a lot of bugs in the code and were a pain to install. Code upgrades were difficult to perform, but all this has improved. Too shut down and start up a unified system, you have to do it in steps and pull/insert power cables at certain points.  Not a very elegant hw solution and many of my customers complained about this.  The old Clarrion gui, Navisphere, was terrible. Total lack of intuitivety. Only  HDS had a worse gui. With VNX came Unisphere and it is a few steps in the right direction. To manage the block part is ok, to manage the nas/file part is not that easy. Still need to use cli and for me this exemplifies EMC lack of good user interfaces.  Same thing on Vplex and Vmax products. We had several problems with SSD drives when used as fast-cache. Worse case was data corruption at one site, other customers experienced that the VNX system worked very slow because of hw problems with a single SSD drive. For me it looked like EMC implemented a lot of functionality, but the VNX  code was not stable enough to avoid serious problems.  We also saw that fast-cache at many customers did not improve the performance at all. There was a bug in the code for the SSD drives, and then we switched of the fast-cache, the customer did not experience any performance drop.  EMC has tons of best practice documents and many customers ended up having to buy more disk drives to get good performance on certain applications.  At one customer we had to use SSD drives for log disk for async mirrorview.  EMC are very good at sales, but VNX is not always delivering what it is promising.  But I have to say that EMC is very good when it comes to support.  My experience is they are very professional in this aspect.

    Gary

  20. billy bathgates
    December 13, 2013 at 5:30 pm

    Hi Jared and Gary, thanks for your comments/insights… I'm checking out 3par and netapp (and IBM SVC) because the HP EVA line is being discontinued… Have been mostly happy with the EVA storage for a long time, I think we will lose a little ease of management (and get an increased price tag, whatever we end up doing), but will probably be OK with either one… Maybe we will end up with a mix for a while… If I can abstract the main operations with scripts so ther array command differences are insulated, we should be ok…

     

    -Billem of Redmond

     

     

Leave a Reply

Your email address will not be published. Required fields are marked *


*

You can add images to your comment by clicking here.