SATA in a USP V. As rare as hens teeth

By | June 3, 2009

This post is for anyone interested in Hitachi storage who feels like they’ve had enough HAM for a lifetime and just wants a taste of something different for a change….

One of the projects I’m currently working on involves deploying a new USP V with native SATA disks side-by-side with FC disks (when I say “native”, I mean the SATA is sitting inside the USP V and on an external array being virtualised with UVM).  Although this configuration has been supported for a while, it’s the first time I’ve seen or done it, and hence the first time I’ve really given it any thought.  So I thought I’d share my thoughts for the greater good.

Seeing as how HDS and HP have supported SATA in a USP V/XP24000 for a while now, you’d think there’d be a bunch of people out there with experience and a boat load of best practice advice doing the rounds, right?  Apparently not!

In fact, the only best practice advice I initially got was that RAID 6 was recommended for the SATA disks.  Fair enough, in a storage array that builds RAID Groups in the way that the USP V does, RAID 6 is a no-brainer for large SATA.  But even knowing what little I know about SATA and the USP V I couldn’t believe that there was no other best practice advice.


Concerns

My overall question was simple –

What is the impact of installing larger, slower disks running a different protocol and a potentially higher failure rate, side-by-side with smaller faster FC disks?

When I asked this question, the most I got back was “you need to do RAID 6 with SATA”.  Thanks, but to be honest not very helpful.  Certainly not good enough for me to press ahead with a production install. 

So to break down my concerns into more detail –

1.  What is the recommendation regarding cache partitioning?

2.  Is the imposed verify-after-write operation that is mandatory for all writes to SATA disk in HDS midrange kit (AMS) also a requirement on a USP V with native SATA?

3.  What is the potential impact of larger, slower and disks with a potentially higher failure rate and resultant elongated rebuild times?

4.  Is there a recommendation for backend layout – SATA sharing BEDs with FC or SSD?

5.  Are there any System Option Modes (SOM) recommended for SATA disk, and if so what are they?

Really simple, and to me, intuitive questions.  Boy was it hard to get people to take me seriously!

So as a result of some digging, I propose the following answers and loose best practice advice –

NOTE:  The following are my opinions and I am absolutely not an authority on the USP V or XP24000.


Answering those concerns

Q1.  Cache partitioning

I highly recommend a dedicated cache partition for SATA.  This is to provide an element of protection for global cache and other cache partitions, effectively ring fencing a portion of cache for SATA. 

Why?  The potential for high cache write pending (WP) is increased with the use of SATA due to the larger size of SATA disks, the slower rotational speed, overall lower throughput, as well as the more limited protocol.

Obviously the size of you cache partition (CLPR) is entirely dependant on the amount of capacity in the SATA pool as well as how hard you will be pushing the SATA.  However, it is important not to size the CLPR too small as doing so can increase the chances of high Write Pending in the SATA CLPR, which in turn can affect the rest of cache (more on this in my answer to question 5). 

A good approach might be to start with a relatively small CLPR (but not too small) and grow it if required.  You can grow CLPRs in increments of 4GB relatively non-disruptively and easier than trying to reduce a CLPR.

Q2.  SATA and Verify-After-Write

In the HDS midrange kit, such as the AMS, there is an imposed verify operation (read) after every write to SATA.  This operation is intended to ensure your data is correctly committed to and retrievable from disk.  However, this brings with it a large performance penalty – increased number of backend read operations for every write operation.  This performance penalty saw SATA in HDS midrange kit perform quite poorly so far as feeds and speeds are concerned, but obviously perform admirably when it comes to data integrity.  Not the end of the world when we consider that SATA is not intended for high performance applications.

The USP V on the other hand is designed to be a high performance array.  So implementing a verity-after-write operation on a USP V in the same way as it is implemented in an AMS array would be disastrous for performance.

Fortunately it just so happens that the guys at Hitachi knew this, and as a result decided to implement the verify operation in circuitry in the disk canister.  This has the positive effect of keeping the verify operation without imposing extra load on the back end directors (BEDs) and back end loops.

It is probably worth pointing out at this point that SATA disks supported in the USP V have an FC <-> SATA bridge in the disk canister allowing them to sit directly onto the existing backend infrastructure, side-by-side with FC.

Net result is that in many cases SATA and FC can realistically site side-by-side on the same BEDs and loops.

But before jumping in with both feet, as always, consideration should be taken with performance centric configurations.  For example, a high performance OLTP System requiring its own dedicated 146GB 15K FC or 146GB SSD (assuming high random read) is probably not best suited sharing its backend with SATA.

Q3.  Potential impact of larger slower disks ……

While this is quite a generic question, the biggest risk, in my opinion, is around the elongated rebuild times associated with large SATA disks.  Rebuilds cause I/O on the backend loops as well as overhead (CPU cycles etc) on the BEDs.  The longer a rebuild, the longer this increased load exists.  On a subsystem under load, rebuilding large SATA disks can take several days.  I don’t know about you, but I don’t want my high performance LUNs sharing a backend with this.

Also, I know that SATA is usually associated with higher failure rates than FC.  However, I have to be honest and say that this is not something I have actually noticed in the real world.  Let’s not forget that build quality for SATA disks, bearings and all, has improved and is of a higher standard for enterprise class SATA.  Couple this with the fact that the disks will be housed in an enterprise array that is designed from the ground up to be a disk friendly environment – think; power, cooling, humidity, vibration……  MY personal opinion is that the notion that enterprise SATA disks fail frequently is a bit of an urban myth.  Happy to be proved wrong…. Well not “happy”.

Elongated rebuilds also increase the window of exposure to a double disk failure.  However, RAID 6 effectively mitigates this.

Q4.  Sharing your backend with SATA

This is issue is somewhat put to bed with the implementation of the verify operation in the disk canister.  With the responsibility for the verify operation being removed from the BED to the disk itself, backend performance should not be hugely impacted by SATA.  Therefore, is it moderately safe to have multiple Tiers of disk share the same backend.  However, h
aving said that, as previously mentioned – I would not personally recommend putting your highest performing disks on the same backend as SATA.

However, whether or not you want SATA stealing your premium priced USP V resources such as cache slots, CPU cycles, internal bandwidth and backend IOPs is another question.

Q5.  Are there any recommended SOMs for SATA

As appears to be standard practice in most installations, System Option Mode 454 should normally be set to ON.

This will base destaging algorithms etc on the average WP rate of all cache partitions, rather than just the highest.  Let me give an example –

You have two cache partitions; CLPR-0 and CLPR-1.  If SOM 454 is set to OFF and either of the cache partitions reaches 70% write pending (WP), aggressive destaging will kick-in across all partitions.  However, if SOM 454 is set to ON, destaging algorithms will be based on the average WP rate across all cache partitions.

Other than this, I’m not aware of any other SOMs that are of use particularly for SATA.


Resultant best practices

So my own personal best practice for USP Vs deployed with internal SATA are as follows –

  • RAID 6
  • Dedicated CLPR for SATA
  • SOM 54 turned ON
  • Isolation of high performance Array Groups on BEDs with no SATA disks.
  • Tier 2 FC Array Groups can probably co-exist side-by-side on the same BEDs without any conflicts.

Oh, I suppose I should also expand a little on my previous reference to the different protocol.  Differences between SATA and FC amount to more than merely physical differences.  Lets not forget that the USP V and it ancestors have been supporting SCSI based disks for many years and as a result the USP V (actually its engineers) have extensive knowledge and experience with the SCSI protocol and how it behave under just about every condition.  The same could not be said about SATA.  This is no doubt one of the reasons why the AMS imposed the verify-after-write operation at such a high performance cost.  How does the protocol behave under all possible conditions etc is an important consideration.  And then there’s SATA Native Command Queueing versus SCSI Command Tag Queueing etc…….  When you add it up, there is a boat load of differences that need to be taken in to account.

As always, just my thoughts.  I don’t speak for any vendor and Im not an authority on the USP V.

Thoughts, experience, feedback and general comments welcome!


FINALLY……….
Give this blog a pat on the back

If you like the content, including the technical deep dives, on this blog, then please take a moment to vote for us in the StorageMonkeys non-vendor blog poll.  We are currently way down in the poll.

StorageMonkeys is a great storage related site that you will probably enjoy if you are in to storage.  Thanks for your comments and support.

Nigel

UPDATE: It is only right of me to point out that there is healthy existing install base of USP V with internal SATA (as no doubt there is with the HP XP).  Not a huge percentage of the overall USP V isntall base, but enough to make me confident that this has been done elsewhere before.  Just a shame there is not better awareness of the requirements and best practices.

You can also follow me  on twitter  – http://www.twitter.com/nigelpoulton
I only talk about storage and not what I’m watching on TV (although that would be Lost, 24, Fringe…….)

13 thoughts on “SATA in a USP V. As rare as hens teeth

  1. Dale Clutterbuck

    Aside from all the technical reasons, it’s very expensive real estate for SATA drives. The "per slot" cost overheads from controllers, licenses and RAID6 can make the per drive cost look rather insignificant.

  2. Chris M Evans

    Nigel, nice post.  Do you suppose SATA internally hasn’t been adopted because of the drive for using UVM, so therefore SATA externally?  Also if you work out the cost per GB of the array itself, the I/O the array can do, justifies using it for more intense workloads?  I think that the overall acquisition costs mean using a USP for SATA data is not attractive.
     
    Chris

  3. Nigel Poulton

    Dale/Chris,

    I agree with both of you that the cost per GB or per slot makes it cost prohibitive in most scenarios.  I would certainly never expect to see a USP V with only SATA drives.  High performance and expensive infrastructure for low performance cheap(er) disks.  We all agree that makes no sense.

    I also think that under normal circumstances licensing is geared toward using external storage….  So yes I think there is an element of “vendor push” toward using external.  Now of course this is good for the vendors because they get to ship more midrange arrays.  But as well seem to agree, it makes sense from a technical and cost point of view too.

    In the situation Im currently in, it actually makes quite good sense as floor space in the Data Centre is at an absolute premium, and power and cooling is a real issue.  This will be a 3 frame USP V with the 3rd frame barely being required.  Only 5 SATA Array Groups are currently required so they can easily be housed in the USP V.  With the space and power limitations its makes good sense to put the SATA internally in the USP V and would be very hard to get approval for an additional virtualised array.

    But as I say, this is the first time Ive seen it done, and I spoke to loads of people from HDS and HP and nobody I spoke to (initially) had done it.  Well actually the HP XP group apparently has loads of customers doing it, but wouldn’t touch me with a barge pole when I mentioned it was for a USP V and not an XP.  Guess I’ll have to take their word for it, but I’d be surprised because it makes sense in so few situations.

    Nigel

  4. Dave

    Nigel,
    Happy to provide insight on your 5 questions. Please let me know how to contact you directly (not a stealth sales scam).

  5. yveke321

    Hi Nigel,
    Very good post!  We have been asking HDS for a comparison/recommendation of using a "Tiering in the box" vs Tiering with externalised arrays(bot HDS only and 3rd party scenario’s).  So far their only response has been: it depends…
    The questions you are asking, and have answered to your best ability, are very difficult to get answered by HDS themselves.  This is better addressed by some of their competitors…
    Rgds,
     
    Yveke.

  6. Nigel Poulton

    Hi Yveke321,

    Sorry for the slow approval of your comment, Ive been away on holiday without internet access – cold turkey!

    Glad you liked the post and I hope it was helpful.  Its a constant disappointment how hard it is to get good information out of HDS.  I know they have some top top people working from them so I dont know why its always so hard – sigh!

    Nigel

  7. Barry Whyte

     I’ve always thought that because EMC don’t have another alternative, and so put SATA in DMX, everyone else had to follow suit for the tick box (more than anything) Especially with USP when you could actually buy a much cheaper external controller than use up expensive real-estate in the monolith,

  8. Nigel Poulton

    Hi Barry,

    I also thought it was just a tick box activity and Ive never seen it before on a USP V.  But there are definitely situations where it works from a floor space, power and cooling perspective.  I suppose a good thing about the USP V is that you have the choice – external or internal……..

    Nigel

  9. Garry

    I am assuming that for DKC you are using FC  and SATA only for DKU?

  10. Nigel Poulton

    Hi Garry,

    Thanks for the comment.

    Its a 3 frame USP V (R0, R1 and R2).

    The Tier 1 stuff is going on BED 1 and 2 in R0 and lower R1 and lower R2.  Obviously this is the largest loop in the box with max of 356 disks on the loop.

    Tier 2 and Tier 3 (1TB SATA) is going on BED 3 and 4 in in R1 upper and R2 upper.

    This decision was made because we didnt want to mix the Tier 1 stuff on the same BEDs/loops as the SATA.  Normally I think we would have put the larger slower disks on the larger loop but we couldnt do this because there are too many Tier disks to fit on the smaller loop so putting them on the larger loop was the only way to keep them isolated from the SATA.

    Im intruiged as to why you would suggest putting the FC in the DKC (R0) and the SATA in the DKU???  Is there a reason you suggest this?

    Nigel

  11. Garry

    I was under impression that you were configuring full SATA frame with no Tier 1 disk (VTL, backups target, archive target).
    I have seen a similar setup in a fully populated config (5 frames 1PB SATA) with some enterprise Tier 1disk in DKC designated for VTL, DEDUP, Backup Metadata – take your pick. Also some none dynamic pools for journaling purposes for future HUR migrations (from USP ‘s wihout V ).  That first bigger loop would have several pools (a small Tier 1  and a none dynamic pool from drives in DKC, and a big one based on SATA disk frames R1 and R2 )
    As far as not mixing, I am under impression that HDS is using switched loop on BEDs to access disk what should minimize negative effect of mixing devices on the same BED. Also it all depends on  how you will be configuring pools per loop.
    Each environment is unique and different, so my comments may not make sense in your case.
     

  12. Annmarie

    One other technical point to note is that during a sparing operation of a failed SATA drive, I believe you cannot perform HDP configuration changes across the array. The box is effectively ‘locked’.
    This could become quiet a limitation on a large multi-tiered array, considering the length of rebuild times for a 1TB SATA drive… and 2TB on the way.

  13. Garry

    Annmarie,
    You are correct that by default setting you cannot carve new V-VOLs on a frame during disk failover/rebuilds or some other maintenance/license related events. To avoid that limitation it is a good practice to have some spare V-VOLs created in advance in various (I assume standard) sizes.
    It is an on/off switch on a frame that can be changed by HDS SE. Also in the future code releases this is suppose to become a customer configurable option.  

Leave a Reply

Your email address will not be published. Required fields are marked *


*

You can add images to your comment by clicking here.