Blogketing and Catch-22's

By | July 12, 2007

FYI – Since many hardware vendors are making a song and dance (and some of them little else) over the green issue I thought Id change the font colour for this post to green.  I imagine this is actually more than some vendors are actually doing 😉 

Anyway – Barry Burke, the internets resident storage anarchist , has invented a new portmanteau – “blogketing”.  A contraction of the word blog and the word marketing.  Quite clever, and as you won’t find the word in any dictionary yet, Barry kindly provides a comprehensive range of real world examples in his own blog – just read any of his posts, he’s quite the expert.   But hey, who of us doesn’t dabble in a little blogketing from time to time 😉

Barry takes issue with Hu Yoshida’s not so subtle marketing of several Hitachi products in his recent blog titled “Take back you storage ”.  I think its safe to say all readers of Hu’s blog will agree that this one may be a little over the top on the marketing.  Although it’s a step up from bitching about the competition – something which Brocade seem to be doing a lot of these days – I hope the corporate blogs out there don’t descend into pure marketing machines.  Of course we expect some but lets keep it in check. 

The strange thing is, it only seems like 5 minutes ago when some of us were bemoaning the lack of energy that HDS was putting into its marketing .  And here we are now, seeing them regularly lambasted for being overly energetic with some of their marketing.  Not to worry though, I imagine in due time they will master the art.  As for Hu, he may be a little unpolished when it comes to slipping in the marketing blurb.  However, I’m confident if he reads any of the EMC blogs out there, from the likes of Barry and Chuck , he will quickly learn the finer arts and hone that skill of subtlety 😀

Actually, did anybody else notice that Hu mentions the HP and Sun versions of the USP.  Im not sure Ive noticed him do that before.  Hmmmmmm!?!?

Anyway, enough of the banter, that’s not why Im putting pen to paper tonight….

Barry has recently nailed up a list of what he and EMC have learned about thin provisioning through a year or so of experience implementing Calerra.  He is referring to it as a list of catch-22’s for thin provisioning.  A list which he occasionally mentions and has asked for feedback on. 

A while back I was going to provide some feedback, but at the time I was too engrossed in series 1 of Heroes.  Now that I’m finished series 1, I’ve a whole load more time on my hands so here’s some feedback –

 

Barry states that bad things happen when somebody carelessly dumps their entire MP3 collection onto a thinly provisioned LUN.

Well, ….yes bad things certainly could happen….  I suppose it depends on your interpretation of bad.  Bad things will not happen to your critical systems unless you removed your brain while designing your environment and planned your pools and storage layout so badly that your mission critical systems share a common pool with your user’s home drives and the likes.  I don’t think you will find that design in any best practice guides (even for Calerra).  So the simple answer to that being; the use of multiple pools, good planning and good storage admins.  Nothing new there then.

 

Barry also states that bad things will start to happen if the PO for important additional storage gets held up, or the drives don’t arrive on time.

That one I have to agree with.  However, I have seen and worked in environments where delayed PO’s or shipments of disks have caused problems even with standard LUNs that aren’t making use of thin provisioning – if you need space and don’t get it in time you will be hurt.  So this is not a problem confined solely to thin provisioning.  Of course the situation could be made worse with the use of thin provisioning – everything sharing the effected pool could take the hit.  But remember, not all LUNs will be thinly provisioned and you will likely have multiple pools to mitigate this.  So again, good design and good management can help reduce the risk.

 

Along the same line, of bad things happening, Barry states that this will also affect “critical applications that can’t be allowed to EVER get a ‘disk full’ error”. 

Again, that would only be the case under an extremely bad design.  Of course, for the purpose of scare mongering, Barry will mention things like that in his blogvert, but in real life Im certain he would never suggest such a bad design.  I mean who in their right mind would thin provision LUNs to an app that “can’t be allowed to EVER get a disk full error”.  Even if you did, you would make sure it had its own pool so that it could realise the performance benefits of wide striping (Hitachi implementation of thin provisioning) while being protected from other systems misuse of pool space.

 

Barry then points out that deleting a file in most filesystems does not actually delete the file, never mind allow the space to be reclaimed by arrays supporting thin provisioning.  He also points out that filesystems often avoid re-using space occupied by deleted files until there are no other free blocks  available.  Essentially, from the OS point of view it may appear that less pool space is being used than actually is.

I agree again, in part.  I just think this kind of thing can be easily overcome with the use of an agent on the host that provides a mor

e informed view to the server administrator – include them in the wider picture….. you know, talking to the other guys in the IT department and working like a team 😉

I also see that HP are talking about the EVA supporting new functionality in Windows Server 2007 that enables shrinking of LUNs……  although I haven’t had a chance to look into this, I wonder if it could provide a solution to the issue of reclaiming unused space in the future.  May be somebody more informed can elaborate on this Windows Server 2007 functionality?

I also see a plus side to not reclaiming space occupied by deleted files, but Im saving this one for later.

 

Along the a similar line he mentions that Windows Vista keeps copies of multiple versions of files on disk to assist in restores, consuming more pool space.  

Firstly I don’t know of anybody SAN attaching Windows Vista.  But when they do, either disable this feature, or if that’s not possible, don’t thin provision LUNs to those hosts until it is possible.

So far not much that can’t be avoided with good planning and execution……

 

Barry then says that performance can be significantly reduced by thin provisioning LUNs

Well….. show me a storage admin task that doesn’t affect performance when executed badly?  Again nothing new there.  On the flip side however – if you do thin provisioning right, at least on Hitachi storage, performance can actually be improved! 

Barry bases his point on the premise that improved storage utilisation boils down to storing more data on the same number of spindles.  Obviously disks have a limited number of I/Os they can service, and squeezing every last inch of space out of a disk will often push a disk past its I/O limits.

He actually gives an example to illustrate the point.  But what he doesn’t seem to have noticed that he has already provided us with a solution ………  Earlier in his post he made several references to unwanted data being dumped onto thinly provisioned LUNs – user dumping MP3 collection, routinely deleted files, may be even runaway apps that don’t need to keep all of the data.  He has also pointed out that this space cannot easily be reclaimed by the array.  Now although this may at first look like a bad thing for thin provisioning, it can also provide some performance benefits. 

Basically many hosts will consume an amount of space that they don’t really use.  Beautiful in its simplicity (err well…).  

So we have a problem providing a solution, does that qualify as circular logic Tongue out and make it onto the Catch-22 list?

And if you create your thin provisioning pools from slices of physical disks and use the other slices for normal LUNs, you won’t hit the disks inherent IOPs limit so quickly, if ever.

And what about the trend of larger and larger disk drives?  They bring with them the very same IOPs problem that Barry is talking about – as drives get bigger we tend to put more LUNs on them and demand more IOPs from them, but their underlying I/O limitations do not increase in concert with their size.  I haven’t seen Barry bemoan the fact that the Symm supports huge drives which bring the same problem.  I seem to be the only one who cares about that !  Go figure.

Anyway, time to wrap this post up.  In my opinion, Barry has painted a picture of thin provisioning, and at first glance it might look good, may be even a masterpiece.  But to the discerning eye, there is a lot of white space left untouched.  May be he knew that he hadn’t finished the painting, after all he did ask for feedback and was no doubt trying to add some balance to the debate at the time.

I do wonder if this will end up being like the RAID6 debate?  “RAID6” being pretty much a cuss word to many an EMC’er.  Until they joined the party that is Laughing

And although my knowledge of the thin provisioning is based on the Hitachi implementation, lets not forget that there are far more established thin provisioning players out there and they seem to be making a success of it.  If you read Anil Gupta’s blog you will see that he is seeing that 3PAR has a lot of satisfied customers.  So thin provisioning has to work for some people.

Nigel


6 thoughts on “Blogketing and Catch-22's

  1. the storage anarchist (barry burke)

    Thanks for noticing! And for taking the time to provide another perspective – definately a valuable addition to the discussion.
     
    But you do realize that you’re actually making my point for me, right ?
    When you combine all your work-arounds to the problems I’ve identified, you come to a very simple conclusion: to avoid the problems I’ve described, don’t even THINK about using thin provisioned volumes for anything important. Or at least, put important things each into their own private pool, so they can’t be damaged by anything else (which sorta reduces the ROI for thin provisioning).
     
    And as relates to the reclamation of space, the Longhorn/WS2K7 shrink LUN stuff is a good start, but many have already noted that it’s an expensive way to reclaim space – basically you’ll have to compress the used part of the volume back into the beginning of the LUN (ala SpeedDisk), then "free" up the unused end of it. And then, you have to re-extend the thinly-allocated volume again (to get room to grow again).
     
    NetApp is apparently offering a host-based tool with SnapDrive that does just what you describe – walk the NTFS volume and return any blocks no longer in use by the file system to WAFL (although they developed this really to get around another problem altogether related to Snap Pool management). The question is whether or not you really want to add an agent like this to every single host in your environment – in my experience, customers are reluctant, if not downright resistant to adding agents to their servers – especially one that are going to twiddle the bits in their file systems.
     
    I also don’t disagree that there are a lot of happy 3par customers. Oddly, though, I’ve found that most 3par customers aren’t using thin provisioning capabilities for very much of their actual storage allocations – most are using traditional fat allocations for the majority of their LUNs. And there seem to be near universal complaints that the local- and remote- replication capabilities are different for the two allocation schemes.
     
    Finally, I don’t think thin provisioning is like the denial experienced around RAID 6 at all…in fact, EMC has already said publicly (at EMC World) that we’ll implement thin provisioning across all of our platforms, not just Celerra. Not a matter of "if", just "when."
     
    The whole point of my discussions on this topic is to provide some semblance of a balanced view of thin provisioning . Left unchecked, the blogketing would otherwise be misleading at least some people into using the technology in places where it’s not appropriate. And this risk is most likely not the experienced storage administrators, but their management.
    Think about it – what if your CIO only purchased 30% of the storage you said you needed because he knew that 30% was your historical utilization rate, and he believed that thin provisioning was the way to cut costs? Don’t laugh – I’ve already seen it happen (almost) in the real world.

  2. stephen2615

    Where can I get a copy of Heroes?  I need to take the chill pill that Nigel obviously took for a while.
     
    As far as thin provisioning goes, I finally figured out that out of the two hundred of so hosts on the SAN, about 10% of them may contain unstructured data.  The rest are database systems that tend to use a high proportion of the "disk" that is given to them.  After a discussion with a number of architectural people over the last few days, I intend to lay down a more comprehensive request system for space on the SAN.  I wont accept "give us 4 TB" anymore.  I want to see actual figures that state the application usage and expected growth.  One of those 4 TB requests actually ended up only wanting about 800 GB with no real growth expected in the next year.  Sure, if they need more space, then I will give them another LUN when they really need it.
     
     
    Unstructured data such as file servers, etc is a right real pain in the butt.  But instead of giving more space on the arrays, we are looking at archiving to less expensive solutions.  We also have a strict policy on MP3 files, etc.  If they are on the network drives, you could loose them and more often than not they are gone after "automated" prompting from the data management staff.  They all end up on the users PC but then I don’t care about that.
     
    So, yes no matter what terrific products come out, good planning is always necessary to over come issues.   If I did not do any planning, I would spend a greater proportion of my day listening to those MP3’s on my desktop and perhaps watch Heroes???
     
    So, would thin provisioning make a lot of difference in a mostly structured data environment?
     
     
    Stephen

  3. the storage anarchist

    The MP3 thing was just a metaphor for unexpected junk being dropped onto your shares. I know in reality MP3’s aren’t going to be the culprit in most well-run shops – at EMC, we’re even blocked from saving a file with the .MP3 or .AAC suffix on a file share (don’t ask me how).
     
    The point is that you don’t really have a lot of control over how much data gets dropped onto a volume, thin or not. Could be umpteen versions of a PowerPoint or Word document as they go through revisions prior to delivery – or whatever. And once they’re there, it’s too late – the space is allocated, and probably won’t be freed or re-used if you delete the files after they’ve already landed.
     
    I know many 3par customers have in fact moved back to "fat" partitions simply because of the storage entropy adage:  "data will to fill all available space"…if you let users think they have 2TB, they’ll be much more sloppy about deleting stuff they don’t need than if you restrain them to 200GB.
     
    In a mostly structured data environment, thin provisioning has less of a value for improving utilization, specifically because the database engines tend to pre-allocate all available space. Still, it is easier for the DB admin to expand a table in place than to relocate tables into new LUNs. So it can stil have value in that it could allow you to defer the acquisition of physical storage until you actually need more storage in the allocation pool (depending on how much you care about the workload of your DB admins ).

  4. Nigel (mackem)

    Stephen,

    Im pretty sure you can download Heroes from iTunes.  Believe me, as good as blogging is….. it’s way downstream from Heroes!

    As for how valuable it is in a mostly structured environment – As far as the space saving side goes, I dont see it being as useful as with unstructured data.  A little like CoW, I feel that Thin Provisionings space saving features are for your lower tier type apps.  However, negates the need to manual expand a filesystem etc.  Then there are also the potential performance benefits you can get from the Hitachi implementation it may also be suitable to higher tier apps form the performance point of view.

    Barry,

    Yes I realise that in some respects Im reinforcing some of your points. 

    I wasn’t trying to say that Thin Provisioning is everything that the marketing and blogketing guys are making it out to be.  My main aim was to point out that it’s a tool, like many other technologies such as the recently discussed CoW, that can be used for good in many situations.  Oh and like everything else in IT/storage, with good planning and correct selling it certainly can yield benefits.

    While the Hitachi implementation is very young at the moment Im pleased to see it.  Its only a matter of time before we see bigger FC disks supported internal to the USP, as well as support for external storage (they already support external storage for CoW).  And Im interested to see how they integrate it with replication and copy services.  Going forward I imagine all good storage arrays will support Thin/Dynamic provisioning, inline de-dup and several other nice technologies.

    As for the installation of agents on hosts, I know that customers generally don’t like this.  But many HDS customers already have HDS agents installed.  So may be upgrades to these agents could include some of this functionality.  Of course I appreciate how low level and complex filesystem drivers are so know this is not easy to implement or to sell to a customer.

    Oh and I see you really are busy keeping everyone out there in check regarding Thin Provisioning (http://www.byteandswitch.com/document.asp?doc_id=128609&f_src=byteandswitch_default).  I had actually penned a response to Mary but then I saw your comments and decided they covered pretty much the points that I was making 😉

    Nigel

  5. Nigel Hall

    With regard to structured data and thin provisioning, I suspect we’ll see things change quite soon as application and database vendors see the advantages of using thin provisioned volumes. Or more likely, as IT customers start demanding support for thin provisioning. I think Tony Asaro said it first, but all arrays will likely support thin provisioning in the future. It just makes so much sense.

    As for supporting structured data today, 3Par have a white paper on their web site demonstrating how the Oracle 10G Auto Extend feature works with their thin provisioning product – http://tinyurl.com/34964x. I’m not an Oracle DBA, so maybe there are some gotchas in there somewhere, but it looks like it makes sense. Anybody tried it?

  6. Mel

    Hi guys I watch Heroes and other get US and UK TV shows online for free over at this website called Oliv3r.net the address is http://www.oliv3r.net. All the files are in HD so you get really great copys! They also have a image gallery full of images with Heroes Season 2 in there, http://www.tvgallery.netI think you will be happy with the links.

Leave a Reply

Your email address will not be published. Required fields are marked *


*

You can add images to your comment by clicking here.