Grrrrrr big beefy manly SANs

By | May 5, 2007

Off the back of a few posts Ive been reading recently, including Snigs post below on Brocade Oversubscription, I just thought Id sound off a little on the networking side of SANs.

I for one am no fan of ISLs.  They are the bane of too many a storage admins life, at times mine included.  But because most data centres are planned out on the back of a standard sized postage stamp, they are often a necessary evil just to keep things ticking over.

But I didn't always think like this.  You see, way back (well its not that long ago actually but it sounds better this way) when I was first starting into Storage Area Networking I really liked the "networky" side of SANs.  ISLs, trunk groups, trunk masters, name server merges….. you know, the type dark arts that brought us respect amongst out IT peers and drove rates up.

I'd previously done a bit of Cisco IP networking and because of that, I got on well with the network guys and we had a bit of Windows vs Unix type banter going.  For example, I liked the fact that "my switches" (SAN switches) usually cost a lot more than the cheapex ones the network team bought.  I also liked the way that frames absolutely screamed through my switches without touching the sides.  Whereas the frames that passed through the network guys IP switches strolled in comparison, many of which stopped for a cup of tea on their way, and in fact some of them frequently went missing en route.  Not mine though, no "missing frames adds" on the walls of my switches.  But still, the network guys had one thing over me that I didn't like.  Their networks were much bigger and complicated than mine, and their diagrams were so much more impressive than mine (back in those days "complicated" and "impressive" were the same thing, for me at least).

So I really liked the idea of connecting lots of switches into a single fabric (dual fabric design of course) and have the associated impressive (read complicated) diagrams hanging off the walls around my desk.

Fortunately those days are gone.  Ive grown up a lot since then……. errr….well……  Anyway, these days simplicity is the key.  Ive just seen too many irritating little problems with ISLs and bigs SANs.  And troubleshooting SANs is just so damn annoying at times.

One thing that had a real influence on me was some work that I did for a telco company who have a policy of no ISLs in their SAN environments.  A lot of people initially sniggle at the idea, and to be honest, I too raised an eyebrow when I first heard this.  But I have to say that in their SAN environments problems were noticeably fewer and farther between, and when they did occur they were so much more limited in their scope and so much easier to troubleshoot.

Just off the top of my head the following lists some of the advantages that I see in smaller SANs, especially SAN islands –

  • Firmware upgrades are so much easier.
  • Other maintenance is much simpler
  • Troubleshooting is much simpler
  • Change Management people are far more comfortable with small SANs
  • Performance is better
  • Management and monitoring "can" be better in some ways

Sure, there are drawbacks, even with today's technologies.  I just wonder how many of the future SAN related technologies, I'm thinking particularly services, are going to be geared around larger interconnected fabrics and networks?  Methinks that the SAN is no longer merely an interconnect, and that ISLs and the other dark arts mentioned above will still be plaguing us in the years to come.

Feel free to educate me.


8 thoughts on “Grrrrrr big beefy manly SANs

  1. stephen2615

    In my last job, I decided to make a single fabric with all switches in it.  Our (not so SAN orientated) project manager had kittens and tried desperately to stop me from doing it but heck, I am game so I did it.  I did not do it with no knowledge of what I was doing.  Extensive training and testing were performed before I turned it on.

    This was a largish core/edge Cisco solution.  I used a special non trunking VSAN for the management traffic.  It make my life quite easy.  Cisco were a bit coy on this setup and could only state one problem with it as far as non hardware issues and that was the chance of FCNS dying but it will automatically restart.  Considering, each VSAN has its own, I took that chance.  IF the principal switch died, I could have an issue or two but such is life.  The other core switch would take over and it still offered an alternative path for all the systems.

    Troubleshooting the SAN was easy but they were only minor issues that made me even more aware of what I was doing.I had the odd ISL and some were quite big in size.  I quite liked my setup.

    The network people never cared about the SAN switches until they saw the MDS 9513 and as it is big (in size and performance), I feel they felt a bit let down by their own equipment.  Some even tried to take over the SAN but I told them what I thought about THAT!!!

    Mind you I had to sit down and talk about the Optical network when it came to our DWDM native SAN extension.  So, in a round about way, they are part of the SAN.

    In SAN fabric services are there but no one seems to want to use them.  I was very keen on the Cisco Network Accelerated Serverless Backup but I think no one is game to use it due to the cost.  Sooner or later, fabric services will take off.

    I tend to think that with the ever expanding IP performance,  FC SAN's  might  have a limited lifespan or they will decrease.  I doubt it will be in the next few years but eventually, FCoE and iSCSI will take off.So, in the interim, give me big and complex SAN's…. It justifies my pay.


  2. snig

    I think the biggest problem people have in troubleshooting an environment is not understanding the entire design.  But, when you design something from the ground up and have to troubleshoot a problem it has never been a problem for me. 

    I love designing huge, complex environments.  But, when simple works I find it easier to go that route.

  3. stephen2615

    How do I give my postings paragraphs?  It looks messy when I post.

  4. snig

    Just hit the Enter key.  I’m using Firefox and everything works fine for me.

  5. Nigel (mackem)

    Stephen, looks fine to me 😉

    PS.  I always have problems with formatting if I cut and paste from Word into WordPress – the paragraphing always disappears.  So I always have tp type directly into wordpress.

    Hope that helps

  6. c2olen


    your comment like "my box is bigger and faster than yours" above triggered me.
    Before I started working in my current environment (Cisco/EMC)  I too thought that SAN switching was always/better faster than the IP kind.

    Needles to say I was VERY supprised to see that this shop tends to do their synchronous replication over IP, and the asynchronous replication over FC. Both over 200KM links. The IP features like fast-write acceleration and compression seems to speed IP up quite a bit over this distance.
    The FC latency is about 5ms over this distance, so the synchronous replication over FC is simply not feasible.

  7. stephen2615

    Needless to say I am gob smacked that FCIP synch replication is better than native fibre synch replication.  Surely, there must be some spoofing going on there.Heck, even 10 ms would keep me happy with our Exchange geo clusters and True Copy over 200 km.  I think it is another EMC/Cisco conspiracy theory that needs to be proved wrong.  Don’t get me wrong, I love the Cisco gear.Stephen

Leave a Reply

Your email address will not be published. Required fields are marked *


You can add images to your comment by clicking here.