Earlier this week Dell wrecked my day when they announced their intention to acquire 3PAR, the company that – in my personal opinion – has the best storage array architecture on the market.

Why did this wreck my day? 

Personally I felt that 3PAR, and their top-notch architecture, deserved somebody better than Dell – no offence Dell.  Let’s face it though, Dell isn’t exactly synonymous with great enterprise storage…  If Id had my pick, I would have preferred HP pick up 3PAR. But hey, life goes on.

Best of Breed

However, now that I've had a chance to think, I’ve made an about-turn and changed my opinion.  Here’s why -

Dell already have a quality iSCSI array in guise of the Dell EqualLogic line, and I think that iSCSI has a great future in the SMB space.  Big tick in that box for Dell.

Now they have a best of breed FC block storage array that can compete with the Symmetrix and USP gorillas of the storage world!  And let’s face it, while it might not be growing at a huge pace, block storage shows no signs of going away in the enterprise space.  And even if FCoE eventually gets its act together and gobbles up the FC market, the 3PAR technology will adapt to that as easily as any other FC block storage array out there.  So, big tick in the Enterprise storage box for Dell too.

In fact, Dell now look better positioned than HP at the high end.  HP currently still OEM the Hitachi enterprise line of products in the enterprise block storage space.  These days companies are scrambling to own the stack, and Dell owning the 3PAR technology looks better than HP OEM’ing the Hitachi technology – emphasis on “owning”.

Let’s also remember that a few weeks ago Dell announced their intention to acquire Ocarina Networks.  I’ve been to Ocarina HQ, talked with some of their guys and know a little about their technology.  Smarts like those provided by the likes of Ocarina that allow for compression, de-dupe and other means of storing our data more efficiently have a great future in my opinion.  If Dell are able to integrate the Ocarina IP into the EqualLogic and 3PAR lines then they will have a really compelling story.  So that’s another big tick in the box for Dell.

With the above in mind, the Dell storage portfolio looks to be at serious risk of becoming synonymous with “Best of Breed”.  Surely not! :-S

What’s missing

With so many ticks in boxes, what is still missing for Dell?

Well, a line-up that includes a top-notch block array (3PAR), a top-notch iSCSI array (EqualLogic) and some top-notch storage optimisation technologies (Ocarina) deserves a top-notch NAS device. 

I’m not convinced that Exanet fits the top “top-notch” moniker in today’s ultra-cometetive scale-out NAS world

Looking at what is potentially available on the market, Isilon Systems must be in the potential line of fire.  They recently became a Billion dollar company so they won’t be cheap, but heck, you get what you pay for right! 

I remember hearing rumours that HP’s first choice scale-out NAS acquisition might have been Isilon when they went shopping for scale-out NAS and ended up buying IBRIX.  So, what better way to land a heavy kick to the cahoonas of one of your most fierce rivals than to go out and land something they supposedly weren’t able to land.  Of course I am only speculating that HP would have preferred to buy Isilon!  Let me just add that I personally like the looks of IBRIX, or the X9000 range as HP now call it.

Anyway, adding Isilon to the Dell storage portfolio would pretty much give Dell a store line-up to be reckoned with.  Granted, a million miles away from being integrated with each other, but nonetheless really cool and compelling technologies.

Without any doubt though, if Dell are to complete the acquisition of 3PAR and then went on to pick up someone like Isilon, then we would be waking up to a world with a new storage superpower called “Dell”.

All that would then remain in order to truly compete with the likes of Cisco, HP and IBM would be a networking platform – may be something innovative and different like Xsigo might be in order?

Oh and of course they would need to go out there and invest in acquiring a decent server platform

PS. You can join in the real-time tech talk by following me on Twitter – @nigelpoulton

• Dennis Martin (@demartek)
• Dave Graham (@davegraham)
• Stu Miniman (@stu)

For an hour solid, we talked FCoE and the realities of deploying the technology.

As well the the recording of the call, which I recommend listening to, there will be several short resultant articles posted on Wikibon covering topics such as -

• CIO Considerations
• CTO considerations
• Operation and organisational considerations
• Getting Rid of Stuff – what can FCoE help me throw out.

This was the first ever Peer Incite call to hit the 200 maximum participants limit.  Proof that FCoE and network convergence are hot topics in the Data Center.

If you’re considering deploying FCoE, then I recommend you and take a listen.

I can be contacted via the Contact Me page or on Twitter at (@nigelpoulton)

Nigel

In this inaugural episode of the Technical Deep Dive Podcast I'm joined by Greg Ferro from Etherealmind.com and Camden Ford from Xsigo Systems.  Cam talks to us about Xsigo Systems and the technologies and solutions they offer.

This is available as a two-parter due to length.  Part 1 is about 30 minutes long and Part 2 is about 40.  What can I say, you cant dive deep in 20 minutes!

Comments welcome!

As early as last summer I was involved in specking and purchasing some HP BladeSystem Matrix based solutions – essentially a shrink-wrapped Rack based solution that had compute, networking, storage and management tools.  Granted, HP Matrix is an early attempt and not much different from the norm, but a step towards the Rack Area Network (RAN).

So… Rack based solutions are on their way, and the way I see it – from speaking with peers, customers and vendors – the following two high level Rack based designs will be predominant and will slug it out over the next few years:

1.  The FCoE RAN Solution

Of the two solutions, this one most closely resembles what we know today.  The only major difference being the use of FCoE between the server and the Top of Rack (ToR) switch.  This solution requires Converged Network Adapters (CNA), copper twinax cabling and FCoE ToR switches.  As it happens this is really the only practical place that FCoE can currently be deployed.  Fortunately, however, the FCoE products in this space (the RAN) are maturing quickly – we already have 2nd generation, single chip, single driver code base, high performance CNAs shipping and supported by most good server vendors….

In the FCoE based RAN solution there is very little in the ways architectural change – no blurring of the server/network edges and no change to the design of servers or networks.  This gives the comfort factor. 

Anyway, the sketch below shows a high level view of of this type of solution -

2.  The IOV solution

This second solution is slightly more innovative.  It requires slight changes to existing server/blade designs, there is some blurring of the server and network edges, and some re-thinking of network design and management is required.  Not quite the same comfort factor that the FCoE RAN solution gives,but as the saying goes – No pain, No gain! 

The IOV solution can be summarised in the following -

Servers and blades are reduced to pure compute and memory.  The I/O components are disaggregated from the server chassis and re-housed in an external ToR I/O Director.  Servers connect to the external I/O cards by either PCIe cables or IB.  These I/O adapters can be CNAs or traditional NICs and HBAs.  They are next generation in that each one can be carved into multiple logical adapters which can each be dynamically assigned and unassigned to any server and VM within the Rack.  The I/O adapters and I/O Directors have built-in switching functionality, enabling traffic to be switched either within the I/O adapter or between adapters within the same I/O Director without the need to travel up to a traditional network switch (hairpinning in the adapter or I/O Director).  Essentially, access layer switching will be moved on to the PCIe I/O adapter!

The diagram below shows this at a high level -

No room in the RAN

Personally I like the idea of using PCIe as the main interconnect within the Rack.  Every chipset on every server already has a bunch of PCIe bandwidth that is essentially….. well…. FREE!  Who doesn’t like the sound of that!?  10Gbps CEE and FCoE licensing of ports is … well…. definitely not free.

Of course there is the other side.  PCIe muscling Ethernet out of the RAN will not go down well with some, nor will implementing switches within NICs/CNAs and I/O Directors.  Not only will this tread on certain vendors markets and margins, it also brings with it several network design and management challenges.  But what the heck… we grow from our challenges and come out the better for it – right?  Point being, knee-jerk self-preservation type reactions from the network guys should be expected

Final thought on FCoE

While the IOV solution could potentially muscle Ethernet out of the RAN, it can still branch out to FCoE switches in the core.  So either way, FCoE will play a role. 

And if we are being creative, we could run our I/O up to the ToR I/O Director over PCIe within the RAN and then branch out via a CNA in one of those I/O Directors to a core switch with FCoE ports.  One way of utilising FCoE ports that are currently available in core switches.

Interesting times!

PS. I will be featuring on the Wikibon FCoE Fact vs Fiction call on 2nd February along with Stu Miniman, Dave Graham and  Dennis Martin  If you’re interesting in FCoE put it in your calendar.

The term, as well as the concept, of Rack Area Networking is one I’m hearing more and more often.  As a result of this, as well as the fact that I’m convinced this is going to be one of the most interesting and important areas of Data Center computing over the next few years, I’ve decided to write a mini-series on the topic. 

This post is instalment number 1 and is intended to introduce the concept and get the ball rolling.  The whole thing is a bit of me thinking out-loud and attempting to generate some awareness and conversation around the topic, as .  So please pitch in!

Rack Area Network – the concept

For me, Rack Area Networking, or RAN for short, is an umbrella term for most of the clever networking and I/O virtualization stuff that goes on within a rack – a 42u rack. 

With it being “Rack Area”, it is a close proximity network and as a result operates over very high-speed low-latency interconnects.  

Physically, RAN technologies include a new generation of at least the following: I/O adapters, cabling, Top of Rack (ToR), and may be even End of Row (EoR), switches.  However, for reasons which will become clear, the emphasis is heavily on the clever – technologies that enable the flexible, the dynamic and the virtual aspects.

For example, the I/O Adapters driving the RAN evolution are not just faster than the legacy adapters they are replacing, they have built-in cleverness – hardware virtualization and huge flexibility! 

Some of the other technologies that define and operate within the RAN include – SR-IOV, MR-IOV, vNIC, vHBA, CNA, FCoE, Hairpin-turns, switching in the adapter, VNTag, VN-Link…. just to name a few.  In future posts we will discuss most of them.

As well as the above new hardware and technologies, the RAN also requires and includes a new generation of management software and functionality.  True value is often in the software – the glue that holds it all together and makes it all happen.

The best part being, there are early RAN technologies already out there in the wild.  And they are already delivering real-world tangible benefits.

Some technologies driving the evolution…

It’s really important to note that while technologies in the RAN are experiencing a period of accelerated evolution, it is most definitely an evolution.  The changes are happening fast, but they are not huge disruptive changes.  For the most part, they are improvements and enhancements, albeit major, on what we already know and are comfortable with.  E.g. take PCIe adapters and create multiple virtual adapters (vNIC and vHBA) in hardware….

Just a few of the currently shipping RAN technologies include -

• HP Virtual Connect Flex-10
• IBM Virtual Fabric Solution w/ Emulex UCNA
• Cisco UCS w/ Palo adapter
• Xsigo I/O Director
• Virtensys VIO switches

NOTE: Let me know if I’ve missed any major RAN technologies off the list

Some of the above technologies are very much generation 1 and only a small step towards the RAN, bringing only small benefits.  Whereas others are a major step with huge benefits.  All vendors are scrambling to take the lead in this evolving area.  In later posts we’ll dig deep into most of them.

Blurring the Lines and Causing Havoc

Naturally, many of these technologies are challenging and threatening the traditional server/network edge configurations we are used to. 

Hairpin turns, switching in the adapter and avoiding edge switches are just some of the paradigm shifts that RAN technologies might force us to consider.  Such topics are the subject of intense and engaging debate.  All very interesting and some of the concepts are very cool!

In upcoming posts we’ll talk about the likes of – SR-IOV MR-IOV Hairpin-turns VirtenSys Flex-10 VEB Xsigo VNTag NextIO VNLink InfiniBand PCIe

Thanks for dropping by and feel free to throw in your penny’s worth.

Nigel

I am available as an independant freelance consultant and can be reached via the Contact Me page.

If you're thinking that I might just bemaking some of this stuff up or inventing buzwords, then you need to check out my follow-on RAN and IOV related posts listed below -

IOV vNICs and vHBAs

IOV and introducing hairpin turns

In my opinion the show was a real success with over 1,500 attendees, of which over 1,100 were end users and reseller delegates and the remainder made up of general riff-raff such as vendors, press and the likes….

One of the things I like to see at shows like these is hardware.  What can I say, Im just the kind of guy that gets a kick out of seeing hardware.  So for the rest of you out there like me – sit back and enjoy……..

First up, I was impressed to see a Symmetrix V-Max, blue strip light ablaze.  The only disappointment was that whenever I popped by to have a chat, somebody else was always being given an overview

Next up the IBM HS22 BladeCenter that was used to demo the new IBM Virtual Fabric, which is based on technology from IBM, BLADE Networks and Emulex.  A much needed addition to the IBM portfolio in my opinion.

Oh and while on the theme of IBM, here is another rack of IBM kit

As the IBM Virtual Fabric solution has an Emulex CNA in it, next up is Shawn Walsh from Emulex showing us that the Emulex UCNA is real and not a myth.

And a close up on a desk with a pen for lined up to give an idea of size

Then if you followed the girls with “Thin” written on their T-shirts you couldn’t miss the 3PAR InServe kit on show.  I plan on writing about 3PAR RAID MP and Persistent Cache, both of which are potentially very interesting technologies.  But seeing as 3PAR are attending the upcoming GestaltIT Field Day I might wait and see if I can glean some deep tech info from them.

Brocade also turned up with a rack load of kit, although hugely disappointing for me was the lack of an FCoE 10-24 blade in the DCX director.  Not to worry though, there was a B8000 top of the rack CEE/FCoE switch to keep me happy.

And a Brocade dual port CNA

I have some video footage from the Brocade booth that I will post some time next week.  So stay tuned.

Even the internet connected laptops were also of decent spec.  Below is a smart little HP laptop alongside my personal 11.1” Sony job – check out the grease marks on my trackpad and spacebar

And last but not least, the quality of freebies was good.  Aside from the standard pens and stress balls, I was particularly impressed with –

iPod Shuffle
Solio solar powered USB charger
3-in-1 pen/laser pen/1GB memory stick (James Bond style)
2GB micro SD with standard SD adapter 

The economy must be recovering!

As well as the above mentioned video footage from the Brocade booth, I also got some video footage from the Arista networks booth.  Keep an eye out for that as I plan to post them in the following week…

Nigel

You can follow me on Twitter where I talk about storage technologies (@nigelpoulton) and I am also available for hire as a consultant.

In the video, William Lloyd Scull Senior Network Architect at BLADE Network Technologies, demos a feature of the IBM BladeCenter Virtual Fabric (comprised of Emulex OneConnect UCNA, BLADE Network Technologies Blade Switches and IBM BladeCenter hardware).

In the video William dynamically reduces network bandwidth allocated to a vSwitch from 2500Mbps down to 1900Mbps, as well as talks us through some of the other features and some of the hardware involved in the Virtual Fabric solution.

Its an interesting video if you want quick and dirty overview of Virtual Fabric for IBM BladeCenter and a glimpse at what it looks like with the lid off.  The video was not rehearsed, and although Id gone through the same thing with William the day before he had no idea I would turn up again the next day with my video camera and ask him to perform to the world

So thanks to William for being a good sport.  Enjoy……..

First impressions for me are similar to my thoughts on HP Virtual Connect Flex-10…….  In fact this is very similar to VC Flex-10 but with a more clear roadmap for FCoE and is built on a solid Converged Network Adapter (OneConnect UCNA) from Emulex.  However, as good as it is, its very generation 1.  But that’s not a huge issue because these are early days in this area.  Everybody else is in a similar position.  To my knowledge there is nobody doing hairpin turns in silicon yet and we are all a long way off MR-IOV and the new rack area landscape which that makes possible……..

So…. an important technology for IBM, may be slightly better than Virtual Connect Flex-10 but is probably not quite as good as some of the stuff Cisco is doing with Palo… But a solid foundation to build on (see my previous post on the Emulex OneConnect Universal Converged Network Adapter for more info).

Thoughts and comments welcome as usual…..

Nigel

Oh and you can follow me on Twitter where I talk about storage technologies (@nigelpoulton) and I am also available for hire as a consultant.

Most will know by now that Im all for converged Ethernet unified fabrics.  Still, Im well aware that some of the vendor offerings are very “Generation 1” and have that version 1.0 feel to them.  Todays launch from Emulex honestly didn’t have that feel to it.  In fact it seems that the products shipping around CEE are getting more and more mature by the day.  So let me take a few minutes to discuss some of the product highlights…..

Architectural Overview

At a high level, the OnceConnect UCNA is a high performance single chip CEE adapter.

Lets qualify that statement…..

High Performance:  Sure, anybody can call their products, especially 10Gbps Ethernet adapters, “high performance”.  However, Emulex can back this up with the fact that this product provides hardware offloads for TCP/IP, iSCSI and FCoE.  This is marketed under the name vEngine (add a “v” to any technology and it automatically sounds cooler). 

Basically this adapter is a workhorse.  It will do the protocol related work for you, freeing up your CPU to do other tasks.  This enables it to be truly high performance, not just for FCoE but also for iSCSI and TCP/IP.  This is great – especially in Hypervisor estates.

Single Chip:  There is a single ASIC on the card that does the above.  No requirement for dedicated ASICs for each protocol.  ASICs also add to the High Performance claim, as ASICs are still very much the way to go with high performance I/O adapters – so called merchant silicon doesn’t quite cut the mustard. 

At the time of writing this article this is the only adapter on the market with all of the above hardware offloads on a single chip.  Although it is only fair to mention that FCoE functionality will be released later this year, but as its already late October, this will be soon.

Pay-As-You-Go and future proofing

The above is all good, but a bit overkill if you don’t need it all just yet.  Well……. You can buy the adapter as a 10Gbps Ethernet adapter, without any of the additional hardware offloads etc, at the pricepoint of a 10Gbps Ethernet adapter.  But in the future, if/when you require iSCSI offload or FCoE then you can easily unlock these features with a license.

Sounds almost perfect for the kind of staged deployments of CEE and FCoE that I think a lot of companies will adopt – Buy future proofed hardware now that allows you to keep your options open.  Deploy the UCNA now as a 10Gbps Ethernet adapter and be in a position to press ahead with FCoE etc in the future without the need to rip and replace your I/O adapters!  What is there not to like about it? 

Remember that we already do this with our switches and storage arrays, and it works well.  Why not apply the same model to I/O adapters.

This is all of course assuming that the progressive licensing doesn’t make the overall cost more expensive!!  From tweeting about this at SNW Europe today this is the biggest concern from fellow tweeters.

Oh and you can run software initiated iSCSI etc over the adapter without licensing the iSCSI offload….. 

All-in-all it seems very flexible to me.
 

NOTE: Of particular interest to me was the fact that the core features, as well as the base cost, of this adapter are 10Gbps Ethernet.  This is very interesting when you consider Emulex are traditionally a Fibre Channel company.  Clearly Emulex are moving with the market here and recognising Ethernet as the dominant technology and building on that.  Emulex also have people on IEEE 802.1 committees such as DCB.  Now that’s what I call not betting against Ethernet.

A must for the Virtual Data Center

In my last post regarding CEE I talked about the importance of CEE and its associated speeds and intelligence in the light of advances being made in Hypervisor based environments. 

I would add to that – features and functions such as those seen on the Emulex OnceConect UCNA are equally as vital….

As the number of VMs per CPU core increases to greater than 10, the I/O and networking components need to keep step.  What is the point of CPU technologies enabling huge numbers of VMs per CPU core if your I/O subsystems can’t keep up!

I think most people can accept the need for FCoE offloads as this is the norm in FC environments – and FCoE is Fibre Channel, just over a different Layer 2.  But….. offloads for iSCSI also become more and more important for iSCSI shops as they also deploy more and more VMs per physical server.
 

RuptureMonkey opinion:  Protocol offloads are absolutely vital to Virtual Data Centers.

Don’t Blink

There is no doubt that the I/O world is changing in front of our very eyes.  Be careful not to blink as you might miss something important.  Technologies such as CEE and protocol offloads (as well as many others) are key. 

Other related stuff

Also mentioned in the launch were technology agreements with IBM regarding 10Gbps Ethernet NICs and 16Gbps native FC HBAs, both for IBM Power Systems.  The 16Gbps FC design win being an industry first!  Technology is marching on and Emulex are certainly up at the front.

This is on top of the recent announcement around the IBM Virtual Fabric for BladeCenter – where Emulex, BLADE Network Technologies and IBM have collaborated to bring to market a very good blade based I/O solution, comparable, and possibly superior to, HP Virtual Connect Flex-10.  I saw IBM Virtual Fabric demonstrated in a VMware environment today and it looks a great technology.  The guys were also kind enough to rip a blade out and let me see inside one of the blades.

Finally, Emulex also announced OneCommand Manager which replaces HBAnywhere, but I havent seen this yet to be able to comment.

Random info:  Apparently World of Warcraft has ~15,000 blade servers. Cool!

Nigel

NOTE: Im using the term CEE. I should probably use DCB, but Im not.  For disambiguation of the interchangeable terms CEE, DCB and DCE see here.
 
 

 
 
Half baked

First up, and in the spirit of honesty, I have to mention that the DCB/CEE related standards are not yet fully baked.  Formal ratification is expected early to mid 2010.  However, enough is defined for major vendors to be shipping the technology.  And more than merely shipping it, some are betting large chunks of their business on it. 
 
 
CEE 10 second overview

Really quickly, CEE is currently a 10Gbps full duplex lossless Ethernet with dynamic prioritisation/bandwidth allocation and some other goodies.

Specifically -

• Priority based Flow Control (802.1Qbb)
• Enhanced Transmission Selection (ETS 802.1Qaz)
• Congestion Notification (802.1Qau)

If you know the above technologies, then you will know that they are compelling and pervasive.  If you don’t know them then trust me, they are good.
 
 
The Grand Key

CEE is a key technology of the future.  For starters it is absolutely key to FCoE.  In fact without CEE there would be no FCoE, at least no compelling FCoE.  But CEE is not just for FCoE, its features and benefits can and will increasingly be leveraged by many areas of IT infrastructure…  iSCSI as well as NFS over UDP are just a couple that are commonly talked about.

Point being, FCoE is only one of the many new options made possible by CEE.
 
 
Smarter AND faster

Yes there is non-CEE 10Gbps Ethernet.  For the purposes of this post Im going to refer to it as “Dumb 10Gbps Ethernet” (and Im writing it in pink because its for girls).  This Dumb 10Gbps Ethernet is 10Gbps Ethernet without the goodness of CEE that we just mentioned above.  Sure its cheaper than the CEE version.  But, we all know that cheaper usually isn’t better.  Pay peanuts, get monkeys.   Trust me, in the long run you will want CEE….

Not convinced?  Let me draw a parallel that I hope will help create an “Aha” moment –
 

Just like processor technology, Ethernet can and will get faster and faster and faster.  But, like processor technology, there comes an inflection point – where getting faster and faster, but not smarter and smarter, becomes almost pointless.  The change in focus needs to be towards smarter and not just faster.

Ask yourself the following question – what are the real game changers and compelling aspects of Intel’s current raft of “Nehalem” processors?  Is it the GHz?  Or is it the built-in virtualisation intelligence and all the associated benefits (Im thinking Intel VT….)?

Same goes for Ethernet, speed is not everything, intelligence matters too.  CEE brings both speed and intelligence to the network.

Ask Hypervisor architects and admins if they would like to go back to the old days of non-hypervisor aware CPUs.  Im pretty sure they will tell you where to go.  Same goes, and will go, for people deploying and using CEE.  Once bitten forever smitten.
 

RupturedMonkey statement: “CEE brings intelligence to the network.”

 
 
Hubs versus Switches

In my opinion, the whole Dumb 10Gbps Ethernet versus CEE smacks of the old Hubs .vs. Switches debate of days gone bye. 

Is there anybody wishing they still deployed hubs at the centre of their networks?  No.

Point being…. CEE is here and its changing the game. 
 
 
Cable once and most of most

All of a sudden the possibility of cable once is a reality.  CEE has the capability to run most, if not all, of most companies portfolio of network traffic over a single cable.  But more than that – over a single PCIe card and to single edge switch port.  Who wouldn’t want that? 

But there’s even more – it will simplify changes and network management, as well as bring down the cost of power, cooling and all of that jazz too!
 
 
Unabridged opinions and conclusions

So if you didn’t know previously, you know now – Im liking the look of this whole converged network unified fabric thing.  I see CEE as an essential building block of every modern Data Centre!

So with that, let me finish with a word or two to the naysayers and FUD-slingers –
 

Do yourselves a favour and don’t waste your time and energy trying to slow the unstoppable forward march of technology.  Believe me, it will roll right over you like a steamroller over a cockroach rat something tiny and insignificant   Where would Intel, AMD or any of the major server vendors be if they had resisted the march of VMware?  After all, they would all ship more units if every server image required dedicated hardware…..  Short answer is, they would be in a world of hurt.

RupturedMonkey advice: Don’t get left behind… or rolled over by a steamroller.

Its a brave new world out there, and we as IT pro's as well, as the enabling technologies, need to move with the times.  CEE is doing just that – moving things forward. 

By all means, choose to limp forward with bog standard 10Gbps Ethernet “sans” the goodies of DCB/CEE.  If you do, then good luck to you, but dont look on enviously as the rest of us race forward into the light.

Nigel

Please feel free to share your thoughts below. You can also follow me on Twitter @nigelpoulton.  I only talk about storage and technology and the conversation is often very interesting.

I received a comment on the site today from a confused reader.  The point of confusion is a common one and not something to be embarrassed about.  Part of the comment read as follows

“…..but it’s [FCoE] problem is that it consists of 3 different protocols (IP, FC, SCSI) just enlarging it’s frame size with unneccessery information. This is a step backwards."

The above is clearly a misunderstanding.  It would be accurate to say that DCB/CEE is designed to transport all three of the above protocols, but FCoE does not consist of these.  So if it doesn’t consist of these then what is it…….

FCoE in a single sentence

Put in to a single sentence “FCoE is the encapsulation of Fibre Channel frames within Ethernet frames”.

If you have a good quality dive suit then put it on…… we are about to perform a deep dive…………..

Squeezing it in

Some will know and some will not, but a standards based Ethernet frame with VLAN Tagging, including Tag Control Information can be 1522 bytes in length.  On the other hand a Fibre Channel frame can be up to 2148 bytes in length.  Clearly, this poses a problem – How can an Ethernet frame be wrapped around a fibre channel frame, if the fibre channel frame is bigger?

Two of the most viable and obvious solutions to this challenge include –
 

1. Fragment the FC frames so that they fit within standard Ethernet frames.
2. Make bigger Ethernet frames

From a performance and simplicity point of view, option 1 should be avoided at all costs due to the fact that fragmentation adds complexity to the encapsulation and extraction process.  Complexity brings overheads, which in turn would slow the protocol down – which if we remember back to my initial FCoE post, is highly undesirable in networks transporting SCSI as a payload. 

Option 2, on the other hand, keeps the encapsulation/extraction process very simple – 1 FCoE frame for every FC frame. 

No fragmentation and reassembly overhead, no tinkering with the headers or payload, and vitally, no unnecessary protocol induced latency.  These are big wins for networks transporting SCSI.  This seamless handling of frames offered by option 2 keeps the encapsulation/extraction process very simple, very lightweight and ultimately, very fast!

To put this in perspective, here is as good a place as any to point out that the encapsulation and extraction process occurs at every hop on the FCoE network (we won’t talk about multi-hop FCoE here).  In other words, every time an FCoE frame traverses and FCoE switch, it is re-encapsulated ready to be moved on to its next hop. 
 

NOTE: Due the broadcast nature of the Ethernet networks, frames need a new source and destination MAC address for each hop.  Point being, if the encapsulation/extraction process was arduous, it would significantly impact the performance of FCoE.

Option 2 also keeps hardware designs simpler, allowing for cheaper hardware than would be required if complicated encapsulation and extraction techniques had to be implemented.

Fortunately the folks on the standards body (INCITS T11 FC-BB-5) decided on option 2.

Jumbo to the Rescue

But in order to go with option 2, we need to invent larger Ethernet frames.  Well actually we don’t need to invent them because they already exist and are called Jumbo Frames. Baby Jumbo frames (~2.5KB) provide enough space, plus some headroom for FCoE frames.
 
The diagram below shows an FC frame encapsulated within an FCoE frame.
 

Please feel free to correct the above diagram if its incorrect or unclear….

Although Jumbo Frames are not an officially recognised IEEE 802 standard, in fact due to technical reason they likely never will be, they are extremely widely implemented, well understood and about as standard as anything non-standard can be.  In fact, if jumbo frames did not exist, they would have to be invented to enable FCoE networks.

The same comment that came in from a reader talked about poor framing efficiency in FCoE.  Lets take a quick look at this while we’re here.

Data is protocol overhead

By “framing efficiency” we mean the number of control bytes sent versus total number of data byes sent.  A protocol with a high percentage of control bytes would be seen as inefficient.  Because of going with option 2 from above, FCoE actually has quite good framing efficiency, as it doesn’t carry baggage required to reassemble an FC frame from multiple FCoE frames.
 

I’ve heard it said that to techie people data is protocol overhead

Keep your hands off my bandwidth

Another reader has recently asked the following question

“….how is the IP traffic and the FCoE traffic kept from consuming one another’s bandwidth…”.

The answer is to do with Ethernet Priorities and I touched on them in a previous post and in the comments to that post, but while we’re here with the diagram above lets add a bit more meat to the answer.

I’ve written VLAN Tag in the above diagram, lets break it open a little more –
 

The above is a representation of an 802.1Q VLAN field.  This field also lists the Ethernet Priority (technically referred to as the Priority Code Point or PCP).  This is leveraged by Priority based Flow Control as discussed in earlier posts and is absolutely critical in the implementation of a lossless fabric, without which FCoE would be dead in the water.  PCP can be between 0-7 and each type of traffic will be encoded with its own PCP allowing features such as losslessness and bandwidth allocation etc to be implemented.  FCoE would have one value, iSCSI another etc…..

Hope that helps clear some things up.  Let me know if you have any more questions.  I’m enjoying the discussion that this is generating, and don’t be embarrassed about asking questions(obviously I don’t know all the answers but Im sure somebody out there will).

Nigel

I am a freelance consultant and available for consulting work via the Contact Me form