FCoE: Framing deep dive

By | October 9, 2009

My current series of posts on FCoE and CEE have generated a lot of questions and feedback via the comments section and on Twitter.  In this post I will attempt to address some of the questions and misunderstandings – especially around encapsulation and framing efficiency……

I received a comment on the site today from a confused reader.  The point of confusion is a common one and not something to be embarrassed about.  Part of the comment read as follows

“…..but it’s [FCoE] problem is that it consists of 3 different protocols (IP, FC, SCSI) just enlarging it’s frame size with unneccessery information. This is a step backwards."

The above is clearly a misunderstanding.  It would be accurate to say that DCB/CEE is designed to transport all three of the above protocols, but FCoE does not consist of these.  So if it doesn’t consist of these then what is it…….


FCoE in a single sentence

Put in to a single sentence FCoE is the encapsulation of Fibre Channel frames within Ethernet frames.

 

If you have a good quality dive suit then put it on…… we are about to perform a deep dive…………..

Squeezing it in

Some will know and some will not, but a standards based Ethernet frame with VLAN Tagging, including Tag Control Information can be 1522 bytes in length.  On the other hand a Fibre Channel frame can be up to 2148 bytes in length.  Clearly, this poses a problem – How can an Ethernet frame be wrapped around a fibre channel frame, if the fibre channel frame is bigger?

Two of the most viable and obvious solutions to this challenge include –
 

  1. Fragment the FC frames so that they fit within standard Ethernet frames.
  2. Make bigger Ethernet frames

From a performance and simplicity point of view, option 1 should be avoided at all costs due to the fact that fragmentation adds complexity to the encapsulation and extraction process.  Complexity brings overheads, which in turn would slow the protocol down – which if we remember back to my initial FCoE post, is highly undesirable in networks transporting SCSI as a payload. 

Option 2, on the other hand, keeps the encapsulation/extraction process very simple – 1 FCoE frame for every FC frame. 

No fragmentation and reassembly overhead, no tinkering with the headers or payload, and vitally, no unnecessary protocol induced latency.  These are big wins for networks transporting SCSI.  This seamless handling of frames offered by option 2 keeps the encapsulation/extraction process very simple, very lightweight and ultimately, very fast!

To put this in perspective, here is as good a place as any to point out that the encapsulation and extraction process occurs at every hop on the FCoE network (we won’t talk about multi-hop FCoE here).  In other words, every time an FCoE frame traverses and FCoE switch, it is re-encapsulated ready to be moved on to its next hop. 
 

NOTE: Due the broadcast nature of the Ethernet networks, frames need a new source and destination MAC address for each hop.  Point being, if the encapsulation/extraction process was arduous, it would significantly impact the performance of FCoE.

Option 2 also keeps hardware designs simpler, allowing for cheaper hardware than would be required if complicated encapsulation and extraction techniques had to be implemented.

Fortunately the folks on the standards body (INCITS T11 FC-BB-5) decided on option 2.

Jumbo to the Rescue

But in order to go with option 2, we need to invent larger Ethernet frames.  Well actually we don’t need to invent them because they already exist and are called Jumbo Frames. Baby Jumbo frames (~2.5KB) provide enough space, plus some headroom for FCoE frames.
 
The diagram below shows an FC frame encapsulated within an FCoE frame.
 

Please feel free to correct the above diagram if its incorrect or unclear….

Although Jumbo Frames are not an officially recognised IEEE 802 standard, in fact due to technical reason they likely never will be, they are extremely widely implemented, well understood and about as standard as anything non-standard can be.  In fact, if jumbo frames did not exist, they would have to be invented to enable FCoE networks.

The same comment that came in from a reader talked about poor framing efficiency in FCoE.  Lets take a quick look at this while we’re here.

Data is protocol overhead

By “framing efficiency” we mean the number of control bytes sent versus total number of data byes sent.  A protocol with a high percentage of control bytes would be seen as inefficient.  Because of going with option 2 from above, FCoE actually has quite good framing efficiency, as it doesn’t carry baggage required to reassemble an FC frame from multiple FCoE frames.
 

I’ve heard it said that to techie people data is protocol overhead πŸ˜‰

Keep your hands off my bandwidth

Another reader has recently asked the following question

“….how is the IP traffic and the FCoE traffic kept from consuming one another’s bandwidth…”.

The answer is to do with Ethernet Priorities and I touched on them in a previous post and in the comments to that post, but while we’re here with the diagram above lets add a bit more meat to the answer.

I’ve written VLAN Tag in the above diagram, lets break it open a little more –
 

The above is a representation of an 802.1Q VLAN field.  This field also lists the Ethernet Priority (technically referred to as the Priority Code Point or PCP).  This is leveraged by Priority based Flow Control as discussed in earlier posts and is absolutely critical in the implementation of a lossless fabric, without which FCoE would be dead in the water.  PCP can be between 0-7 and each type of traffic will be encoded with its own PCP allowing features such as losslessness and bandwidth allocation etc to be implemented.  FCoE would have one value, iSCSI another etc…..

Hope that helps clear some things up.  Let me know if you have any more questions.  I’m enjoying the discussion that this is generating, and don’t be embarrassed about asking questions(obviously I don’t know all the answers but Im sure somebody out there will).

Nigel

I am a freelance consultant and available for consulting work via the Contact Me form

8 thoughts on “FCoE: Framing deep dive

  1. mike

    To put this in perspective, here is as good a place as any to point out that the encapsulation and extraction process occurs at every hop on the FCoE network (we won’t talk about multi-hop FCoE here).  In other words, every time an FCoE frame traverses and FCoE switch, it is re-encapsulated ready to be moved on to its next hop. huh?

    The outer encapsulation of FCoE is just an Ethernet. Why would it need to be decapsulated and encapsulated at each hop. That would be silly. The MAC address doesn’t change within the broadcast domain. The reason why multi-hop FCoE is not widely supported (Cisco does it using thier own methods) is because you can’t yet guarantee shortest path through the switched network.

    Isn’t that what TRILL is for?

  2. Brad Hedlund

    Mike,
    If there is an intermediate Ethernet switch between the CNA and FCoE FCF, correct, the intermediate switch does not change any encapsulation, just forwards. However if the FCoE FCF sends the frame to another upstream FCoE FCF, the encapsulation will change (new DMAC of the new next hop FCoE FCF).

    If you haphazardly design your Ethernet the shortest path may not be guaranteed, no different than a bad FC design. Designing Ethernet and FCoE with common sense best practices does not need TRILL.

    Cheers,
    Brad

  3. Nigel Poulton

    Hi Mike,

    Good spot, I was referring to to hops between FCFs. I will edit the post and clarify that point.   However, what Brad says is true (obviously as he is a Data Centre networking legend).

    I may at some point put up a post talking in more detail about the encapsulation process including FCFs and FCoE_LEPs etc….  But as this is kind of a tutorial series of posts it would have been too much to include that detail in this post.  IF I make them too long then nobody will read them πŸ˜‰

    Thanks for joining the dicsussion and poiting out the ambiguity.

    @Brad thanks for keepings things straight

    Nigel

  4. mike

    Nigel, thanks for doing these. They are very helpful and I hope you do continue them.Brad, I guess I don’t understand why the dmac would change in FCoE. Why wouldn’t the DMAC remain the target HBA? Hrm. I have to go read some more.  :)Thanks!

  5. stephen2615

    Any takers on the job market Nige?  You might have to work with some MDS’s..   We have someone keen at work to do something.  Pity about no hardware or budget and to make it worse, almost 95% of our thousands of servers are blades.
    I was reading some of the files on the t11 site for FCoE and they do make for good err umm reading.  Cisco has a document written by Silvano Gai called:
    FCoE end -to-end: adding details (T11/09-539v0)
    http://www.t11.org/ftp/t11/pub/fc/bb-6/09-539v0.pdf
    and it mentions something called a FC-CMP or FC Control and Management Plans.  This appears to be servers.   A Google search offers nothing.  I tend to not be bothered with the techical details of whats in a frame but when I see a server doing something in the middle of this cloud, I have to ask.  What is this FC-CMP?
     

  6. Nigel Poulton

    Hi Stephen,

    No takers on the job market yet……. might have to take a non FCoE related role πŸ™

    As for FC-CMP.  I have to hold my hand up and say I’m not sure……  But then Silvano Gai is the "geezer" when it comes to FCoE and is on the bleeding edge. Not only am I several steps behind him – I am several light years behind.  If I find out though I will let you know more and probspost on it.

Leave a Reply

Your email address will not be published. Required fields are marked *


*

You can add images to your comment by clicking here.