I received a comment on the site today from a confused reader.  The point of confusion is a common one and not something to be embarrassed about.  Part of the comment read as follows

“…..but it’s [FCoE] problem is that it consists of 3 different protocols (IP, FC, SCSI) just enlarging it’s frame size with unneccessery information. This is a step backwards."

The above is clearly a misunderstanding.  It would be accurate to say that DCB/CEE is designed to transport all three of the above protocols, but FCoE does not consist of these.  So if it doesn’t consist of these then what is it…….

FCoE in a single sentence

Put in to a single sentence “FCoE is the encapsulation of Fibre Channel frames within Ethernet frames”.

If you have a good quality dive suit then put it on…… we are about to perform a deep dive…………..

Squeezing it in

Some will know and some will not, but a standards based Ethernet frame with VLAN Tagging, including Tag Control Information can be 1522 bytes in length.  On the other hand a Fibre Channel frame can be up to 2148 bytes in length.  Clearly, this poses a problem – How can an Ethernet frame be wrapped around a fibre channel frame, if the fibre channel frame is bigger?

Two of the most viable and obvious solutions to this challenge include –
 

1. Fragment the FC frames so that they fit within standard Ethernet frames.
2. Make bigger Ethernet frames

From a performance and simplicity point of view, option 1 should be avoided at all costs due to the fact that fragmentation adds complexity to the encapsulation and extraction process.  Complexity brings overheads, which in turn would slow the protocol down – which if we remember back to my initial FCoE post, is highly undesirable in networks transporting SCSI as a payload. 

Option 2, on the other hand, keeps the encapsulation/extraction process very simple – 1 FCoE frame for every FC frame. 

No fragmentation and reassembly overhead, no tinkering with the headers or payload, and vitally, no unnecessary protocol induced latency.  These are big wins for networks transporting SCSI.  This seamless handling of frames offered by option 2 keeps the encapsulation/extraction process very simple, very lightweight and ultimately, very fast!

To put this in perspective, here is as good a place as any to point out that the encapsulation and extraction process occurs at every hop on the FCoE network (we won’t talk about multi-hop FCoE here).  In other words, every time an FCoE frame traverses and FCoE switch, it is re-encapsulated ready to be moved on to its next hop. 
 

NOTE: Due the broadcast nature of the Ethernet networks, frames need a new source and destination MAC address for each hop.  Point being, if the encapsulation/extraction process was arduous, it would significantly impact the performance of FCoE.

Option 2 also keeps hardware designs simpler, allowing for cheaper hardware than would be required if complicated encapsulation and extraction techniques had to be implemented.

Fortunately the folks on the standards body (INCITS T11 FC-BB-5) decided on option 2.

Jumbo to the Rescue

But in order to go with option 2, we need to invent larger Ethernet frames.  Well actually we don’t need to invent them because they already exist and are called Jumbo Frames. Baby Jumbo frames (~2.5KB) provide enough space, plus some headroom for FCoE frames.
 
The diagram below shows an FC frame encapsulated within an FCoE frame.
 

Please feel free to correct the above diagram if its incorrect or unclear….

Although Jumbo Frames are not an officially recognised IEEE 802 standard, in fact due to technical reason they likely never will be, they are extremely widely implemented, well understood and about as standard as anything non-standard can be.  In fact, if jumbo frames did not exist, they would have to be invented to enable FCoE networks.

The same comment that came in from a reader talked about poor framing efficiency in FCoE.  Lets take a quick look at this while we’re here.

Data is protocol overhead

By “framing efficiency” we mean the number of control bytes sent versus total number of data byes sent.  A protocol with a high percentage of control bytes would be seen as inefficient.  Because of going with option 2 from above, FCoE actually has quite good framing efficiency, as it doesn’t carry baggage required to reassemble an FC frame from multiple FCoE frames.
 

I’ve heard it said that to techie people data is protocol overhead

Keep your hands off my bandwidth

Another reader has recently asked the following question

“….how is the IP traffic and the FCoE traffic kept from consuming one another’s bandwidth…”.

The answer is to do with Ethernet Priorities and I touched on them in a previous post and in the comments to that post, but while we’re here with the diagram above lets add a bit more meat to the answer.

I’ve written VLAN Tag in the above diagram, lets break it open a little more –
 

The above is a representation of an 802.1Q VLAN field.  This field also lists the Ethernet Priority (technically referred to as the Priority Code Point or PCP).  This is leveraged by Priority based Flow Control as discussed in earlier posts and is absolutely critical in the implementation of a lossless fabric, without which FCoE would be dead in the water.  PCP can be between 0-7 and each type of traffic will be encoded with its own PCP allowing features such as losslessness and bandwidth allocation etc to be implemented.  FCoE would have one value, iSCSI another etc…..

Hope that helps clear some things up.  Let me know if you have any more questions.  I’m enjoying the discussion that this is generating, and don’t be embarrassed about asking questions(obviously I don’t know all the answers but Im sure somebody out there will).

Nigel

I am a freelance consultant and available for consulting work via the Contact Me form

Now then, there is no easy way to put this, but to do justice to a topic like this will require a lot of words.  I’ll do my best to keep it succinct, but if you are looking for a high level overview in under 1,000 words then this is probably for you, but thanks for stopping by………  However, if you want a deep dive and opportunity for technical discussion, then this might be what you’re looking for.

Still here? Magic, let’s go…..

First things first, its best to have a quick look at what things look like without CNAs, in order to more fully appreciate some of the problems CNAs are resolving.

 
Before CNAs

In a traditional server with no CNA cards and not connected to a unified fabric it is not uncommon to see the following “network” related sprawl (sorry about the terrible diagram but I wrote most of this post while on an aeroplane) –
 

Diagram 1

 
The FC HBAs tend to run at either 2, 4 or 8Gbps with the NICs usually running at either 100Mbps or 1Gbps.

The multiplicity of NICs is common to meet the demands of multiple traffic types such as; production, backup, management etc.

It is also worth noting that most servers are built with two physical HBA cards installed to provide both redundancy and increased aggregate bandwidth/performance – commonly referred to as I/O multi-pathing and is a must in 99.99% of FC SANs.

But that was then and this is now….. or nearly now

Say hello to the Converged Network Adapter, or CNA for short.

The Converged Network Adapter is exactly what it says it is – an HBA and a NIC converged on to a single PCIe adapter –
 

Diagram 2 – Picture courtesy of Emulex

Digging into the technical detail, some CNAs have a single ASIC that performs both the HBA and the NIC functionality, whereas other have a separate ASIC for each of the distinct functions.  This really depends on the vendor and model.  The important point being – CNAs provide HBA and NIC functionality in hardware, making them fast and reducing CPU overhead.  Implementing functionality without thieving CPU cycles is a big plus-point in server virtualisation environments.

Offload EVERYTHING!!

Considering the fact that stealing CPU cycles is undesirable at the best of times and a federal offence punishable by death in virtual server environments, providing hardware offloads is vital.  To help out in this area, the latest raft of Generation 2 Emulex OneConnect Universal Converged Network Adapters, such as the LP2100x, provide full CPU offload for all protocols on a single chip design and are FIP compliant!  That’s not just FCoE offload, we’re talking about TOE and iSCSI as well – the iSCSI one is interesting and may be a chat for another day!
 

NOTE: I will point out that I had a chat recently with Emulex and was very impressed.  They are laser focussed (pun intended) on FCoE and really understand the market and value position of FCoE.  A real pleasure to chat with passionate like minded people, oh and they didn't hang up on me when I mentioned "Broadcom"   VERY unprofessional of me but I was unbelievably jetlagged when we spoke, so I thank them for overlooking my lack of tact and poor humour.

So with the fact that CNAs provide both HBA and NIC functionality, as well as connecting to 10Gbps Enhanced Ethernet fabrics, it shouldn’t take a rocket scientist to figure out that we could swap out our 6 PCI adapters from Diagram 1 and replace them with just 2 CNAs (2 for redundancy).  If you think about it, this has the potential to reduce PCI adapter count and cable sprawl by crazy%

Saving Space

Early generation CNAs are installed as PCIe adapters in servers and as PCIe mezzanine cards in blade servers.  However, the way forward is to eventually have them embedded on the motherboard – a la’ LOM – vendors currently have this on their roadmap.  But this is no biggy right?  Well actually this has the potential to hugely reduce the amount of space consumed by PCI adapters inside of servers paving the way for higher density in blade servers where real-estate is at a premium!

Now this brings up another interesting benefit.  Not only is FCoE and its associated technologies resolving todays problems, it is actually enabling a better future.  How good would it be to have 6 or more CNAs per blade server!  That’s 60Gbps+ of flexible bandwidth based on today’s current 10Gbps Enhanced Ethernet!

Now this becomes a real winner in light of the current raft virtualisation optimised CPUs on the market – think Intel Nehalem and associated  virtualisation technologies like Intel VT-x VT-c, VT-d VMDq – xpect the ratio of VMs per core to rocket skyward!  In order to keep pace with the advancements in chip technology the I/O subsystem needs to move in step.  CNAs and Lossless Enhanced Ethernet are vital to this.

This point aside, I recently had a hand in a design involving HP BladeSystem c7000 technology.  I remember during the design wishing that the system had CNAs instead of the Flex10 and Virtual Connect technologies.  Could have saved space and cabling as well as potentially given more flexibility.  However, this was prior to even FCoE being ratified by INCITS.

Obvious benefits summarised

So some obvious benefits that CNAs bring include – reduced number of PCI adapters, cables, space, power and cooling.  Nice!

Secondly, by supporting 10Gbps Enhanced Ethernet they provide greater bandwidth than existing adapters allowing more traffic types and volume to travel down the same stretch of cable.  Suddenly the wire once nirvana is now a reality (actually wire twice if you want true I/O multi-pathing).
 

NOTE: A quick heads up on throughput.  While CNAs provide access to 10Gbps CEE networks, not all services, including FCoE, are supported at full bandwidth.  For example, CNAs normally support Ethernet at 10Gbps but FCoE only runs at a max of either 4Gbps or 8Gbps.  This is in line with 4Gbps and 8Gbps FC – there is currently no standard for 10Gbps FC N-Port to F_Port connections.

<OK so this is about the half way point, so you might want to come up for some air here before discussing the interesting stuff>

Clever things with CNAs

Now to the deeper technical stuff………..

With 10Gbps Enhanced Ethernet, FCoE, CNAs and other associated technologies being relatively new, the standards bodies (such as INCITS T11 FC-BB-5 for FCoE) as well as the vendors are able to design with virtualisation in mind.  This is great!

Let’s mention a couple of technologies that bring a lot to the table in Hypervisor environments.

NPIV

The first technology worth mentioning is N_Port ID Virtualisation, otherwise known as NPIV.  NPIV is a T11 INCITS and ANSI standard initially developed by IBM and Emulex, and yes I know it’s not exactly new.

Because NPIV is not a new technology I wont spend much time on it here other than to say NPIV makes it possible for a single HBA to have several N_Port IDs and WWPNs, allowing virtual machines to be uniquely addressable on the SAN and therefore able to be managed on the SAN just like normal physical servers.
 

If it would be useful I can put something up on NPIV.  Just leave a comment at the bottom asking and if enough people ask I’ll post on it.

I/O Virtualisation (SRIOV)

So while NPIV allows multiple VMs to be uniquely addressable on the SAN side of an HBA, on the other side, the side of the servers PCI tree, there is still a 1:1 mapping between physical ports and addressable PCI devices.

The Middle-man

Early virtual server implementations see the hypervisor act as middleman, between the I/O adapter and the Virtual Machine (VM).  The VM never actually talks directly with the I/O adapter, always through a middle-man – the hypervisor.  But like any middleman scenario, it has its advantages and disadvantages.  The middleman would argue that he adds “value”, but is rarely so keen to point out that he also always takes a generous percentage of the spoils (in our case, I/O performance and CPU cycles).

Some of the advantages and disadvantages of having the hypervisor as the middleman might include –

Middle-man Advantages

• Hypervisors often provide snapshot capabilities
• Hypervisors often provide thin provisioning capabilities
• Many existing Hypervisor technologies are currently designed around this model

Middle-man Disadvantages

• Lower I/O performance.  Because the hypervisor handles all I/O to and from a VM, as well as the associated interrupt handling etc, this injects server-side latency in to the I/O path as well as stealing CPU cycles form the main role of the Hypervisor.
• Security concerns.  The fact that all I/O, to and from VMs, is seen and touched by the hypervisor may be a concern to some people :-S
• Limited feature set.  You are restricted to the features and functions provided by the Hypervisor and not exposed to the full feature set available from the manufacturers native driver….

A common example where having the Hypervisor act as the middle man is not seen as desirable is a high throughput OLTP type system.  These are rarely virtualised due to, among other things, the I/O performance impact associated with having the Hypervisor as the middle-man.

So what does SRIOV do to help?

NOTE: Single Root I/O Virtualisation (SRIOV) and Multi Root I/O Virtualisation (MRIOV) are extensions to PCIe brought to us courtesy of the electronics industry consortium known as the PCI Special Interests Group (PCI-SIG).

SRIOV implementations allow an I/O adapter, in co-operation with the Hypervisor, to be sliced up in to multiple virtual adapters.  Each of these uniquely addressable on the servers’ PCI tree.  Each virtual adapter can then be mapped directly to a virtual machine and in turn is directly addressable by that virtual machine – no more middle-man.  SRIOV based adapters may have dedicated I/O paths in silicon and work alongside other related technologies such as Intel VT-c, VT-d, VMDq…. to provide a huge variety of offloads and assists aimed at reducing the load on main CPU and memory systems and thus increasing system performance.  This mode of operation is sometimes referred to as “hypervisor bypass” mode (we should be thinking VMDirectPath by now) and offers close to full line rate, making virtualisation a more realistic option for transaction based and other I/O intensive systems like our example OLTP system.

This also opens the door to things like VMs running drivers provided by the manufacturers making new functionality immediately available without waiting for the Hypervisor middle-man to support it.  Obviously it all needs buy in from your Hypervisor …..

Of course SRIOV is a semi-open standard (I say semi-open because several months ago when I researched it you had to pay to get access to the standards docs) and like most standards each vendor is free to implement the specifics in their own unique and value-add way as well as to add more features and requirements around it.

So in a nutshell, NPIV enables a single port to have multiple discrete addresses on the SAN side, whereas SRIOV allows a single I/O adapter to have multiple discrete addresses internally on the servers PCI bus/tree.  The diagram below from a long time ago on the Cisco website shows an I/O card that can be partitioned in to 128 (0-127) virtual interfaces.
 

Diagram from the Cisco website

Say goodbye to hardware rip and replace

By implementing the above mentioned technologies it becomes possible to run a single cable to a server and then use management software to configure the virtual interfaces according to current requirements.  For example, removing a NIC function and adding an HBA function becomes just a software change – no requirement to physically swap out a PCI card or lift the floor and run new cables.  Simply make the change in software and the new device will be presented to the servers PCI tree, job done!  Sound good to anyone else?

There is more, but I doubt anyone would read more than this in one post.  Other topics can always be discussed in the comments section below….

So if you made it all the way to hear, thanks and I hope it was useful.  Please feel free to join the discussion either here on the blog site via the comments section below, or by following me on Twitter @nigelpoulton.  I only talk about storage and related technologies.

Nigel

PS.  I am an independent consultant and available for hire via the Contact Me page

I think there has been a lively discussion on FCoE over the last 24-48 hours and it would be good to keep it going…… Before I start I’d be interested in any feedback, updates comments and the likes – I havent had a chance to properly research this and my schedule for the next few days makes it difficult.

Anyway……………… in order to accommodate the enhancements and improvements that come as part of what I have been calling “Enhanced Ethernet” (DCE/CEE), as well as facilitating the goal of consolidation to a single unified fabric, there are obviously some physical infrastructure changes required.  In the next few paragraphs we will discuss some of them.
 

NOTE:  When I say Enhanced Ethernet I’m referring to 10GigE, lossless, low latency, ETS, congestion notification as seen in DCE and CEE…

First up, cables!

Unfortunately our existing install base of Cat5 and Cat6 unshielded twisted pair and the likes, for the most part do not meet the demands of the unified fabric.  They don’t kcik it when it comes to latencies and Bit Error Rate etc.  In order to deploy and use Enhanced Ethernet, and therefore FCoE, we need to lay shiny new cables.  No problems though, that’s cheap and easy, right? 

<cough cough>

In order to keep with some of the major goals of convergence (drive down costs and power consumption), a cable and transceiver combination with low power demands and that has a good cost value is required.  This predominant combination is a passive or active twinaxial copper cable with SFP+ transceivers – sometimes referred to as “SFP+ Copper”. 

Twinaxial cables, usually referred to as Twinax or twin-ax, and named as such because they have a dual core.  The specification generally allows for cable runs of up to ~10 metres, although some kit combinations may support longer or shorter runs.  Twinax copper cables are typically ~6mm in diameter.

Twinax can transmit at 10Gbps full duplex (half duplex is not specified or supported in the IEEE DCB standards for 10GigE) over distances of up to 10 metres.  Although at first glance this is a relatively short distance, it is actually fairly well suited to Data Centre environments, which are typically short range high speed networks (remember we are not running these cables to workstations).  Such run lengths are especially suited for routing within a single cabinet or adjacent cabinets, such as from a server to an access layer switch mounted in the top of the rack. 

If longer runs are required then fibre can be used but at a higher cost. 

Twinax copper has been rated with a bit error rate in the region of 10−18 making it ideal for FCoE. 

Bit error rate:  or BER for short, is the ratio of erroneous bits received divided by the total number of bits transmitted.

SFP+, or to give its full name Small Form-factor Pluggable Plus, is an extension of the popular SFP standard seen in Fibre Channel SANs and legacy Gigabit Ethernet.  The design of SFP+ is relatively simple for a transceiver. 

In saying the design is simple, I refer to the fact that much of the signal processing circuitry and logic, often found in the transceiver module (such as with XFP), is removed from the transceiver and relocated to the switch and CNA (Converged Network Adapter).  This allows SFP+ to be smaller and cheaper in comparison to the more complex XFP, allowing for higher density switches.

As can be seen by the two diagrams below, SFP+ modules exist for both copper and glass and can transmit at 10Gbps.

SFP+ Optical transceiver

SFP+ copper transceiver with casing removed

Obviously this transferring of logic (silicon) from the transceiver to the switch and CNA may merely offset the cost from the transceiver to the switch.  While this may be the case it is generally agreed that such logic is better placed in the switch and CNA and usually allows for overall cheaper manufacturing costs. 

Backplanes

While on the topic of cables, copper and glass…..  As well as standards for carrying 10GigE over copper and fibre cables, the IEEE has also defined standards for backplane implementations.  One such commonly implemented standard is 10GBASE-KR.  This is commonly seen in blade servers as well as routers and switches and utilises a single lane running at a baud rate of 10Gbps, sometimes referred to as 10Gbps Backplane Ethernet.  It supports distances of up to 1m on copper based circuit boards which is plenty of distance for intra-chassis communications.

Other Infrastructure Requirements

At this point I suppose I should mention CNAs.  However, I can’t realistically talk about CNAs without talking about things like NPIV, SRIOV which is a topic and a half in and of itself.  So, for now I’ll skip over CNAs and say a quick word or two about FCoE capable switches.

FCoE capable Switches

A ton of new Ethernet standards as well as a new ULP, new network adapters and new cables inevitably leads to one thing…….. new switches.

Fortunately FCoE and the aforementioned enhancements are evolutionary.  By saying that, I am implying that implementing them in your Data Centre does not have to cause huge upheaval.  Some upheaval yes, but there is no requirement for large scale rip-and-replace.  In fact FCoE and her attendant technologies will happily site side-by-side with the likes of native Fibre Channel and 1Gbps Ethernet and at many levels the adjacent technologies will not even bat an eyelid.

Starting at the Edge

In order to simplify and expedite the adoption of Enhanced Ethernet and especially FCoE, some switch vendors provide edge switches at the access layer that allow companies to start deploying CNAs, at the edge of the network, and have them feed in to existing FC and Ethernet backbones via edge switches. 

For example, a company may deploy a new blade farm fully equipped with FCoE capable 10GigE Converged Network Adapters.  These CNA ports can be connected to the network via edge switches that support GigE, 10GigE, FCoE and FC.  This allows the blades to connect via 10GigE Enhanced Ethernet and then branch out to the existing network core via the 1Gbps Ethernet ports and to the FC SAN via the native FC ports.

These switches tend to be 2u pizza box style switches that are deployed within the server rack.  For some, they are a good place to start, especially in situations where ripping out your existing core and replacing it with shiny new 10GigE Enhanced Ethernet kit is not an option (i.e. just about everywhere).

Also at the Core

Some switch vendors also offer ultra-high performance ultra-scalable 10GigE Enhanced Ethernet FCoE aware switches for the network core.  These are typically modular blade based switches supporting multiple interface types and scaling to hundreds of ports.  Interface types include;10GigE Enhan ced Ethernet, 1Gbps Ethernet, 4Gbps FC and 8Gbps FC.  These switches represent the next generation of data centre switches and represent a huge move toward the virtual data centre and I/O consolidation.

As always, comments and thoughts welcome.

Nigel

You can follow me on Twitter @nigelpoulton – I only talk about storage and virtualisation.

I'm a freelance consultant and can be contacted at nigel at storage-strategist dot com

So the following is a mini tutorial on the flow control and lossless Ethernet in FCoE networks.  If you don’t care about FCoE, then point your browser somewhere else and don’t come back   On the other hand, if you are interested and want to know more, then put your feet up for 10 minutes and read on……..

Not your grandmothers Ethernet

FCoE requires an Enhanced 10GigE Ethernet network – it does not run over standard gigabit Ethernet.  Depending on who you speak to, this Enhanced Ethernet usually goes by one of two names –
 

1.    Converged Enhanced Ethernet (CEE).  Commonly used by IBMers and Brocadites.

2.    Data Centre Ethernet (DCE).  This is trademarked by Cisco and according to the Cisco website refers to the company’s architecture for next generation Ethernet in the Data Centre and is a superset of the DCB standards with some additions including L2MP.

As my wife often tell me that I live in my own little world, and to remain neutral, I'll refer to it as simply Enhanced Ethernet.

Whatever you decide to call it, it encompasses a collection of new technologies required in order for it to be able to transport FCoE traffic (frames).  Although FCoE is not the only driving force behind Enhanced Ethernet, it is certainly a major force.  Some of the technology changes encompassed in Enhanced Ethernet include the following –
 
•    Priority Based Flow Control (PFC) / Lossless behaviour
•    Low latency
•    Improved aggregate bandwidth
•    New link level negotiation protocols
•    Congestion Notification
•    Enhanced Transmission Selection

In this post I’ll concentrate on how lossless behaviour is implemented and achieved.  To do this its best to start somewhere near the beginning –

Its all goes back to SCSI

As the following very simple diagram shows, in FCoE networks, SCSI is encapsulated in FC frames, and FC frames are encapsulated in FCoE frames. 

So indirectly, SCSI is encapsulated in FCoE frames, meaning that FCoE and the underlying Ethernet network must keep SCSI happy.

So how do we keep SCSI happy?  If we cast our minds back, we should hopefully remember that SCSI was originally designed to be used within the confines of a physical server chassis, running uncontested over short parallel cables.  Uncontested = zero contention.  As a result SCSI was not designed to deal well with delays or transmission errors.  In fact, when either occurs, SCSI deals with them poorly.

In a nutshell – drop frames carrying SCSI payloads without efficient recovery capabilities (further up the stack) and you will be in a world of hurt! 

To keep SCSI happy you really need low latency and lossless behaviour.

What is a Lossless network?

Put very simply, a lossless network is a network that does not drop frames. 

The corollary being a “lossy” network – a network that drops frames when congestion occurs.  Your grandmothers GigE (1Gbps full duplex) network is lossy, it drops frames under congestion.

Making Enhanced Ethernet Lossless

The Data Centre Bridging Task Group has decided to implement link layer flow control in Enhanced Ethernet via a mechanism called Priority based Flow Control (802.3Qbb), or PFC for short.  PFC is the how losslessness is achieved on Enhanced Ethernet networks.  I might have just invented the word “losslessness” :-S

However, before we dig into PFC it is worth taking a side-step to briefly talk a little about Ethernet priorities.
 

Ethernet Priorities:  IEEE 802.1p defines 8 priorities for Ethernet.  These priorities allow for the implementation of Classes of Service at the link layer by tagging certain traffic types with an encoded priority.  Implementing Classes of Service allow available bandwidth to be divided in to logical lanes, or virtual links.  These virtual links can then be leveraged by other protocols and services, such as Priority based Flow Control which we will talk about.

The diagram below shows how a physical link between two switches can be divided in to 8 logical lanes/virtual links labelled CoS 0 through CoS 7 –

PFC is IEEE 802.1p Ethernet priorities “aware” and applies intelligence in the form of selective enforcement of the Ethernet PAUSE condition.  This is where the pause condition is selectively applied to particular Classes of Service.  This makes PFC perfect for converged unified fabrics where multiple traffic types and classes of service share a common network.  It also makes PFC superior to native FC BB_Credits which can only apply arbitrary conditions that affect all traffic on the link.

Interestingly, while PFC achieves the same, albeit superior, results as FC BB_Credits – that of creating a lossless network – the technical specifics of the two implementations are vastly different.  On the one hand FC BB_Credits require the sender to assume it cannot send frames until it explicitly knows that the receiver is ready.  On the other hand PFC allows senders to assume that they can always send unless explicitly told not to.

Hitting the Pause Button

PFC enforces the Pause condition per Class of Service by issuing a PFC frame with 8 time fields, one for each previously mentioned CoS/Ethernet priority.  When a switch issues PFC frames it is instructing the connected node to apply the PAUSE condition (i.e. stop sending) to frames with particular CoS values.  The diagram below shows the PAUSE condition applied to all Classes of Service except CoS 3 –

As well as the classes to which it is applied, the duration of the pause is also specified within the PFC frame.  This allows PFC to be selective in which class of traffic it will apply the Pause condition to, making it possible to enforce the Pause condition on a single, or a subset of all 8 classes.  It is also possible to selectively lift the pause condition, such as when congestion has dissipated and there is no need to wait until the Pause timeout expires.  This adds to the functionality of PFC.

PFC frames are handled at the MAC layer similar to how R_RDY’s are handled by FC-1 in native FC networks.  The PFC frame is a standard non-tagged MAC Control frame identified by Ethertype 0×8808 with Op-code 0×0101.  For best performance and efficiency all good FCoE switches handle flow control in hardware.

In Enhanced Ethernet networks FCoE will be assigned to a Class of Service which in turn will be treated a high priority so that when congestion starts to occur in the network (per switch) it will be allowed to continue to operate while other Classes of Service (protocols) might be paused.

Net result – lossless behaviour for FCoE frames on shared Enhanced Ethernet networks (unified fabrics).  All implemented at the link layer by the FCoE capable switches.  Voila!

Comments and thoughts welcome.

Nigel

Follow me on twitter @nigelpoulton.  I only talk about storage and virtualisation etc….

I'm a freelance consultant and can be contacted at nigel at storage-strategist dot com