• Dennis Martin (@demartek)
• Dave Graham (@davegraham)
• Stu Miniman (@stu)

For an hour solid, we talked FCoE and the realities of deploying the technology.

As well the the recording of the call, which I recommend listening to, there will be several short resultant articles posted on Wikibon covering topics such as -

• CIO Considerations
• CTO considerations
• Operation and organisational considerations
• Getting Rid of Stuff – what can FCoE help me throw out.

This was the first ever Peer Incite call to hit the 200 maximum participants limit.  Proof that FCoE and network convergence are hot topics in the Data Center.

If you’re considering deploying FCoE, then I recommend you and take a listen.

I can be contacted via the Contact Me page or on Twitter at (@nigelpoulton)

Nigel

In this inaugural episode of the Technical Deep Dive Podcast I'm joined by Greg Ferro from Etherealmind.com and Camden Ford from Xsigo Systems.  Cam talks to us about Xsigo Systems and the technologies and solutions they offer.

This is available as a two-parter due to length.  Part 1 is about 30 minutes long and Part 2 is about 40.  What can I say, you cant dive deep in 20 minutes!

Comments welcome!

As early as last summer I was involved in specking and purchasing some HP BladeSystem Matrix based solutions – essentially a shrink-wrapped Rack based solution that had compute, networking, storage and management tools.  Granted, HP Matrix is an early attempt and not much different from the norm, but a step towards the Rack Area Network (RAN).

So… Rack based solutions are on their way, and the way I see it – from speaking with peers, customers and vendors – the following two high level Rack based designs will be predominant and will slug it out over the next few years:

1.  The FCoE RAN Solution

Of the two solutions, this one most closely resembles what we know today.  The only major difference being the use of FCoE between the server and the Top of Rack (ToR) switch.  This solution requires Converged Network Adapters (CNA), copper twinax cabling and FCoE ToR switches.  As it happens this is really the only practical place that FCoE can currently be deployed.  Fortunately, however, the FCoE products in this space (the RAN) are maturing quickly – we already have 2nd generation, single chip, single driver code base, high performance CNAs shipping and supported by most good server vendors….

In the FCoE based RAN solution there is very little in the ways architectural change – no blurring of the server/network edges and no change to the design of servers or networks.  This gives the comfort factor. 

Anyway, the sketch below shows a high level view of of this type of solution -

2.  The IOV solution

This second solution is slightly more innovative.  It requires slight changes to existing server/blade designs, there is some blurring of the server and network edges, and some re-thinking of network design and management is required.  Not quite the same comfort factor that the FCoE RAN solution gives,but as the saying goes – No pain, No gain! 

The IOV solution can be summarised in the following -

Servers and blades are reduced to pure compute and memory.  The I/O components are disaggregated from the server chassis and re-housed in an external ToR I/O Director.  Servers connect to the external I/O cards by either PCIe cables or IB.  These I/O adapters can be CNAs or traditional NICs and HBAs.  They are next generation in that each one can be carved into multiple logical adapters which can each be dynamically assigned and unassigned to any server and VM within the Rack.  The I/O adapters and I/O Directors have built-in switching functionality, enabling traffic to be switched either within the I/O adapter or between adapters within the same I/O Director without the need to travel up to a traditional network switch (hairpinning in the adapter or I/O Director).  Essentially, access layer switching will be moved on to the PCIe I/O adapter!

The diagram below shows this at a high level -

No room in the RAN

Personally I like the idea of using PCIe as the main interconnect within the Rack.  Every chipset on every server already has a bunch of PCIe bandwidth that is essentially….. well…. FREE!  Who doesn’t like the sound of that!?  10Gbps CEE and FCoE licensing of ports is … well…. definitely not free.

Of course there is the other side.  PCIe muscling Ethernet out of the RAN will not go down well with some, nor will implementing switches within NICs/CNAs and I/O Directors.  Not only will this tread on certain vendors markets and margins, it also brings with it several network design and management challenges.  But what the heck… we grow from our challenges and come out the better for it – right?  Point being, knee-jerk self-preservation type reactions from the network guys should be expected

Final thought on FCoE

While the IOV solution could potentially muscle Ethernet out of the RAN, it can still branch out to FCoE switches in the core.  So either way, FCoE will play a role. 

And if we are being creative, we could run our I/O up to the ToR I/O Director over PCIe within the RAN and then branch out via a CNA in one of those I/O Directors to a core switch with FCoE ports.  One way of utilising FCoE ports that are currently available in core switches.

Interesting times!

PS. I will be featuring on the Wikibon FCoE Fact vs Fiction call on 2nd February along with Stu Miniman, Dave Graham and  Dennis Martin  If you’re interesting in FCoE put it in your calendar.

The term, as well as the concept, of Rack Area Networking is one I’m hearing more and more often.  As a result of this, as well as the fact that I’m convinced this is going to be one of the most interesting and important areas of Data Center computing over the next few years, I’ve decided to write a mini-series on the topic. 

This post is instalment number 1 and is intended to introduce the concept and get the ball rolling.  The whole thing is a bit of me thinking out-loud and attempting to generate some awareness and conversation around the topic, as .  So please pitch in!

Rack Area Network – the concept

For me, Rack Area Networking, or RAN for short, is an umbrella term for most of the clever networking and I/O virtualization stuff that goes on within a rack – a 42u rack. 

With it being “Rack Area”, it is a close proximity network and as a result operates over very high-speed low-latency interconnects.  

Physically, RAN technologies include a new generation of at least the following: I/O adapters, cabling, Top of Rack (ToR), and may be even End of Row (EoR), switches.  However, for reasons which will become clear, the emphasis is heavily on the clever – technologies that enable the flexible, the dynamic and the virtual aspects.

For example, the I/O Adapters driving the RAN evolution are not just faster than the legacy adapters they are replacing, they have built-in cleverness – hardware virtualization and huge flexibility! 

Some of the other technologies that define and operate within the RAN include – SR-IOV, MR-IOV, vNIC, vHBA, CNA, FCoE, Hairpin-turns, switching in the adapter, VNTag, VN-Link…. just to name a few.  In future posts we will discuss most of them.

As well as the above new hardware and technologies, the RAN also requires and includes a new generation of management software and functionality.  True value is often in the software – the glue that holds it all together and makes it all happen.

The best part being, there are early RAN technologies already out there in the wild.  And they are already delivering real-world tangible benefits.

Some technologies driving the evolution…

It’s really important to note that while technologies in the RAN are experiencing a period of accelerated evolution, it is most definitely an evolution.  The changes are happening fast, but they are not huge disruptive changes.  For the most part, they are improvements and enhancements, albeit major, on what we already know and are comfortable with.  E.g. take PCIe adapters and create multiple virtual adapters (vNIC and vHBA) in hardware….

Just a few of the currently shipping RAN technologies include -

• HP Virtual Connect Flex-10
• IBM Virtual Fabric Solution w/ Emulex UCNA
• Cisco UCS w/ Palo adapter
• Xsigo I/O Director
• Virtensys VIO switches

NOTE: Let me know if I’ve missed any major RAN technologies off the list

Some of the above technologies are very much generation 1 and only a small step towards the RAN, bringing only small benefits.  Whereas others are a major step with huge benefits.  All vendors are scrambling to take the lead in this evolving area.  In later posts we’ll dig deep into most of them.

Blurring the Lines and Causing Havoc

Naturally, many of these technologies are challenging and threatening the traditional server/network edge configurations we are used to. 

Hairpin turns, switching in the adapter and avoiding edge switches are just some of the paradigm shifts that RAN technologies might force us to consider.  Such topics are the subject of intense and engaging debate.  All very interesting and some of the concepts are very cool!

In upcoming posts we’ll talk about the likes of – SR-IOV MR-IOV Hairpin-turns VirtenSys Flex-10 VEB Xsigo VNTag NextIO VNLink InfiniBand PCIe

Thanks for dropping by and feel free to throw in your penny’s worth.

Nigel

I am available as an independant freelance consultant and can be reached via the Contact Me page.

If you're thinking that I might just bemaking some of this stuff up or inventing buzwords, then you need to check out my follow-on RAN and IOV related posts listed below -

IOV vNICs and vHBAs

IOV and introducing hairpin turns

In my opinion the show was a real success with over 1,500 attendees, of which over 1,100 were end users and reseller delegates and the remainder made up of general riff-raff such as vendors, press and the likes….

One of the things I like to see at shows like these is hardware.  What can I say, Im just the kind of guy that gets a kick out of seeing hardware.  So for the rest of you out there like me – sit back and enjoy……..

First up, I was impressed to see a Symmetrix V-Max, blue strip light ablaze.  The only disappointment was that whenever I popped by to have a chat, somebody else was always being given an overview

Next up the IBM HS22 BladeCenter that was used to demo the new IBM Virtual Fabric, which is based on technology from IBM, BLADE Networks and Emulex.  A much needed addition to the IBM portfolio in my opinion.

Oh and while on the theme of IBM, here is another rack of IBM kit

As the IBM Virtual Fabric solution has an Emulex CNA in it, next up is Shawn Walsh from Emulex showing us that the Emulex UCNA is real and not a myth.

And a close up on a desk with a pen for lined up to give an idea of size

Then if you followed the girls with “Thin” written on their T-shirts you couldn’t miss the 3PAR InServe kit on show.  I plan on writing about 3PAR RAID MP and Persistent Cache, both of which are potentially very interesting technologies.  But seeing as 3PAR are attending the upcoming GestaltIT Field Day I might wait and see if I can glean some deep tech info from them.

Brocade also turned up with a rack load of kit, although hugely disappointing for me was the lack of an FCoE 10-24 blade in the DCX director.  Not to worry though, there was a B8000 top of the rack CEE/FCoE switch to keep me happy.

And a Brocade dual port CNA

I have some video footage from the Brocade booth that I will post some time next week.  So stay tuned.

Even the internet connected laptops were also of decent spec.  Below is a smart little HP laptop alongside my personal 11.1” Sony job – check out the grease marks on my trackpad and spacebar

And last but not least, the quality of freebies was good.  Aside from the standard pens and stress balls, I was particularly impressed with –

iPod Shuffle
Solio solar powered USB charger
3-in-1 pen/laser pen/1GB memory stick (James Bond style)
2GB micro SD with standard SD adapter 

The economy must be recovering!

As well as the above mentioned video footage from the Brocade booth, I also got some video footage from the Arista networks booth.  Keep an eye out for that as I plan to post them in the following week…

Nigel

You can follow me on Twitter where I talk about storage technologies (@nigelpoulton) and I am also available for hire as a consultant.

In the video, William Lloyd Scull Senior Network Architect at BLADE Network Technologies, demos a feature of the IBM BladeCenter Virtual Fabric (comprised of Emulex OneConnect UCNA, BLADE Network Technologies Blade Switches and IBM BladeCenter hardware).

In the video William dynamically reduces network bandwidth allocated to a vSwitch from 2500Mbps down to 1900Mbps, as well as talks us through some of the other features and some of the hardware involved in the Virtual Fabric solution.

Its an interesting video if you want quick and dirty overview of Virtual Fabric for IBM BladeCenter and a glimpse at what it looks like with the lid off.  The video was not rehearsed, and although Id gone through the same thing with William the day before he had no idea I would turn up again the next day with my video camera and ask him to perform to the world

So thanks to William for being a good sport.  Enjoy……..

First impressions for me are similar to my thoughts on HP Virtual Connect Flex-10…….  In fact this is very similar to VC Flex-10 but with a more clear roadmap for FCoE and is built on a solid Converged Network Adapter (OneConnect UCNA) from Emulex.  However, as good as it is, its very generation 1.  But that’s not a huge issue because these are early days in this area.  Everybody else is in a similar position.  To my knowledge there is nobody doing hairpin turns in silicon yet and we are all a long way off MR-IOV and the new rack area landscape which that makes possible……..

So…. an important technology for IBM, may be slightly better than Virtual Connect Flex-10 but is probably not quite as good as some of the stuff Cisco is doing with Palo… But a solid foundation to build on (see my previous post on the Emulex OneConnect Universal Converged Network Adapter for more info).

Thoughts and comments welcome as usual…..

Nigel

Oh and you can follow me on Twitter where I talk about storage technologies (@nigelpoulton) and I am also available for hire as a consultant.

Most will know by now that Im all for converged Ethernet unified fabrics.  Still, Im well aware that some of the vendor offerings are very “Generation 1” and have that version 1.0 feel to them.  Todays launch from Emulex honestly didn’t have that feel to it.  In fact it seems that the products shipping around CEE are getting more and more mature by the day.  So let me take a few minutes to discuss some of the product highlights…..

Architectural Overview

At a high level, the OnceConnect UCNA is a high performance single chip CEE adapter.

Lets qualify that statement…..

High Performance:  Sure, anybody can call their products, especially 10Gbps Ethernet adapters, “high performance”.  However, Emulex can back this up with the fact that this product provides hardware offloads for TCP/IP, iSCSI and FCoE.  This is marketed under the name vEngine (add a “v” to any technology and it automatically sounds cooler). 

Basically this adapter is a workhorse.  It will do the protocol related work for you, freeing up your CPU to do other tasks.  This enables it to be truly high performance, not just for FCoE but also for iSCSI and TCP/IP.  This is great – especially in Hypervisor estates.

Single Chip:  There is a single ASIC on the card that does the above.  No requirement for dedicated ASICs for each protocol.  ASICs also add to the High Performance claim, as ASICs are still very much the way to go with high performance I/O adapters – so called merchant silicon doesn’t quite cut the mustard. 

At the time of writing this article this is the only adapter on the market with all of the above hardware offloads on a single chip.  Although it is only fair to mention that FCoE functionality will be released later this year, but as its already late October, this will be soon.

Pay-As-You-Go and future proofing

The above is all good, but a bit overkill if you don’t need it all just yet.  Well……. You can buy the adapter as a 10Gbps Ethernet adapter, without any of the additional hardware offloads etc, at the pricepoint of a 10Gbps Ethernet adapter.  But in the future, if/when you require iSCSI offload or FCoE then you can easily unlock these features with a license.

Sounds almost perfect for the kind of staged deployments of CEE and FCoE that I think a lot of companies will adopt – Buy future proofed hardware now that allows you to keep your options open.  Deploy the UCNA now as a 10Gbps Ethernet adapter and be in a position to press ahead with FCoE etc in the future without the need to rip and replace your I/O adapters!  What is there not to like about it? 

Remember that we already do this with our switches and storage arrays, and it works well.  Why not apply the same model to I/O adapters.

This is all of course assuming that the progressive licensing doesn’t make the overall cost more expensive!!  From tweeting about this at SNW Europe today this is the biggest concern from fellow tweeters.

Oh and you can run software initiated iSCSI etc over the adapter without licensing the iSCSI offload….. 

All-in-all it seems very flexible to me.
 

NOTE: Of particular interest to me was the fact that the core features, as well as the base cost, of this adapter are 10Gbps Ethernet.  This is very interesting when you consider Emulex are traditionally a Fibre Channel company.  Clearly Emulex are moving with the market here and recognising Ethernet as the dominant technology and building on that.  Emulex also have people on IEEE 802.1 committees such as DCB.  Now that’s what I call not betting against Ethernet.

A must for the Virtual Data Center

In my last post regarding CEE I talked about the importance of CEE and its associated speeds and intelligence in the light of advances being made in Hypervisor based environments. 

I would add to that – features and functions such as those seen on the Emulex OnceConect UCNA are equally as vital….

As the number of VMs per CPU core increases to greater than 10, the I/O and networking components need to keep step.  What is the point of CPU technologies enabling huge numbers of VMs per CPU core if your I/O subsystems can’t keep up!

I think most people can accept the need for FCoE offloads as this is the norm in FC environments – and FCoE is Fibre Channel, just over a different Layer 2.  But….. offloads for iSCSI also become more and more important for iSCSI shops as they also deploy more and more VMs per physical server.
 

RuptureMonkey opinion:  Protocol offloads are absolutely vital to Virtual Data Centers.

Don’t Blink

There is no doubt that the I/O world is changing in front of our very eyes.  Be careful not to blink as you might miss something important.  Technologies such as CEE and protocol offloads (as well as many others) are key. 

Other related stuff

Also mentioned in the launch were technology agreements with IBM regarding 10Gbps Ethernet NICs and 16Gbps native FC HBAs, both for IBM Power Systems.  The 16Gbps FC design win being an industry first!  Technology is marching on and Emulex are certainly up at the front.

This is on top of the recent announcement around the IBM Virtual Fabric for BladeCenter – where Emulex, BLADE Network Technologies and IBM have collaborated to bring to market a very good blade based I/O solution, comparable, and possibly superior to, HP Virtual Connect Flex-10.  I saw IBM Virtual Fabric demonstrated in a VMware environment today and it looks a great technology.  The guys were also kind enough to rip a blade out and let me see inside one of the blades.

Finally, Emulex also announced OneCommand Manager which replaces HBAnywhere, but I havent seen this yet to be able to comment.

Random info:  Apparently World of Warcraft has ~15,000 blade servers. Cool!

Nigel

NOTE: Im using the term CEE. I should probably use DCB, but Im not.  For disambiguation of the interchangeable terms CEE, DCB and DCE see here.
 
 

 
 
Half baked

First up, and in the spirit of honesty, I have to mention that the DCB/CEE related standards are not yet fully baked.  Formal ratification is expected early to mid 2010.  However, enough is defined for major vendors to be shipping the technology.  And more than merely shipping it, some are betting large chunks of their business on it. 
 
 
CEE 10 second overview

Really quickly, CEE is currently a 10Gbps full duplex lossless Ethernet with dynamic prioritisation/bandwidth allocation and some other goodies.

Specifically -

• Priority based Flow Control (802.1Qbb)
• Enhanced Transmission Selection (ETS 802.1Qaz)
• Congestion Notification (802.1Qau)

If you know the above technologies, then you will know that they are compelling and pervasive.  If you don’t know them then trust me, they are good.
 
 
The Grand Key

CEE is a key technology of the future.  For starters it is absolutely key to FCoE.  In fact without CEE there would be no FCoE, at least no compelling FCoE.  But CEE is not just for FCoE, its features and benefits can and will increasingly be leveraged by many areas of IT infrastructure…  iSCSI as well as NFS over UDP are just a couple that are commonly talked about.

Point being, FCoE is only one of the many new options made possible by CEE.
 
 
Smarter AND faster

Yes there is non-CEE 10Gbps Ethernet.  For the purposes of this post Im going to refer to it as “Dumb 10Gbps Ethernet” (and Im writing it in pink because its for girls).  This Dumb 10Gbps Ethernet is 10Gbps Ethernet without the goodness of CEE that we just mentioned above.  Sure its cheaper than the CEE version.  But, we all know that cheaper usually isn’t better.  Pay peanuts, get monkeys.   Trust me, in the long run you will want CEE….

Not convinced?  Let me draw a parallel that I hope will help create an “Aha” moment –
 

Just like processor technology, Ethernet can and will get faster and faster and faster.  But, like processor technology, there comes an inflection point – where getting faster and faster, but not smarter and smarter, becomes almost pointless.  The change in focus needs to be towards smarter and not just faster.

Ask yourself the following question – what are the real game changers and compelling aspects of Intel’s current raft of “Nehalem” processors?  Is it the GHz?  Or is it the built-in virtualisation intelligence and all the associated benefits (Im thinking Intel VT….)?

Same goes for Ethernet, speed is not everything, intelligence matters too.  CEE brings both speed and intelligence to the network.

Ask Hypervisor architects and admins if they would like to go back to the old days of non-hypervisor aware CPUs.  Im pretty sure they will tell you where to go.  Same goes, and will go, for people deploying and using CEE.  Once bitten forever smitten.
 

RupturedMonkey statement: “CEE brings intelligence to the network.”

 
 
Hubs versus Switches

In my opinion, the whole Dumb 10Gbps Ethernet versus CEE smacks of the old Hubs .vs. Switches debate of days gone bye. 

Is there anybody wishing they still deployed hubs at the centre of their networks?  No.

Point being…. CEE is here and its changing the game. 
 
 
Cable once and most of most

All of a sudden the possibility of cable once is a reality.  CEE has the capability to run most, if not all, of most companies portfolio of network traffic over a single cable.  But more than that – over a single PCIe card and to single edge switch port.  Who wouldn’t want that? 

But there’s even more – it will simplify changes and network management, as well as bring down the cost of power, cooling and all of that jazz too!
 
 
Unabridged opinions and conclusions

So if you didn’t know previously, you know now – Im liking the look of this whole converged network unified fabric thing.  I see CEE as an essential building block of every modern Data Centre!

So with that, let me finish with a word or two to the naysayers and FUD-slingers –
 

Do yourselves a favour and don’t waste your time and energy trying to slow the unstoppable forward march of technology.  Believe me, it will roll right over you like a steamroller over a cockroach rat something tiny and insignificant   Where would Intel, AMD or any of the major server vendors be if they had resisted the march of VMware?  After all, they would all ship more units if every server image required dedicated hardware…..  Short answer is, they would be in a world of hurt.

RupturedMonkey advice: Don’t get left behind… or rolled over by a steamroller.

Its a brave new world out there, and we as IT pro's as well, as the enabling technologies, need to move with the times.  CEE is doing just that – moving things forward. 

By all means, choose to limp forward with bog standard 10Gbps Ethernet “sans” the goodies of DCB/CEE.  If you do, then good luck to you, but dont look on enviously as the rest of us race forward into the light.

Nigel

Please feel free to share your thoughts below. You can also follow me on Twitter @nigelpoulton.  I only talk about storage and technology and the conversation is often very interesting.

I received a comment on the site today from a confused reader.  The point of confusion is a common one and not something to be embarrassed about.  Part of the comment read as follows

“…..but it’s [FCoE] problem is that it consists of 3 different protocols (IP, FC, SCSI) just enlarging it’s frame size with unneccessery information. This is a step backwards."

The above is clearly a misunderstanding.  It would be accurate to say that DCB/CEE is designed to transport all three of the above protocols, but FCoE does not consist of these.  So if it doesn’t consist of these then what is it…….

FCoE in a single sentence

Put in to a single sentence “FCoE is the encapsulation of Fibre Channel frames within Ethernet frames”.

If you have a good quality dive suit then put it on…… we are about to perform a deep dive…………..

Squeezing it in

Some will know and some will not, but a standards based Ethernet frame with VLAN Tagging, including Tag Control Information can be 1522 bytes in length.  On the other hand a Fibre Channel frame can be up to 2148 bytes in length.  Clearly, this poses a problem – How can an Ethernet frame be wrapped around a fibre channel frame, if the fibre channel frame is bigger?

Two of the most viable and obvious solutions to this challenge include –
 

1. Fragment the FC frames so that they fit within standard Ethernet frames.
2. Make bigger Ethernet frames

From a performance and simplicity point of view, option 1 should be avoided at all costs due to the fact that fragmentation adds complexity to the encapsulation and extraction process.  Complexity brings overheads, which in turn would slow the protocol down – which if we remember back to my initial FCoE post, is highly undesirable in networks transporting SCSI as a payload. 

Option 2, on the other hand, keeps the encapsulation/extraction process very simple – 1 FCoE frame for every FC frame. 

No fragmentation and reassembly overhead, no tinkering with the headers or payload, and vitally, no unnecessary protocol induced latency.  These are big wins for networks transporting SCSI.  This seamless handling of frames offered by option 2 keeps the encapsulation/extraction process very simple, very lightweight and ultimately, very fast!

To put this in perspective, here is as good a place as any to point out that the encapsulation and extraction process occurs at every hop on the FCoE network (we won’t talk about multi-hop FCoE here).  In other words, every time an FCoE frame traverses and FCoE switch, it is re-encapsulated ready to be moved on to its next hop. 
 

NOTE: Due the broadcast nature of the Ethernet networks, frames need a new source and destination MAC address for each hop.  Point being, if the encapsulation/extraction process was arduous, it would significantly impact the performance of FCoE.

Option 2 also keeps hardware designs simpler, allowing for cheaper hardware than would be required if complicated encapsulation and extraction techniques had to be implemented.

Fortunately the folks on the standards body (INCITS T11 FC-BB-5) decided on option 2.

Jumbo to the Rescue

But in order to go with option 2, we need to invent larger Ethernet frames.  Well actually we don’t need to invent them because they already exist and are called Jumbo Frames. Baby Jumbo frames (~2.5KB) provide enough space, plus some headroom for FCoE frames.
 
The diagram below shows an FC frame encapsulated within an FCoE frame.
 

Please feel free to correct the above diagram if its incorrect or unclear….

Although Jumbo Frames are not an officially recognised IEEE 802 standard, in fact due to technical reason they likely never will be, they are extremely widely implemented, well understood and about as standard as anything non-standard can be.  In fact, if jumbo frames did not exist, they would have to be invented to enable FCoE networks.

The same comment that came in from a reader talked about poor framing efficiency in FCoE.  Lets take a quick look at this while we’re here.

Data is protocol overhead

By “framing efficiency” we mean the number of control bytes sent versus total number of data byes sent.  A protocol with a high percentage of control bytes would be seen as inefficient.  Because of going with option 2 from above, FCoE actually has quite good framing efficiency, as it doesn’t carry baggage required to reassemble an FC frame from multiple FCoE frames.
 

I’ve heard it said that to techie people data is protocol overhead

Keep your hands off my bandwidth

Another reader has recently asked the following question

“….how is the IP traffic and the FCoE traffic kept from consuming one another’s bandwidth…”.

The answer is to do with Ethernet Priorities and I touched on them in a previous post and in the comments to that post, but while we’re here with the diagram above lets add a bit more meat to the answer.

I’ve written VLAN Tag in the above diagram, lets break it open a little more –
 

The above is a representation of an 802.1Q VLAN field.  This field also lists the Ethernet Priority (technically referred to as the Priority Code Point or PCP).  This is leveraged by Priority based Flow Control as discussed in earlier posts and is absolutely critical in the implementation of a lossless fabric, without which FCoE would be dead in the water.  PCP can be between 0-7 and each type of traffic will be encoded with its own PCP allowing features such as losslessness and bandwidth allocation etc to be implemented.  FCoE would have one value, iSCSI another etc…..

Hope that helps clear some things up.  Let me know if you have any more questions.  I’m enjoying the discussion that this is generating, and don’t be embarrassed about asking questions(obviously I don’t know all the answers but Im sure somebody out there will).

Nigel

I am a freelance consultant and available for consulting work via the Contact Me form

Now then, there is no easy way to put this, but to do justice to a topic like this will require a lot of words.  I’ll do my best to keep it succinct, but if you are looking for a high level overview in under 1,000 words then this is probably for you, but thanks for stopping by………  However, if you want a deep dive and opportunity for technical discussion, then this might be what you’re looking for.

Still here? Magic, let’s go…..

First things first, its best to have a quick look at what things look like without CNAs, in order to more fully appreciate some of the problems CNAs are resolving.

 
Before CNAs

In a traditional server with no CNA cards and not connected to a unified fabric it is not uncommon to see the following “network” related sprawl (sorry about the terrible diagram but I wrote most of this post while on an aeroplane) –
 

Diagram 1

 
The FC HBAs tend to run at either 2, 4 or 8Gbps with the NICs usually running at either 100Mbps or 1Gbps.

The multiplicity of NICs is common to meet the demands of multiple traffic types such as; production, backup, management etc.

It is also worth noting that most servers are built with two physical HBA cards installed to provide both redundancy and increased aggregate bandwidth/performance – commonly referred to as I/O multi-pathing and is a must in 99.99% of FC SANs.

But that was then and this is now….. or nearly now

Say hello to the Converged Network Adapter, or CNA for short.

The Converged Network Adapter is exactly what it says it is – an HBA and a NIC converged on to a single PCIe adapter –
 

Diagram 2 – Picture courtesy of Emulex

Digging into the technical detail, some CNAs have a single ASIC that performs both the HBA and the NIC functionality, whereas other have a separate ASIC for each of the distinct functions.  This really depends on the vendor and model.  The important point being – CNAs provide HBA and NIC functionality in hardware, making them fast and reducing CPU overhead.  Implementing functionality without thieving CPU cycles is a big plus-point in server virtualisation environments.

Offload EVERYTHING!!

Considering the fact that stealing CPU cycles is undesirable at the best of times and a federal offence punishable by death in virtual server environments, providing hardware offloads is vital.  To help out in this area, the latest raft of Generation 2 Emulex OneConnect Universal Converged Network Adapters, such as the LP2100x, provide full CPU offload for all protocols on a single chip design and are FIP compliant!  That’s not just FCoE offload, we’re talking about TOE and iSCSI as well – the iSCSI one is interesting and may be a chat for another day!
 

NOTE: I will point out that I had a chat recently with Emulex and was very impressed.  They are laser focussed (pun intended) on FCoE and really understand the market and value position of FCoE.  A real pleasure to chat with passionate like minded people, oh and they didn't hang up on me when I mentioned "Broadcom"   VERY unprofessional of me but I was unbelievably jetlagged when we spoke, so I thank them for overlooking my lack of tact and poor humour.

So with the fact that CNAs provide both HBA and NIC functionality, as well as connecting to 10Gbps Enhanced Ethernet fabrics, it shouldn’t take a rocket scientist to figure out that we could swap out our 6 PCI adapters from Diagram 1 and replace them with just 2 CNAs (2 for redundancy).  If you think about it, this has the potential to reduce PCI adapter count and cable sprawl by crazy%

Saving Space

Early generation CNAs are installed as PCIe adapters in servers and as PCIe mezzanine cards in blade servers.  However, the way forward is to eventually have them embedded on the motherboard – a la’ LOM – vendors currently have this on their roadmap.  But this is no biggy right?  Well actually this has the potential to hugely reduce the amount of space consumed by PCI adapters inside of servers paving the way for higher density in blade servers where real-estate is at a premium!

Now this brings up another interesting benefit.  Not only is FCoE and its associated technologies resolving todays problems, it is actually enabling a better future.  How good would it be to have 6 or more CNAs per blade server!  That’s 60Gbps+ of flexible bandwidth based on today’s current 10Gbps Enhanced Ethernet!

Now this becomes a real winner in light of the current raft virtualisation optimised CPUs on the market – think Intel Nehalem and associated  virtualisation technologies like Intel VT-x VT-c, VT-d VMDq – xpect the ratio of VMs per core to rocket skyward!  In order to keep pace with the advancements in chip technology the I/O subsystem needs to move in step.  CNAs and Lossless Enhanced Ethernet are vital to this.

This point aside, I recently had a hand in a design involving HP BladeSystem c7000 technology.  I remember during the design wishing that the system had CNAs instead of the Flex10 and Virtual Connect technologies.  Could have saved space and cabling as well as potentially given more flexibility.  However, this was prior to even FCoE being ratified by INCITS.

Obvious benefits summarised

So some obvious benefits that CNAs bring include – reduced number of PCI adapters, cables, space, power and cooling.  Nice!

Secondly, by supporting 10Gbps Enhanced Ethernet they provide greater bandwidth than existing adapters allowing more traffic types and volume to travel down the same stretch of cable.  Suddenly the wire once nirvana is now a reality (actually wire twice if you want true I/O multi-pathing).
 

NOTE: A quick heads up on throughput.  While CNAs provide access to 10Gbps CEE networks, not all services, including FCoE, are supported at full bandwidth.  For example, CNAs normally support Ethernet at 10Gbps but FCoE only runs at a max of either 4Gbps or 8Gbps.  This is in line with 4Gbps and 8Gbps FC – there is currently no standard for 10Gbps FC N-Port to F_Port connections.

<OK so this is about the half way point, so you might want to come up for some air here before discussing the interesting stuff>

Clever things with CNAs

Now to the deeper technical stuff………..

With 10Gbps Enhanced Ethernet, FCoE, CNAs and other associated technologies being relatively new, the standards bodies (such as INCITS T11 FC-BB-5 for FCoE) as well as the vendors are able to design with virtualisation in mind.  This is great!

Let’s mention a couple of technologies that bring a lot to the table in Hypervisor environments.

NPIV

The first technology worth mentioning is N_Port ID Virtualisation, otherwise known as NPIV.  NPIV is a T11 INCITS and ANSI standard initially developed by IBM and Emulex, and yes I know it’s not exactly new.

Because NPIV is not a new technology I wont spend much time on it here other than to say NPIV makes it possible for a single HBA to have several N_Port IDs and WWPNs, allowing virtual machines to be uniquely addressable on the SAN and therefore able to be managed on the SAN just like normal physical servers.
 

If it would be useful I can put something up on NPIV.  Just leave a comment at the bottom asking and if enough people ask I’ll post on it.

I/O Virtualisation (SRIOV)

So while NPIV allows multiple VMs to be uniquely addressable on the SAN side of an HBA, on the other side, the side of the servers PCI tree, there is still a 1:1 mapping between physical ports and addressable PCI devices.

The Middle-man

Early virtual server implementations see the hypervisor act as middleman, between the I/O adapter and the Virtual Machine (VM).  The VM never actually talks directly with the I/O adapter, always through a middle-man – the hypervisor.  But like any middleman scenario, it has its advantages and disadvantages.  The middleman would argue that he adds “value”, but is rarely so keen to point out that he also always takes a generous percentage of the spoils (in our case, I/O performance and CPU cycles).

Some of the advantages and disadvantages of having the hypervisor as the middleman might include –

Middle-man Advantages

• Hypervisors often provide snapshot capabilities
• Hypervisors often provide thin provisioning capabilities
• Many existing Hypervisor technologies are currently designed around this model

Middle-man Disadvantages

• Lower I/O performance.  Because the hypervisor handles all I/O to and from a VM, as well as the associated interrupt handling etc, this injects server-side latency in to the I/O path as well as stealing CPU cycles form the main role of the Hypervisor.
• Security concerns.  The fact that all I/O, to and from VMs, is seen and touched by the hypervisor may be a concern to some people :-S
• Limited feature set.  You are restricted to the features and functions provided by the Hypervisor and not exposed to the full feature set available from the manufacturers native driver….

A common example where having the Hypervisor act as the middle man is not seen as desirable is a high throughput OLTP type system.  These are rarely virtualised due to, among other things, the I/O performance impact associated with having the Hypervisor as the middle-man.

So what does SRIOV do to help?

NOTE: Single Root I/O Virtualisation (SRIOV) and Multi Root I/O Virtualisation (MRIOV) are extensions to PCIe brought to us courtesy of the electronics industry consortium known as the PCI Special Interests Group (PCI-SIG).

SRIOV implementations allow an I/O adapter, in co-operation with the Hypervisor, to be sliced up in to multiple virtual adapters.  Each of these uniquely addressable on the servers’ PCI tree.  Each virtual adapter can then be mapped directly to a virtual machine and in turn is directly addressable by that virtual machine – no more middle-man.  SRIOV based adapters may have dedicated I/O paths in silicon and work alongside other related technologies such as Intel VT-c, VT-d, VMDq…. to provide a huge variety of offloads and assists aimed at reducing the load on main CPU and memory systems and thus increasing system performance.  This mode of operation is sometimes referred to as “hypervisor bypass” mode (we should be thinking VMDirectPath by now) and offers close to full line rate, making virtualisation a more realistic option for transaction based and other I/O intensive systems like our example OLTP system.

This also opens the door to things like VMs running drivers provided by the manufacturers making new functionality immediately available without waiting for the Hypervisor middle-man to support it.  Obviously it all needs buy in from your Hypervisor …..

Of course SRIOV is a semi-open standard (I say semi-open because several months ago when I researched it you had to pay to get access to the standards docs) and like most standards each vendor is free to implement the specifics in their own unique and value-add way as well as to add more features and requirements around it.

So in a nutshell, NPIV enables a single port to have multiple discrete addresses on the SAN side, whereas SRIOV allows a single I/O adapter to have multiple discrete addresses internally on the servers PCI bus/tree.  The diagram below from a long time ago on the Cisco website shows an I/O card that can be partitioned in to 128 (0-127) virtual interfaces.
 

Diagram from the Cisco website

Say goodbye to hardware rip and replace

By implementing the above mentioned technologies it becomes possible to run a single cable to a server and then use management software to configure the virtual interfaces according to current requirements.  For example, removing a NIC function and adding an HBA function becomes just a software change – no requirement to physically swap out a PCI card or lift the floor and run new cables.  Simply make the change in software and the new device will be presented to the servers PCI tree, job done!  Sound good to anyone else?

There is more, but I doubt anyone would read more than this in one post.  Other topics can always be discussed in the comments section below….

So if you made it all the way to hear, thanks and I hope it was useful.  Please feel free to join the discussion either here on the blog site via the comments section below, or by following me on Twitter @nigelpoulton.  I only talk about storage and related technologies.

Nigel

PS.  I am an independent consultant and available for hire via the Contact Me page