The Blade is Dead! Long Live the Rack!

By | January 15, 2010

OK, so the blade and the blade enclosure are not about to disappear, but the shift towards the Rack as a unit of design and a unit of management suggests we may be about to witness the coronation of the Rack as the new King.  Well.. kind of…

As early as last summer I was involved in specking and purchasing some HP BladeSystem Matrix based solutions – essentially a shrink-wrapped Rack based solution that had compute, networking, storage and management tools.  Granted, HP Matrix is an early attempt and not much different from the norm, but a step towards the Rack Area Network (RAN).

So… Rack based solutions are on their way, and the way I see it – from speaking with peers, customers and vendors – the following two high level Rack based designs will be predominant and will slug it out over the next few years:

1.  The FCoE RAN Solution

Of the two solutions, this one most closely resembles what we know today.  The only major difference being the use of FCoE between the server and the Top of Rack (ToR) switch.  This solution requires Converged Network Adapters (CNA), copper twinax cabling and FCoE ToR switches.  As it happens this is really the only practical place that FCoE can currently be deployed.  Fortunately, however, the FCoE products in this space (the RAN) are maturing quickly – we already have 2nd generation, single chip, single driver code base, high performance CNAs shipping and supported by most good server vendors….

In the FCoE based RAN solution there is very little in the ways architectural change – no blurring of the server/network edges and no change to the design of servers or networks.  This gives the comfort factor. 

Anyway, the sketch below shows a high level view of of this type of solution –

FCoE RAN picture


2.  The IOV solution

This second solution is slightly more innovative.  It requires slight changes to existing server/blade designs, there is some blurring of the server and network edges, and some re-thinking of network design and management is required.  Not quite the same comfort factor that the FCoE RAN solution gives,but as the saying goes – No pain, No gain! 

The IOV solution can be summarised in the following –

Servers and blades are reduced to pure compute and memory.  The I/O components are disaggregated from the server chassis and re-housed in an external ToR I/O Director.  Servers connect to the external I/O cards by either PCIe cables or IB.  These I/O adapters can be CNAs or traditional NICs and HBAs.  They are next generation in that each one can be carved into multiple logical adapters which can each be dynamically assigned and unassigned to any server and VM within the Rack.  The I/O adapters and I/O Directors have built-in switching functionality, enabling traffic to be switched either within the I/O adapter or between adapters within the same I/O Director without the need to travel up to a traditional network switch (hairpinning in the adapter or I/O Director).  Essentially, access layer switching will be moved on to the PCIe I/O adapter!

The diagram below shows this at a high level –

IOV RAN picture


No room in the RAN

Personally I like the idea of using PCIe as the main interconnect within the Rack.  Every chipset on every server already has a bunch of PCIe bandwidth that is essentially….. well…. FREE!  Who doesn’t like the sound of that!?  10Gbps CEE and FCoE licensing of ports is … well…. definitely not free.

Of course there is the other side.  PCIe muscling Ethernet out of the RAN will not go down well with some, nor will implementing switches within NICs/CNAs and I/O Directors.  Not only will this tread on certain vendors markets and margins, it also brings with it several network design and management challenges.  But what the heck… we grow from our challenges and come out the better for it – right?  Point being, knee-jerk self-preservation type reactions from the network guys should be expected 😛

Final thought on FCoE

While the IOV solution could potentially muscle Ethernet out of the RAN, it can still branch out to FCoE switches in the core.  So either way, FCoE will play a role. 

And if we are being creative, we could run our I/O up to the ToR I/O Director over PCIe within the RAN and then branch out via a CNA in one of those I/O Directors to a core switch with FCoE ports.  One way of utilising FCoE ports that are currently available in core switches.

Interesting times!

PS. I will be featuring on the Wikibon FCoE Fact vs Fiction call on 2nd February along with Stu Miniman, Dave Graham and  Dennis Martin  If you’re interesting in FCoE put it in your calendar.

6 thoughts on “The Blade is Dead! Long Live the Rack!

  1. Cam Ford

    Hi Nigel,

    Interesting article. You have clearly addressed the Hardware side of the RAN, but what about the management model. The big question I would ask is……does converged fabrics…..either FCoE/CEE or IOV approach make management easier or more difficult. How does the RAN change the existing management model in the datacenter? As you have pointed out, the HP Blade Matrix is just a bundle of what everyone already does today, so there is not much gain in management simplicity with this approach…..only a simplified SKU for ordering….assuming that the stock bundle meets your needs.

    Now, I will also make the case that FCoE convegence actually makes management more difficult. There is a lot of comfort in having separate FC and Ethernet networks…..from a division of responsibility perspective, from a debug perspective, and from a provisioning and management perspective. On the other hand, I believe that the IOV approach actually simplifies the management model quite a bit. It readically simplfies provisioning, allows for granular control of expensive FC and Ethernet resources, and provides greater flexibility in change managment….and at he same time keeps the division of responsibility roughy the same…..but you know my view.

  2. J Michel Metz

    This is truly eye-opening stuff. I've never thought about running PCIe cables within a rack. I'd be interested to hear more about it. Got any pointers about where to look?

  3. Nigel Poulton Post author

    Hi Cam,

    Interestingly, when configuring the HP Blade MAtrix solution we dropped a couple of the software components from the “Matrix” solution and suddenly it wasn’t “Matrix” any more.. so went from something like a 2 week “time to ship” to 12 weeks! Just for taking a couple software licenses out! Anyway, Im digressing..

    As for management – if Im being honest, I try not to get into management discussion as I would probably rather pull my own teeth out 😉 Its not a very sexy topic – although I think iPhone type apps are a fresh idea. On FCoE and CEE/DCB management Im hoping things like ETS, DCBX and may be even some vendor value-add management tools might make things like dynamic b/width management etc simpler. If you have a single cable for LAN and SAN and manage it all through software then surely that has to be better than the old way of separate networks (cards, cables ports…) for LAN and SAN and adding more cards cables and ports if more b/width is needed….?

    Debugging, firmware management and things like that are definitely a concern though in FCoE networks.


  4. Nigel Poulton Post author

    J Metz,

    Im planning on writing more about it and a collaborativ post with a couple of other guys.

    A coupoe of companies might include NextIO and VirtenSys.


  5. John Obeto

    Hello Nigel,

    As you rightfully point out, the FCoE RAN solution is basically an evolutionary step from where we are today.
    In a fully-converged architecture (thanks for the moniker, HP), a revolutionary improvement from today’s infrastructure must  be delivered.
    As to your tweet, there is room in there for Microsoft.
    If we take the fact that Microsoft is not a hardware vendor as the prime factor, then it behooves Microsoft to make sure that their server software designs take these sort of advances into consideration.
    They will play in that space, no matter what it is.

  6. Nigel Poulton Post author


    I was thinking about what you said about management in your comment above and had an aha! moment. I misunderstood what you were getting at.

    Yes I think an IOV solution makes managing the RAN and your I/O environment far simpler. It offers a flexibility that will be a great addition to virtual server environments. I/O flexibility has been lagging and IOV solutions like those offered by Xsigo will bring it up to speed.

    Unified fabric technologies like FCoE (DCB) can still play a part though… but more than likely outside of the RAN and in the network core….


Leave a Reply

Your email address will not be published. Required fields are marked *


You can add images to your comment by clicking here.