OK, so the blade and the blade enclosure are not about to disappear, but the shift towards the Rack as a unit of design and a unit of management suggests we may be about to witness the coronation of the Rack as the new King. Well.. kind of…
As early as last summer I was involved in specking and purchasing some HP BladeSystem Matrix based solutions – essentially a shrink-wrapped Rack based solution that had compute, networking, storage and management tools. Granted, HP Matrix is an early attempt and not much different from the norm, but a step towards the Rack Area Network (RAN).
So… Rack based solutions are on their way, and the way I see it – from speaking with peers, customers and vendors – the following two high level Rack based designs will be predominant and will slug it out over the next few years:
1. The FCoE RAN Solution
Of the two solutions, this one most closely resembles what we know today. The only major difference being the use of FCoE between the server and the Top of Rack (ToR) switch. This solution requires Converged Network Adapters (CNA), copper twinax cabling and FCoE ToR switches. As it happens this is really the only practical place that FCoE can currently be deployed. Fortunately, however, the FCoE products in this space (the RAN) are maturing quickly – we already have 2nd generation, single chip, single driver code base, high performance CNAs shipping and supported by most good server vendors….
In the FCoE based RAN solution there is very little in the ways architectural change – no blurring of the server/network edges and no change to the design of servers or networks. This gives the comfort factor.
Anyway, the sketch below shows a high level view of of this type of solution –
2. The IOV solution
This second solution is slightly more innovative. It requires slight changes to existing server/blade designs, there is some blurring of the server and network edges, and some re-thinking of network design and management is required. Not quite the same comfort factor that the FCoE RAN solution gives,but as the saying goes – No pain, No gain!
The IOV solution can be summarised in the following –
Servers and blades are reduced to pure compute and memory. The I/O components are disaggregated from the server chassis and re-housed in an external ToR I/O Director. Servers connect to the external I/O cards by either PCIe cables or IB. These I/O adapters can be CNAs or traditional NICs and HBAs. They are next generation in that each one can be carved into multiple logical adapters which can each be dynamically assigned and unassigned to any server and VM within the Rack. The I/O adapters and I/O Directors have built-in switching functionality, enabling traffic to be switched either within the I/O adapter or between adapters within the same I/O Director without the need to travel up to a traditional network switch (hairpinning in the adapter or I/O Director). Essentially, access layer switching will be moved on to the PCIe I/O adapter!
The diagram below shows this at a high level –
No room in the RAN
Personally I like the idea of using PCIe as the main interconnect within the Rack. Every chipset on every server already has a bunch of PCIe bandwidth that is essentially….. well…. FREE! Who doesn’t like the sound of that!? 10Gbps CEE and FCoE licensing of ports is … well…. definitely not free.
Of course there is the other side. PCIe muscling Ethernet out of the RAN will not go down well with some, nor will implementing switches within NICs/CNAs and I/O Directors. Not only will this tread on certain vendors markets and margins, it also brings with it several network design and management challenges. But what the heck… we grow from our challenges and come out the better for it – right? Point being, knee-jerk self-preservation type reactions from the network guys should be expected 😛
Final thought on FCoE
While the IOV solution could potentially muscle Ethernet out of the RAN, it can still branch out to FCoE switches in the core. So either way, FCoE will play a role.
And if we are being creative, we could run our I/O up to the ToR I/O Director over PCIe within the RAN and then branch out via a CNA in one of those I/O Directors to a core switch with FCoE ports. One way of utilising FCoE ports that are currently available in core switches.