On the topic of I/O Virtualisation, we look into a product that takes I/O adapters (NIC and HBA) out of physical servers and places them into a Top of Rack switch and extends PCIe out of the server chassis to the ToR switch.  The notion of extending PCIe out of the server and plumbing a rack with PCIe is very interesting.

In this inaugural episode of the Technical Deep Dive Podcast I'm joined by Greg Ferro from Etherealmind.com and Camden Ford from Xsigo Systems.  Cam talks to us about Xsigo Systems and the technologies and solutions they offer.

This is available as a two-parter due to length.  Part 1 is about 30 minutes long and Part 2 is about 40.  What can I say, you cant dive deep in 20 minutes!

Comments welcome!

As early as last summer I was involved in specking and purchasing some HP BladeSystem Matrix based solutions – essentially a shrink-wrapped Rack based solution that had compute, networking, storage and management tools.  Granted, HP Matrix is an early attempt and not much different from the norm, but a step towards the Rack Area Network (RAN).

So… Rack based solutions are on their way, and the way I see it – from speaking with peers, customers and vendors – the following two high level Rack based designs will be predominant and will slug it out over the next few years:

1.  The FCoE RAN Solution

Of the two solutions, this one most closely resembles what we know today.  The only major difference being the use of FCoE between the server and the Top of Rack (ToR) switch.  This solution requires Converged Network Adapters (CNA), copper twinax cabling and FCoE ToR switches.  As it happens this is really the only practical place that FCoE can currently be deployed.  Fortunately, however, the FCoE products in this space (the RAN) are maturing quickly – we already have 2nd generation, single chip, single driver code base, high performance CNAs shipping and supported by most good server vendors….

In the FCoE based RAN solution there is very little in the ways architectural change – no blurring of the server/network edges and no change to the design of servers or networks.  This gives the comfort factor. 

Anyway, the sketch below shows a high level view of of this type of solution -

2.  The IOV solution

This second solution is slightly more innovative.  It requires slight changes to existing server/blade designs, there is some blurring of the server and network edges, and some re-thinking of network design and management is required.  Not quite the same comfort factor that the FCoE RAN solution gives,but as the saying goes – No pain, No gain! 

The IOV solution can be summarised in the following -

Servers and blades are reduced to pure compute and memory.  The I/O components are disaggregated from the server chassis and re-housed in an external ToR I/O Director.  Servers connect to the external I/O cards by either PCIe cables or IB.  These I/O adapters can be CNAs or traditional NICs and HBAs.  They are next generation in that each one can be carved into multiple logical adapters which can each be dynamically assigned and unassigned to any server and VM within the Rack.  The I/O adapters and I/O Directors have built-in switching functionality, enabling traffic to be switched either within the I/O adapter or between adapters within the same I/O Director without the need to travel up to a traditional network switch (hairpinning in the adapter or I/O Director).  Essentially, access layer switching will be moved on to the PCIe I/O adapter!

The diagram below shows this at a high level -

No room in the RAN

Personally I like the idea of using PCIe as the main interconnect within the Rack.  Every chipset on every server already has a bunch of PCIe bandwidth that is essentially….. well…. FREE!  Who doesn’t like the sound of that!?  10Gbps CEE and FCoE licensing of ports is … well…. definitely not free.

Of course there is the other side.  PCIe muscling Ethernet out of the RAN will not go down well with some, nor will implementing switches within NICs/CNAs and I/O Directors.  Not only will this tread on certain vendors markets and margins, it also brings with it several network design and management challenges.  But what the heck… we grow from our challenges and come out the better for it – right?  Point being, knee-jerk self-preservation type reactions from the network guys should be expected

Final thought on FCoE

While the IOV solution could potentially muscle Ethernet out of the RAN, it can still branch out to FCoE switches in the core.  So either way, FCoE will play a role. 

And if we are being creative, we could run our I/O up to the ToR I/O Director over PCIe within the RAN and then branch out via a CNA in one of those I/O Directors to a core switch with FCoE ports.  One way of utilising FCoE ports that are currently available in core switches.

Interesting times!

PS. I will be featuring on the Wikibon FCoE Fact vs Fiction call on 2nd February along with Stu Miniman, Dave Graham and  Dennis Martin  If you’re interesting in FCoE put it in your calendar.

I thought it was pretty cool, so gave it a shout on Twitter.  It seems other folks thought it was more than pretty cool, with comments like way cool being thrown about.

While I think the idea is cool, I’m not sure how interested companies would be –> management and configuration changes to production kit from an iPhone ….. sounds a bit ahead of its time to me.   Cool, yes.  But is cool what major companies and managers of large Data Centres are looking for?  Remember that Xsigo kit is pretty squarely pitched at enterprise customers.  Would such applications cause more worries and concerns than they would solve problems?

Think about it this way……. Matt Davis pinged me back saying that he had once done “symcli over ssh over VPN ….. via my iPhone” to administer a Symmetrix DMX!!  Not sure what your initial thoughts are on hearing that, but mine were trepidation.  Sure, that’s pretty damn cool, but pretty flipping scary too!  Kudos to Matt, but more scary than cool in my books

But that is comparing apples to oranges, SYMCLI is an entirely different beast to the Xsigo UI.  If nothing else, one thing that the Xsigo iPhone app does demonstrate, is the simplicity of administering Xsigo kit.  I’d feel a lot safer administering a Xsigo director via an iPhone app than performing SYMCLI via an iPhone. 

Thought:  Hows about administering Cisco UCS via an iPhone?  I hear rumours that Cisco UCS is so complicated to configure, Cisco will only let you have it if they do the configuration work! 

Anyhow, even if the Xsigo kit is so simple to manage ….. not sure how comfortable the powers that be would be with it?  Id be interested in other peoples thoughts on this.

So assuming enterprise data centres are not quite ready (Im open to being wrong) for staff administering production kit via their iPhone, what would they be happy doing via an iPhone? 

How’s about iPhone apps for monitoring and reporting?

Not quite as cool, but once people become comfortable with monitoring and reporting via an iPhone…. may be bolder moves like configuring and managing might be more palatable.

Video courtesy of the XsigoVirtualIO’s Channel on YouTube

Heck may be apps like this will help propel the iPhone in to large corporate accounts as a serious business phone.  Probably about as close to enterprise as Apple are going to get.

Initial opinion: Very cool, unsure how popular it would be in enterprise accounts. May be slightly ahead of its time.

UPDATED OPINION (6th Jan 2010): The more and more I think about this, the more and more I like it.  We will no doubt be seeing more and more apps like these in the future, and this could be an absolute godsend under certain circumstances.  A great little tool to have in your toolbox.  It probably wont be long before we wonder how we got by without apps like these.  But the hard part will still be trying to convince the risk averse "steady as she goes" powers that be.

I’m interested to hear what anybody else thinks.  I've also placed a related poll on the Homepage please cast your vote.

 DISCLAIMER: I use a BlackBerry…….. and wish it was an iPhone

So…. when discussing I/O Virtualisation (IOV) it’s usually not long before the concept of hair-pinning arises.  The conversation usually goes like this –

If a single physical port is being carved into multiple virtual ports, can you switch traffic between two virtual ports sharing the same hardware, without the I/O having to leave the physical adapter?

The diagram below shows this concept, which is sometimes called Virtual Ethernet Bridging, switching within the adapter, hair-pinning or a hairpin turn –

In the above diagram there is a single server running two Virtual Machines.  The server has a single physical NIC configured to present two virtual NICs (vNIC).  vNIC 1 is assigned to VM1 and vNIC 2 is presented to VM2.  When VM1 sends data to VM2, in the diagram above, the physical NIC switches the packets without having to send them on to the edge switch – requiring the physical NIC to have a built-in switch, blurring the lines between server and network and a boat load more…..!

Sound interesting?  Sound disruptive?  Sound bold?  Sound cool? …….. It should!

Some Theoretical Examples

Like you might expect, there are several different ways that hair-pinning can be implemented and the diagrams below show 3 possibilities –

SR style

The above Single Root (SR) style implementation has a single physical NIC in a single physical server, with vNICs being presented only to VMs executing on a single physical server.  It can switch frames between vNICs carved from the same physical NIC.

  MR Style

The above can be either Single Root (SR) or Multi Root (MR) and removes the physical NIC from the server chassis and re-houses it in an external I/O Aggregator such as those offered by Xsigo and VirtenSys.  The potential exists to switch frames between vNICs from either the same physical or different physical NICs.  It can switch frames between VMs on different physical machines without touching the network.

Hairpinning in the switch

The above example actually performs the hairpin turn in the edge switch rather than the I/O adapter.  A single physical switch port is carved in to multiple virtual switch ports (referred to as VP in my above sketch).  Normally Ethernet switches don’t switch frames back out over the same prot that they arrived on. 

Like most new technologies and disruptive concepts, there are die-hard advocates for each approach.  There are also equally die-hard opponents.  Naturally this creates some interesting and often intense discussion – after all, taking a moment to think about the possibilities, this could potentially be very disruptive to existing networking models, and as always, there will be people who will not take this lying down.

Obviously there are pro’s and con’s as well as pre-req’s for each approach, and I intend to highlight and discuss some of them in future posts.

Hopefully this has shwd some light for now, and feel free to pitch in with thoughts, experiences and questions. 

Nigel

PS. FAO Standalone Sysadmin, the diagrams are not crayon   Oh and I kind of like them!

PPS.  Just so that folks don’t think Im making this stuff up again go search terms such as VNTag and VN-Link.  These are technologies that Cisco is all over and they relate to being able to hairpin on physical switch ports.  In fact when I get a minute I might get my crayons out again and add some detail to the diagrams

Other RAN and IOV related posts –

Introducing the RAN concept

IOV vNICs and vHBAs

And before anybody asks……. yes the stubble is designer, no Im not wearing a toupee, and the bags under my eyes are from 16 hours of travelling and then chatting in the hotel bar until silly o’clock in the morning

Nigel Poulton Tech Field Day Interview from Stephen Foskett on Vimeo.

Feel free to check out the following link to the GestaltIT Tech Field Day home page on Vimeo, with links to all of the uploaded videos form the event.  It was an amazing event with some very interesting technologies and companies in participation.

Nigel

If you work deploying VMware, Hyper-V, XenServer etc or if you have anything to do with the so called Virtual Data Centre, then you need to be all over IOV.

This is the second post in my mini-series on RAN and IOV.  In this particular post Im going to talk about the concept virtual adapters – Virtual NICs and Virtual HBAs.

The vNIC and the vHBA

The concept is simple: Take a physical NIC, perform some magic on it, and make it appear to the OS as multiple NICs.  Same goes for HBAs.

The diagram below shows a single physical NIC carved into 4 virtual NICs (vNIC) and a single HBA carved in to 4 virtual HBAs (vHBA). 

The benefits of such technologies should be obvious – higher utilisation, requirement for fewer physical NICs, fewer cables, and fewer edge switch ports – just to name a few.

Another added benefit is flexibility.  Assume you have a 10Gbps NIC in a server which you have carved in to 2 vNICs.  That server now has a requirement for an additional NIC.  You no longer have to power down the server, open it up, install a new physical card and then wait for new cables to be laid.  Instead, you can simply create a new vNIC, from the already installed physical NIC, and have it dynamically discovered and initialised by the OS.  All done in software – no cracking the server open and no waiting for cables!  Talk about reducing the time taken to implement a change, not to mention reducing the risk (there is always added risk when opening up servers and messing around in the floor void….).

The CNA

In the above diagram we labelled the vNIC solution as Good.  If we swap out that IOV capable NIC and replace it with a CNA (Converged Network Adapter) that can act as both a NIC and a CNA, then we suddenly have the ability to carve vNICs and vHBAs from a single physical Adapter.  The diagram below has been expanded to now include a CNA based solution.  The CNA solution is labelled as “Better”.

NOTE: I should point out that in most IOV solutions most of the  legwork is done in hardware.  The vNIC and vHBA devices are created in hardware, as well as most modern CNAs providing protocol offloads…..

Single Root

The above approach – of creating multiple virtual adapters from a single physical adapters located within a single server – falls under the category of Single Root (SR).  Single Root being another way of saying single server (PCIe root complex).  Single Root approaches are limited to presenting their virtual adapters to a single PCIe root complex – operating systems executing within a single physical server.

While talking about Single Root technologies I need to mention SR-IOV.  SR-IOV is a semi-open PCI-SIG standard for SR style I/O Virtualisation.  As with all standards, it will take time to take-off and become widely deployed, and is open to implementation interpretation (some vendors may implement SR-IOV slightly differently to others).

True PCI-SIG SR-IOV requires the following components to be SR-IOV aware in order to support it -

• BIOS
• OS/Hypervisor
• Physical I/O Adapter
• Driver

Changes to the above components are required due to the fact that SR-IOV changes the architecture and model for PCIe adapters.  It introduces the concept of Virtual Functions (VF) which look and feel like a normal physical I/O adapter.  However, VFs are a lightweight version of a physical I/O adapter and inherit some configuration options from their parent physical I/O adapter.  As a result, vNICs and vHBAs are enumerated on a servers PCIe device tree as VFs, and the BIOS, OS and driver must understand this.

Citrix recently demoed XenServer working with SR-IOV NICs and Intel VT-d technology.

While SR-IOV is a great technology and is destined to play a role in driving IOV forward, it is very early days for the technology and the many of the currently shipping IOV technologies are using proprietary techniques and not PCI-SIG SR-IOV.  Some of these technologies include -

• HP Virtual Connect Flex-10
• IBM Virtual Fabric

NOTE: While the Virtual Fabric for IBM BladeCenter solution is not currently SR-IOV, the chip that powers the Emulex CNA that sits at the heart of the solution, is SR-IOV capable….. just waiting for the other components (BIOS, OS, Drivers…) to catch up.

Good, better, BEST!

So far we have talked about SR style solutions where the vNIC and vHBA devices are only available to Operating Systems executing on the same physical server that the adapter is installed in.  While these technologies are all good and a step in the right direction, there exists a superior solution – Multi Root (MR).

Taking IOV to the next step involves removing the physical I/O adapters from the physical server chassis and re-house them in an external device that I am generically referring to as the I/O Aggregator. 

The diagram below has been expanded to include an example I/O Aggregator approach.

Such technologies can be referred to as Multi Root (MR).

There are already Multi Root I/O Aggregator style solutions shipping from the likes of NextIO, VirtenSys and Xsigo – all are delivering next generation IOV benefits today!

Of the currently available solutions, these MR technologies offer the greatest levels of virtualisation and flexibility and for me represent the future.  By removing the I/O adapter from within the physical confines of the server chassis, you enable any vNIC or vHBA to be assigned to any server.  Your physical server becomes entirely stateless from an I/O perspective!

I used to be excited about LOM style CNA implementations…. until I discovered I/O Aggregators.

NOTE: PCI-SIG also have a specification for MR-IOV.  However, I do not know of anybody deploying it at the moment

Moving Home

Opinion time here, but they way I see it, the I/O adapter is folding its underwear and packing its bags ready to ship out of the server chassis into a bigger, better and more comfortable new home – the I/O Aggregator. 

PCIe adapters in servers…… don’t be so yesterday

Would love to know what anybody else thinks.

Nigel

You can follow me on Twitter. I’m @nigelpoulton and I only talk about technology.

I am also available as an independant freelance consultant and can be contacted via the Contact Me page.

Other RAN and IOV related posts –

IOV and introducing hairpin turns

Introducing the RAN concept

The term, as well as the concept, of Rack Area Networking is one I’m hearing more and more often.  As a result of this, as well as the fact that I’m convinced this is going to be one of the most interesting and important areas of Data Center computing over the next few years, I’ve decided to write a mini-series on the topic. 

This post is instalment number 1 and is intended to introduce the concept and get the ball rolling.  The whole thing is a bit of me thinking out-loud and attempting to generate some awareness and conversation around the topic, as .  So please pitch in!

Rack Area Network – the concept

For me, Rack Area Networking, or RAN for short, is an umbrella term for most of the clever networking and I/O virtualization stuff that goes on within a rack – a 42u rack. 

With it being “Rack Area”, it is a close proximity network and as a result operates over very high-speed low-latency interconnects.  

Physically, RAN technologies include a new generation of at least the following: I/O adapters, cabling, Top of Rack (ToR), and may be even End of Row (EoR), switches.  However, for reasons which will become clear, the emphasis is heavily on the clever – technologies that enable the flexible, the dynamic and the virtual aspects.

For example, the I/O Adapters driving the RAN evolution are not just faster than the legacy adapters they are replacing, they have built-in cleverness – hardware virtualization and huge flexibility! 

Some of the other technologies that define and operate within the RAN include – SR-IOV, MR-IOV, vNIC, vHBA, CNA, FCoE, Hairpin-turns, switching in the adapter, VNTag, VN-Link…. just to name a few.  In future posts we will discuss most of them.

As well as the above new hardware and technologies, the RAN also requires and includes a new generation of management software and functionality.  True value is often in the software – the glue that holds it all together and makes it all happen.

The best part being, there are early RAN technologies already out there in the wild.  And they are already delivering real-world tangible benefits.

Some technologies driving the evolution…

It’s really important to note that while technologies in the RAN are experiencing a period of accelerated evolution, it is most definitely an evolution.  The changes are happening fast, but they are not huge disruptive changes.  For the most part, they are improvements and enhancements, albeit major, on what we already know and are comfortable with.  E.g. take PCIe adapters and create multiple virtual adapters (vNIC and vHBA) in hardware….

Just a few of the currently shipping RAN technologies include -

• HP Virtual Connect Flex-10
• IBM Virtual Fabric Solution w/ Emulex UCNA
• Cisco UCS w/ Palo adapter
• Xsigo I/O Director
• Virtensys VIO switches

NOTE: Let me know if I’ve missed any major RAN technologies off the list

Some of the above technologies are very much generation 1 and only a small step towards the RAN, bringing only small benefits.  Whereas others are a major step with huge benefits.  All vendors are scrambling to take the lead in this evolving area.  In later posts we’ll dig deep into most of them.

Blurring the Lines and Causing Havoc

Naturally, many of these technologies are challenging and threatening the traditional server/network edge configurations we are used to. 

Hairpin turns, switching in the adapter and avoiding edge switches are just some of the paradigm shifts that RAN technologies might force us to consider.  Such topics are the subject of intense and engaging debate.  All very interesting and some of the concepts are very cool!

In upcoming posts we’ll talk about the likes of – SR-IOV MR-IOV Hairpin-turns VirtenSys Flex-10 VEB Xsigo VNTag NextIO VNLink InfiniBand PCIe

Thanks for dropping by and feel free to throw in your penny’s worth.

Nigel

I am available as an independant freelance consultant and can be reached via the Contact Me page.

If you're thinking that I might just bemaking some of this stuff up or inventing buzwords, then you need to check out my follow-on RAN and IOV related posts listed below -

IOV vNICs and vHBAs

IOV and introducing hairpin turns

The way that I see it, Xsigo (pronounced “see-go”) are of particular interest for two reasons -

Firstly, they are playing in the steaming hotbed that is the I/O consolidation and virtualisation space.  This area of Data Center computing is probably experiencing its biggest period of change and upheaval since the birth of Local Area Networking.  Also, the I/O subsystems of modern servers and blades are becoming increasingly important in modern data centers.

In a nutshell, just about everything is changing in the Data Center I/O space!

RupturedMonkey advice to vendors: Now is not a time to stand still or try and defend your traditional core competencies. Move with the times or risk falling behind!

RupturedMonkey advice to consultants and architects: Now is not a good time to take a professional snooze. If you do, you might find that you dont recognise the world you wake up to. Stay awake!

The second reason Xsigo Systems are of interest is because they have an absolutely kick-ass product – the Xsigo VP780 I/O Director.  So lets talk about it……

The Xsigo VP780 I/O Director

Before digging in to the specs and architecture, I should point out that the VP780 I/O Director is the only offering from Xsigo!  Also, as well as being the only product they currently offer, it is also pitched squarely at enterprise customers.  I suppose one of their responses to that would be that it allows them to be laser focussed, but for me, my initial reaction was that this makes them a bit of a one trick pony……  Compare their I/O consolidation portfolio to the likes of Brocade, and especially Cisco, and you will see what I mean.

During their presentation at the recent GestaltIT Tech Field Day they did say that they are working on similar but scaled down offerings for the SMB, but nothing to announce at the moment.

However, despite being the only noteworthy member of the Xsigo family, the VP780 is no wimp!  On the contrary, in a one-on-one it would probably fancy its chances against any of its competitors.  I certainly wouldn’t bet against it from a technology point of view! 

Specs and Techs

The VP780 is a 4RU high-speed low-latency 780Gbps I/O consolidation platform that was over two and a half years in the making.  It provides PXE boot and boot from SAN across 20Gbps connections to your servers and makes cable once and do the rest in software a reality!

Below is a picture of the front panel of the one on display at GestaltIT Tech Field Day -

From a high level architecture point of view the VP780 has -

• Server-side connectivity via 20Gbps Infiniband XFP ports
• Network-side connectivity via 15 hot-plug slots that can be loaded with 1Gbps Ethernet, 10Gbps Ethernet and 4Gbps FC.
• Passive midplane
• High-speed low-latency internal switching fabric

Hopefully the scribble below will be helpful as I attempt to dig deeper and explain some of the main components. Later in the week I will record a whiteboard session and upload as a complimentary post.

What does it do – in a nutshell

In a nutshell, the Xsigo I/O Director does for I/O what VMware does for CPUs.  Only it has an added benefit that it removes the physical limits of the server chassis –> instead of installing your NICs and HBAs in your servers and then carving them into virtual adapters that can only be used by that server, you install your NICs and HBAs in the Xsigo VP780 chassis so that they can be carved up and dynamically allocated to any connected server.  In doing this, you are effectively moving the edge of the network out of the server, enabling servers to be entirely stateless from an I/O perspective. Cool.

Connections to your servers… the hardware stuff

The VP780 has 24 x 20Gbps Infiniband ports for connections to your servers (server-side in the above diagram). They utilise copper or optical CX4 cables and XFP interfaces and are terminated at the server side on Host Channel Adapters (HCA in Infiniband parlance and yes each connected server needs an HCA). These HCAs are not used directly by the OS, instead, host-side drivers work together with the Xsigo I/O Director to ensure that the appropriate vNIC and vHBA devices are available to the OS.  Of critical importance is that thes vNIC and vHBA devices work exactly as physical NICs and HBAs and the OS is none the wiser.

Some quick comments on these physical aspects –

1. The 20Gbps XFP interfaces are not hot swappable and not upgradeable to 40Gbps QDR Infiniband.  These Infiniband HCA and switch ports are more energy efficient, lower-latency and higher-throughput than their 10GigE counterpoarts.  They also support longer distances over copper meaning that copper is an option more than it is for 10GigE which currently has practical limits between 5-10 meters.

2. As these are CX4 Infiniband connections, you will need Infiniband Host Channel Adapters (HCA) in your servers.

3. XFP and copper CX4 is more power hungry than SFP+ and copper commonly used with 10Gbps CEE. However, XFP optical is not more power hungry than SFP+ optical.

On point 1 from above – this should not be seen as a major issue. 20Gbps to your servers is lightning fast by todays server standards, and with the current wave of PCIe 2.0, you’re unlikely to be pushing beyond 20bps anyway. 40Gbps models are planned, as well as faster HCA cards, although this will be down to Mellanox as the silicon is 3^rd partied form Mellanox.

Also, in reality,  how many people are racing to crack open servers and blades to upgrade I/O cards? Most people seem to be opting to buy newer servers and blades when higher bandwidth I/O is required – may be Nehalem-EX…  20Gbps is more than fast enough for the vast majority of todays applications and servers.

On point 2. Don’t be put off by the word Infiniband!

FUD watch: Infiniband is not a disease, nor is it dead!  It is actually a rock solid ultra high performance low-latency channel interconnect designed for data center use and high performance computing. In fact many of the worlds fastest supercomputers use and are built around Infiniband.  So its definitely alive and well.

You don’t have to learn a boat-load of new Infiniband skills. You will run a copper CX4 cable from the HCAs in your servers to the Xsigo director and that is about as much Infiniband as you will ever see or need to configure in the solution. The rest is normal Ethernet and FC.

Connections to your servers… the clever stuff

So, if all of this talk about Infiniband hasn’t scared you off, well done.

The Xsigo VP780 I/O Director allows you to carve its NIC and HBA resources in to virtual NICs (vNIC) and virtual HBAs (vHBA). Each of these virtual vNICs and vHBAs acts exactly like a normal physical NIC or HBA and thanks to some clever work in the server side drivers, Operating Systems (ESX, Windows, Linux etc) see just like they would physical NICs and HBAs.

Another thing not to be underestimated is that fact that the physical NIC and HBA hardware sits outside of the physical server or blade chassis.  This enables the physical servers to be stateless from and I/O point of view, and for virtual resources to be moved around from physical server to physical server with great ease.  The VP780 owns the server profiles which include MAC addresses and WWPNs etc and allows up to 16 vHBAs and 32 vNICs to be assigned to a single physical server, all of which can be created and deployed in literally seconds with no reboots.   Ideal for VMware and the c c cl cll cllllll clllllll cloud!?  Think thats the first time Ive said the “c” word in a blog.

In my opinion, the previous paragraph is of huge importance.  If you don’t think this is huge, then I suggest that you re-read it and take a minute or two to think about it.  This is flexibility like no other solution I know of.

Is it just me, or does this look and feel very much like MR-IOV (PCI-SIG Multi-Root I/O Virtualisation)? 

Does anybody else do anything like this?

NOTE: I’ll post on this in the near future, but I personally think that MR-IOV has huge potential to rock the I/O consolidation world, and I’m not alone in thinking that! However, there is a case for Infiniband being a better Rack Area Networking (RAN?) interconnect than PCIe. One for a future post if people are interested.

Connecting to existing backbones

On the network side of the VP780, there are 15 slots that can be populated with various modules.  Currently 3 module types are available -

1. 1 x 10Gbps Ethernet module
2. 10 x 1Gbps Ethernet module
3. 2 x 4Gbps HBA module

This gives you traditional LAN And SAN Connectivity, with FCoE and iSCSI offload being on the map.  So ,connecting to your existing LAN and SAN is “as easy as organising a tweet-up at TechFieldDay”   Your up-stream LAN and SAN is oblivious to the fact that the I/O is not initiated at the server chassis and just hums away as normal (NPIV is implemented on the SAN side of the HBAs).  The diagram below shows native FC connections coming out of the network side of the VP780 at the demo lab at VMware. 

At the moment, the VP780 has no support for FCoE.  Not a huge drawback as the standard and shipping products are still young, however, if they drag their heels over this they will fall behind in an important new and emerging market.  Something like the Emulex UCNA with its 10Gbps Ethernet, FCoE and iSCSI offload all on a single module would be like the cherry on the cake for this.

FUD Watch:  Be careful to note that the VP780 is not a switch.  True, it can switch frames between servers without passing traffic to the upstream network switch, but it does not HAVE to. It can forward the frames to the upstream switch if the switch supports hairpin switching.  So it does not have to alter existing network management models. However, there are several scenarios such as HPC or RDMA where performing hairpin switching over the internal IB fabric is beneficial for performance reasons. Choice is a good thing!

Nice Management GUI

While visiting with Xsigo at GestaltIT Tech Field Day I got my hands on some Xsigo kit including the management interface.  I was able to present vNIC’s and vHBA’s to ESX servers and have them picked and recognised on the fly by virtual machines. I was also able to play a little with some simple QOS features – increasing and decreasing bandwidth is very simple and also dynamic.  All simple stuff and worked a treat.  Oh and it has a CLI.

There is a ton more I could say, but this is already pretty long so I’ll wrap up with some final thoughts…..

Conclusion

The VP780 does some interesting stuff and in some respects is ahead of the curve. For instance –

• Performance to the server is 20Gbps over Infiniband
• Flexibility. Removing the physical NICs and HBAs from the server or blade chassis makes this a hugely flexible solution.
• Number of vNIC and vHBA devices that can be carved per physical card and presented to each host is more than most of the competition. E.g. HP VC Flex-10 and IBM Virtual Fabric can only create 4 virtual functions per port. This offer superior utilisation as a result.

As always though there is no perfect solution.  There is currently no FCoE, and both the Infiniband and the 10Gbps Ethernet options use XFP transceivers which are not as good as SFP+ when it comes to the likes of size, cost and power consumption.

There is also the fact that Xsigo are a relatively small and new company pitching to the enterprise.  From a technology point of view they are brilliant, but one has to wonder if they will still be around in 10 years time supporting and developing their products?

However, when all is said and done, I really like what they are offering. My final question for Camden Ford after his presentation was “can I have one for my garage”.  Says it all really.

If you are looking in to I/O consolidation and virtualization then you should definitely at least check out Xsigo… unless you’re too chicken!

Thoughts and comments welcome.

Nigel

You can follow me on Twitter where I talk about storage technologies (@nigelpoulton)

I am also available for hire as a free-lance consultant.

In my opinion the show was a real success with over 1,500 attendees, of which over 1,100 were end users and reseller delegates and the remainder made up of general riff-raff such as vendors, press and the likes….

One of the things I like to see at shows like these is hardware.  What can I say, Im just the kind of guy that gets a kick out of seeing hardware.  So for the rest of you out there like me – sit back and enjoy……..

First up, I was impressed to see a Symmetrix V-Max, blue strip light ablaze.  The only disappointment was that whenever I popped by to have a chat, somebody else was always being given an overview

Next up the IBM HS22 BladeCenter that was used to demo the new IBM Virtual Fabric, which is based on technology from IBM, BLADE Networks and Emulex.  A much needed addition to the IBM portfolio in my opinion.

Oh and while on the theme of IBM, here is another rack of IBM kit

As the IBM Virtual Fabric solution has an Emulex CNA in it, next up is Shawn Walsh from Emulex showing us that the Emulex UCNA is real and not a myth.

And a close up on a desk with a pen for lined up to give an idea of size

Then if you followed the girls with “Thin” written on their T-shirts you couldn’t miss the 3PAR InServe kit on show.  I plan on writing about 3PAR RAID MP and Persistent Cache, both of which are potentially very interesting technologies.  But seeing as 3PAR are attending the upcoming GestaltIT Field Day I might wait and see if I can glean some deep tech info from them.

Brocade also turned up with a rack load of kit, although hugely disappointing for me was the lack of an FCoE 10-24 blade in the DCX director.  Not to worry though, there was a B8000 top of the rack CEE/FCoE switch to keep me happy.

And a Brocade dual port CNA

I have some video footage from the Brocade booth that I will post some time next week.  So stay tuned.

Even the internet connected laptops were also of decent spec.  Below is a smart little HP laptop alongside my personal 11.1” Sony job – check out the grease marks on my trackpad and spacebar

And last but not least, the quality of freebies was good.  Aside from the standard pens and stress balls, I was particularly impressed with –

iPod Shuffle
Solio solar powered USB charger
3-in-1 pen/laser pen/1GB memory stick (James Bond style)
2GB micro SD with standard SD adapter 

The economy must be recovering!

As well as the above mentioned video footage from the Brocade booth, I also got some video footage from the Arista networks booth.  Keep an eye out for that as I plan to post them in the following week…

Nigel

You can follow me on Twitter where I talk about storage technologies (@nigelpoulton) and I am also available for hire as a consultant.