While eating out recently with a group of top industry techies* in Tokyo Japan, we had a short light-hearted discussed of the above.  There was a definite difference of opinion; some of the people involved felt it made absolutely no difference at all, whereas others (including me)

think it may make a difference, albeit my be small….  May be we are all wrong and it makes a big diference!?

* Top industry techies included: Greg Knieriemen, Chris Evans, Michael Hay, Devang Panchigar, Robin Harris and the Rickatron himslef Mr. Rick Vanoever.  The Ghelfster was also there, and although she is top notch, strictly speaking Carli is not a techie.

While I do think it makes a difference, I’m not suggesting it makes a huge difference – its certainly nowhere near as important as it is with consumer goods – but I’m open to being wrong.  After all, I’m a techie and not in marketing.

Let me just throw together a quick list of some of the reasons I think it makes a difference -

1. It’s a conversation point for techies.  Having folks talk about your company and your kit is almost always a good thing.
2. It helps in marketing.  And we all know that effective marketing is often more important than technical merit or technical superiority.
3. Things that are aesthetically pleasing on the eye leave us with positive thoughts and feelings about the object. And that doesn’t just apply to good-looking girls!

While on the topic, I’m also one of those that thinks getting your product on poplar TV shows and films is a good idea.  An example that immediately comes to mind is the frequent and positive appearances of Cisco and Apple kit on the popular TV show 24.  In one episode, Chloe O’brien (renowned for being able to hack though any IT security system) makes a statement along the lines of “they have the new Cisco proprietary firewall system, its impossible to hack”.  Corny, I know, but its sewing information and impressions to people.

And then there’s the recent appearance of an EMC VMAX on the same show.  While some point out that one of the VMAX arrays had smoke coming out of it, the important thing was that it powered back up and still worked

WARNING: Don’t test burning your own VMAX in your own Data Center.  The above was performed by professionals with all required safety equipment at hand

And let’s face it, Apple, Cisco and EMC know how to market a product!

If names are important then surely…..

Also, there is no doubt that companies (including vendors of Data Center hardware) think product names are important.  So if something as seemingly unimportant as a product name (when compared to security or data availability etc) is important, then surely physical product aesthetics are also important?  Both are part of product image and branding.

TIP: Go hire an ex-Apple dude to design the front doors and front bezels to your Data Center kit.  It won’t break the bank or introduce bugs to your code, but it might have a positive impact on branding and and may be even sales.

Just my thoughts.  Please take a minute to register your opiion in the poll below -

Feel free to chime in with your own opinions, as well as images of the good the bad and the ugly of Data Center products.  Here’s quick image that I’ll leave you to make your own mind up on -

Nigel

Recently, I've even talked to industry experts as well as HDS folks about the fact that they only have two of the major three components that make up the increasingly popular vertical stack – with networking being the missing link.

NOTE: I define the three major components as Compute, network, and storage.  I know there’s more to it than that, but they suffice for this short article.

I don't know if it's just me, but the fact that Hitachi has its own networking kit – from Ethernet switches to MPLS – puts a whole different spin on their planned vertically integrated stack.

We already know they do storage, they recently announced that they will soon start shipping their BladeSymphony server platform (which has been around for a while in ASIA/Japan) to the rest of the world as part of their Hitachi Unified Compute Platform (UCP). But Hitachi built networking was notably missing from the announcement.

IMPORTANT NOTE:  I am in no way suggesting Hitachi have any plans of pushing this networking kit out with their Unified Computing Platform (UCP) solution when it finally ships. In fact as far as Im aware, they have no intention of doing this (yet).  This post is merely pointing out to others that Hitachi does have its own networking kit and there are some obvious synergies for the future….

Anyway, the whole thing came to light today while attending the Hitachi uValue convention in Tokyo Japan.  First up, Chris Evans pointed the following poster out to me -

Then we walked around the corner only to be presented with a boatload of the actual kit on display.  Feast your eyes on these -

Note: The booth guys (Japanese) tell me that Alaxala is a joint venture between Hitachi and NEC. Seems to be true from a quick search on Google.

Granted…… In true Hitachi style, they are not the prettiest things in the data centre – no doubt the old "form over function" thing – but appearance is far from the most important thing when choosing a product.

Other than the above, I literally know nothing about the kit – the booth guys spoke very little English, and in true British style I speak about 0.8 words of Japanese.  So all I know is that the Ethernet switches that do 1Gbps and 10Gbps.  There is nor Fibre Channel. But when I asked the booth guys (the ulitmate source of all truth and knowledge) about competing with other vertical stacks such as VCE/vBlock, I got the impression they were suggesting they might be working on FC.  Of course these are just booth guys and we understood a grand total of about 25 words between us

Although they apparently sell a fair amount in Japan (and potentially ASIA but Im not sure about the last part sue to language barriers), I know full well that you don’t just break in to the networking space.  However, apparently the switching technology in the back of the Hitachi BladeSymphony kit is Alaxala.  So that’s a potential toe in the door of the Data Centre (assuming HDS manage to make a good go of selling Hitachi BladeSymphony blade servers and their Unified Computing Platform, UCP).

Just thought Id let people know that in the background Hitachi does actually have in-house networking skills and kit.

Courteous comments welcome.

Nigel

DISCLAIMER: Hitachi picked up the bill for this trip to the Hitachi uValue convention in Tokyo (flights, food and hotel). For the record I will have an airport parking charge of ~£75GBP to pick up myself when I land tomorrow.

In this inaugural episode of the Technical Deep Dive Podcast I'm joined by Greg Ferro from Etherealmind.com and Camden Ford from Xsigo Systems.  Cam talks to us about Xsigo Systems and the technologies and solutions they offer.

This is available as a two-parter due to length.  Part 1 is about 30 minutes long and Part 2 is about 40.  What can I say, you cant dive deep in 20 minutes!

Comments welcome!

The term, as well as the concept, of Rack Area Networking is one I’m hearing more and more often.  As a result of this, as well as the fact that I’m convinced this is going to be one of the most interesting and important areas of Data Center computing over the next few years, I’ve decided to write a mini-series on the topic. 

This post is instalment number 1 and is intended to introduce the concept and get the ball rolling.  The whole thing is a bit of me thinking out-loud and attempting to generate some awareness and conversation around the topic, as .  So please pitch in!

Rack Area Network – the concept

For me, Rack Area Networking, or RAN for short, is an umbrella term for most of the clever networking and I/O virtualization stuff that goes on within a rack – a 42u rack. 

With it being “Rack Area”, it is a close proximity network and as a result operates over very high-speed low-latency interconnects.  

Physically, RAN technologies include a new generation of at least the following: I/O adapters, cabling, Top of Rack (ToR), and may be even End of Row (EoR), switches.  However, for reasons which will become clear, the emphasis is heavily on the clever – technologies that enable the flexible, the dynamic and the virtual aspects.

For example, the I/O Adapters driving the RAN evolution are not just faster than the legacy adapters they are replacing, they have built-in cleverness – hardware virtualization and huge flexibility! 

Some of the other technologies that define and operate within the RAN include – SR-IOV, MR-IOV, vNIC, vHBA, CNA, FCoE, Hairpin-turns, switching in the adapter, VNTag, VN-Link…. just to name a few.  In future posts we will discuss most of them.

As well as the above new hardware and technologies, the RAN also requires and includes a new generation of management software and functionality.  True value is often in the software – the glue that holds it all together and makes it all happen.

The best part being, there are early RAN technologies already out there in the wild.  And they are already delivering real-world tangible benefits.

Some technologies driving the evolution…

It’s really important to note that while technologies in the RAN are experiencing a period of accelerated evolution, it is most definitely an evolution.  The changes are happening fast, but they are not huge disruptive changes.  For the most part, they are improvements and enhancements, albeit major, on what we already know and are comfortable with.  E.g. take PCIe adapters and create multiple virtual adapters (vNIC and vHBA) in hardware….

Just a few of the currently shipping RAN technologies include -

• HP Virtual Connect Flex-10
• IBM Virtual Fabric Solution w/ Emulex UCNA
• Cisco UCS w/ Palo adapter
• Xsigo I/O Director
• Virtensys VIO switches

NOTE: Let me know if I’ve missed any major RAN technologies off the list

Some of the above technologies are very much generation 1 and only a small step towards the RAN, bringing only small benefits.  Whereas others are a major step with huge benefits.  All vendors are scrambling to take the lead in this evolving area.  In later posts we’ll dig deep into most of them.

Blurring the Lines and Causing Havoc

Naturally, many of these technologies are challenging and threatening the traditional server/network edge configurations we are used to. 

Hairpin turns, switching in the adapter and avoiding edge switches are just some of the paradigm shifts that RAN technologies might force us to consider.  Such topics are the subject of intense and engaging debate.  All very interesting and some of the concepts are very cool!

In upcoming posts we’ll talk about the likes of – SR-IOV MR-IOV Hairpin-turns VirtenSys Flex-10 VEB Xsigo VNTag NextIO VNLink InfiniBand PCIe

Thanks for dropping by and feel free to throw in your penny’s worth.

Nigel

I am available as an independant freelance consultant and can be reached via the Contact Me page.

If you're thinking that I might just bemaking some of this stuff up or inventing buzwords, then you need to check out my follow-on RAN and IOV related posts listed below -

IOV vNICs and vHBAs

IOV and introducing hairpin turns

The way that I see it, Xsigo (pronounced “see-go”) are of particular interest for two reasons -

Firstly, they are playing in the steaming hotbed that is the I/O consolidation and virtualisation space.  This area of Data Center computing is probably experiencing its biggest period of change and upheaval since the birth of Local Area Networking.  Also, the I/O subsystems of modern servers and blades are becoming increasingly important in modern data centers.

In a nutshell, just about everything is changing in the Data Center I/O space!

RupturedMonkey advice to vendors: Now is not a time to stand still or try and defend your traditional core competencies. Move with the times or risk falling behind!

RupturedMonkey advice to consultants and architects: Now is not a good time to take a professional snooze. If you do, you might find that you dont recognise the world you wake up to. Stay awake!

The second reason Xsigo Systems are of interest is because they have an absolutely kick-ass product – the Xsigo VP780 I/O Director.  So lets talk about it……

The Xsigo VP780 I/O Director

Before digging in to the specs and architecture, I should point out that the VP780 I/O Director is the only offering from Xsigo!  Also, as well as being the only product they currently offer, it is also pitched squarely at enterprise customers.  I suppose one of their responses to that would be that it allows them to be laser focussed, but for me, my initial reaction was that this makes them a bit of a one trick pony……  Compare their I/O consolidation portfolio to the likes of Brocade, and especially Cisco, and you will see what I mean.

During their presentation at the recent GestaltIT Tech Field Day they did say that they are working on similar but scaled down offerings for the SMB, but nothing to announce at the moment.

However, despite being the only noteworthy member of the Xsigo family, the VP780 is no wimp!  On the contrary, in a one-on-one it would probably fancy its chances against any of its competitors.  I certainly wouldn’t bet against it from a technology point of view! 

Specs and Techs

The VP780 is a 4RU high-speed low-latency 780Gbps I/O consolidation platform that was over two and a half years in the making.  It provides PXE boot and boot from SAN across 20Gbps connections to your servers and makes cable once and do the rest in software a reality!

Below is a picture of the front panel of the one on display at GestaltIT Tech Field Day -

From a high level architecture point of view the VP780 has -

• Server-side connectivity via 20Gbps Infiniband XFP ports
• Network-side connectivity via 15 hot-plug slots that can be loaded with 1Gbps Ethernet, 10Gbps Ethernet and 4Gbps FC.
• Passive midplane
• High-speed low-latency internal switching fabric

Hopefully the scribble below will be helpful as I attempt to dig deeper and explain some of the main components. Later in the week I will record a whiteboard session and upload as a complimentary post.

What does it do – in a nutshell

In a nutshell, the Xsigo I/O Director does for I/O what VMware does for CPUs.  Only it has an added benefit that it removes the physical limits of the server chassis –> instead of installing your NICs and HBAs in your servers and then carving them into virtual adapters that can only be used by that server, you install your NICs and HBAs in the Xsigo VP780 chassis so that they can be carved up and dynamically allocated to any connected server.  In doing this, you are effectively moving the edge of the network out of the server, enabling servers to be entirely stateless from an I/O perspective. Cool.

Connections to your servers… the hardware stuff

The VP780 has 24 x 20Gbps Infiniband ports for connections to your servers (server-side in the above diagram). They utilise copper or optical CX4 cables and XFP interfaces and are terminated at the server side on Host Channel Adapters (HCA in Infiniband parlance and yes each connected server needs an HCA). These HCAs are not used directly by the OS, instead, host-side drivers work together with the Xsigo I/O Director to ensure that the appropriate vNIC and vHBA devices are available to the OS.  Of critical importance is that thes vNIC and vHBA devices work exactly as physical NICs and HBAs and the OS is none the wiser.

Some quick comments on these physical aspects –

1. The 20Gbps XFP interfaces are not hot swappable and not upgradeable to 40Gbps QDR Infiniband.  These Infiniband HCA and switch ports are more energy efficient, lower-latency and higher-throughput than their 10GigE counterpoarts.  They also support longer distances over copper meaning that copper is an option more than it is for 10GigE which currently has practical limits between 5-10 meters.

2. As these are CX4 Infiniband connections, you will need Infiniband Host Channel Adapters (HCA) in your servers.

3. XFP and copper CX4 is more power hungry than SFP+ and copper commonly used with 10Gbps CEE. However, XFP optical is not more power hungry than SFP+ optical.

On point 1 from above – this should not be seen as a major issue. 20Gbps to your servers is lightning fast by todays server standards, and with the current wave of PCIe 2.0, you’re unlikely to be pushing beyond 20bps anyway. 40Gbps models are planned, as well as faster HCA cards, although this will be down to Mellanox as the silicon is 3^rd partied form Mellanox.

Also, in reality,  how many people are racing to crack open servers and blades to upgrade I/O cards? Most people seem to be opting to buy newer servers and blades when higher bandwidth I/O is required – may be Nehalem-EX…  20Gbps is more than fast enough for the vast majority of todays applications and servers.

On point 2. Don’t be put off by the word Infiniband!

FUD watch: Infiniband is not a disease, nor is it dead!  It is actually a rock solid ultra high performance low-latency channel interconnect designed for data center use and high performance computing. In fact many of the worlds fastest supercomputers use and are built around Infiniband.  So its definitely alive and well.

You don’t have to learn a boat-load of new Infiniband skills. You will run a copper CX4 cable from the HCAs in your servers to the Xsigo director and that is about as much Infiniband as you will ever see or need to configure in the solution. The rest is normal Ethernet and FC.

Connections to your servers… the clever stuff

So, if all of this talk about Infiniband hasn’t scared you off, well done.

The Xsigo VP780 I/O Director allows you to carve its NIC and HBA resources in to virtual NICs (vNIC) and virtual HBAs (vHBA). Each of these virtual vNICs and vHBAs acts exactly like a normal physical NIC or HBA and thanks to some clever work in the server side drivers, Operating Systems (ESX, Windows, Linux etc) see just like they would physical NICs and HBAs.

Another thing not to be underestimated is that fact that the physical NIC and HBA hardware sits outside of the physical server or blade chassis.  This enables the physical servers to be stateless from and I/O point of view, and for virtual resources to be moved around from physical server to physical server with great ease.  The VP780 owns the server profiles which include MAC addresses and WWPNs etc and allows up to 16 vHBAs and 32 vNICs to be assigned to a single physical server, all of which can be created and deployed in literally seconds with no reboots.   Ideal for VMware and the c c cl cll cllllll clllllll cloud!?  Think thats the first time Ive said the “c” word in a blog.

In my opinion, the previous paragraph is of huge importance.  If you don’t think this is huge, then I suggest that you re-read it and take a minute or two to think about it.  This is flexibility like no other solution I know of.

Is it just me, or does this look and feel very much like MR-IOV (PCI-SIG Multi-Root I/O Virtualisation)? 

Does anybody else do anything like this?

NOTE: I’ll post on this in the near future, but I personally think that MR-IOV has huge potential to rock the I/O consolidation world, and I’m not alone in thinking that! However, there is a case for Infiniband being a better Rack Area Networking (RAN?) interconnect than PCIe. One for a future post if people are interested.

Connecting to existing backbones

On the network side of the VP780, there are 15 slots that can be populated with various modules.  Currently 3 module types are available -

1. 1 x 10Gbps Ethernet module
2. 10 x 1Gbps Ethernet module
3. 2 x 4Gbps HBA module

This gives you traditional LAN And SAN Connectivity, with FCoE and iSCSI offload being on the map.  So ,connecting to your existing LAN and SAN is “as easy as organising a tweet-up at TechFieldDay”   Your up-stream LAN and SAN is oblivious to the fact that the I/O is not initiated at the server chassis and just hums away as normal (NPIV is implemented on the SAN side of the HBAs).  The diagram below shows native FC connections coming out of the network side of the VP780 at the demo lab at VMware. 

At the moment, the VP780 has no support for FCoE.  Not a huge drawback as the standard and shipping products are still young, however, if they drag their heels over this they will fall behind in an important new and emerging market.  Something like the Emulex UCNA with its 10Gbps Ethernet, FCoE and iSCSI offload all on a single module would be like the cherry on the cake for this.

FUD Watch:  Be careful to note that the VP780 is not a switch.  True, it can switch frames between servers without passing traffic to the upstream network switch, but it does not HAVE to. It can forward the frames to the upstream switch if the switch supports hairpin switching.  So it does not have to alter existing network management models. However, there are several scenarios such as HPC or RDMA where performing hairpin switching over the internal IB fabric is beneficial for performance reasons. Choice is a good thing!

Nice Management GUI

While visiting with Xsigo at GestaltIT Tech Field Day I got my hands on some Xsigo kit including the management interface.  I was able to present vNIC’s and vHBA’s to ESX servers and have them picked and recognised on the fly by virtual machines. I was also able to play a little with some simple QOS features – increasing and decreasing bandwidth is very simple and also dynamic.  All simple stuff and worked a treat.  Oh and it has a CLI.

There is a ton more I could say, but this is already pretty long so I’ll wrap up with some final thoughts…..

Conclusion

The VP780 does some interesting stuff and in some respects is ahead of the curve. For instance –

• Performance to the server is 20Gbps over Infiniband
• Flexibility. Removing the physical NICs and HBAs from the server or blade chassis makes this a hugely flexible solution.
• Number of vNIC and vHBA devices that can be carved per physical card and presented to each host is more than most of the competition. E.g. HP VC Flex-10 and IBM Virtual Fabric can only create 4 virtual functions per port. This offer superior utilisation as a result.

As always though there is no perfect solution.  There is currently no FCoE, and both the Infiniband and the 10Gbps Ethernet options use XFP transceivers which are not as good as SFP+ when it comes to the likes of size, cost and power consumption.

There is also the fact that Xsigo are a relatively small and new company pitching to the enterprise.  From a technology point of view they are brilliant, but one has to wonder if they will still be around in 10 years time supporting and developing their products?

However, when all is said and done, I really like what they are offering. My final question for Camden Ford after his presentation was “can I have one for my garage”.  Says it all really.

If you are looking in to I/O consolidation and virtualization then you should definitely at least check out Xsigo… unless you’re too chicken!

Thoughts and comments welcome.

Nigel

You can follow me on Twitter where I talk about storage technologies (@nigelpoulton)

I am also available for hire as a free-lance consultant.

Enjoy the video and then feel free to comment on my thoughts below –

First up, Arista is start-up networking company, and like all networking companies…….. they have Cisco in their crosshairs.

Although low cost seems to be their trump card, to be fair to Arista they are also touting performance as another key advantage.  Better performance, literally at a fraction of the Cisco price – if the guys at the booth are to be believed.  However, as I mention in a previous post, there is more to value than price.  It may turn out to be very short sighted to jump all over short term cap-ex savings at the true expense of the long term good.

However, the current down economy is probably creating ‘too good to pass-up’ opportunities for companies like Arista, who come in at a value price-point- most companies being forced to think more and more about cost, thus creating windows of opportunity for those coming in at significantly lower price-points.

However, when at the booth, the Arista guys did a great job ensuring that their products didn’t have that ‘value’ feel.  Despite the fact that they are apparently way cheaper than Cisco, they weren’t over-stressing that point and wanted to talk a lot about performance.

Technology and performance overview

So a quick overview of what Arista have on the table from a product and performance point of view….

Basically they are offering high-speed low latency 10Gbps Ethernet products.  Oh and apparently they are cheap…cheap enough for my garage??.

However, if you listened closely to the video, they only have DCB/CEE “on the roadmap”.  This, in my opinion is a huge shortfall in the current shipping products.  As each day goes by I am becoming more and more of the opinion that to be a true data center switch, you need CEE and FCoE.  Yes, 10Gbps without the goodness of CEE and FCoE might suffice for now, however, it will be found wanting in the very near future.  More on this later though….

I am impressed at the published latencies and port count as well as the dual hot-swap power supplies and reversible airflow (to change the airflow from front-to-rear, to rear-to-front requires swapping out the fans).  However, my gut reaction is that this is almost a gimmick in comparison to features like CEE and FCoE. Sorry that I keep coming back to this but I can’t stress it enough.
 

NOTE:  In the interest of fairness, should point out that when at HP Colorado Springs recently I was knocked back when I was told that one of the hardest things to engineer for new servers and blade chassis is the power supply units (PSU).  So I am careful not to totally dismiss the reversible airflow and redundant hot plug PSUs as totally insignificant.

7100 Series Data Center Switches

The 10Gbps 7100 series SFP+ based switches come in either 24 or 48 port-count models and you can choose between a non-oversubscribed and oversubscribed models.  You can also buy similar 10GBASE-T models which take standard RJ45 connectors and work with existing Cat 5/6/… cables.

When first at the booth (without my camera) I raised my eyebrows when introduced to the 10GBASE-T model.  As I raised my eyebrows the guy giving me the tour (different guy to the one on the video) pre-empted me and said “your are going to say ‘Wow’ aren’t you. But I had to admit that I was actually going to say “Why”…..

My initial reaction was why would anybody want to do that from a power cooling and latency standpoint.  Essentially SFP+ up to 10 metres has people talking about ~0.1W nominal power draw whereas 10GBASE-T RJ45 cables are closer to 4W at approx 10metres, but obviously offers greater distances at the expense of more power.  And that’s not to mention the inferior transceiver based latency that 10GBASE-T has compared to SFP+. 

Obviously a 10GBASE-T model may make the jump to 10Gbps Ethernet simpler for some people, I personally believe this approach is a false economy of sorts and hedges up the route to the full benefits of 10Gbps, and greater, based on CEE. 

Pass-Through module

They also had a 10Gbps Pass-through module on display.  This pass-through module has 14 internal and 14 external linespeed connection, with no oversubscription and is designed for the IBM BladeCenter H.  The module only has 4 physical ports on the external faceplate and achieves the 14 external port count via 10GBASE-CR QuadSFP to SFP+ splitter cables.  This is the copper twinax 4-way splitter cable seen in the video.  The module and is apparently very competitively priced compared to competitive offerings from the likes of BLADE Network Technologies.  Although features and functionality might be less than that of the BLADE Networks offering, apparently performance is not.

Nice, but no game changers

So, a lot of talk about feds and speeds.  Interesting for some, but talking feeds and speeds, so Im told, is sooooo last year.  EMC and HDS used to do it all the time until analysts and customers told them they didn’t care.  Feeds and speeds are fine, but not everything (at least not to the vast majority).

So for feeds and speeds the Arista kit seems to be very good and very competitively priced. If latency and performance are critical for you then Arista are definitely worth a look.

However, away from feeds and speeds was where there were signs of weakness. 

In my opinion, for a networking product to be a true Data Center product it has to play in the converged networking space (IP, FC, RDMA..).  At the moment, Arista products do not.

First, there is no Converged Enhanced Ethernet (CEE/DCB).  For me this is vital to the future of Data Center networking.  I know they guy on the video said that Arista doesn’t do anything that is not standards based, alluding to DCB/CEE standards not yet being fully ratified.  However, Im willing to bet they do jumbo frames, and I’d like to see them produce the IEEE 802 standards documents for jumbo frames

Sorry, that was a cheap shot, but I think comments like these are poor and will not help you out in the future when you are playing catch up.

Second, these are pure Ethernet products.  No Fibre Channel ports.  This is understandable as they have no FC legacy or expertise. 

In the very short term, while people use the edge in approach for implementing a unified fabric, this means that Arista products will not be an option.  But more importantly, beyond this, once end-to-end FCoE becomes mainstream, where will this leave Arista?  Having no FC expertise that I’m aware of will make implementing FCoE (FCF and all the other FCoE related functions) in their products very time consuming and difficult.  They will likely have to acquire a company that has FCoE or hire some engineers who know it, and hire them fast!

Although Data Center networking is converging on Ethernet, it is not converging on the Ethernet that is widely deployed today.  In fact, the Ethernet that it is converging on (CEE) bares less resemblance to todays 1Gbps lossy Ethernet than most realise.  So just because you are an Ethernet company today, and Data Center networking is convergi ng on Ethernet, does NOT automatically mean you are well placed for the future.

Looking to the future

So…….. normally, it is the large companies, those with the large market share and legacy architectures that struggle to change with the times (think Cisco).  The newer and more agile companies (think Arista) often beat them out of the blocks and are tearing away around the first bend before the likes of Cisco are even out of the blocks.  Not so this time! 

Cisco are actually heading around the first corner with early deployments of CEE and FCoE on the new and expanding Nexus platform and NX-OS.

Far from being caught flat-footed or even found trying to protect their existing markets, Cisco have made bold moves into the Blade Server market with UCS as well coming to market with the new and ever expanding Nexus platform and NX-OS.  If anything Cisco are already heading around the first bend with early deployments of CEE and FCoE.

Its one thing to make a better or faster “switch” than Cisco.  Buts altogether another thing to build better solutions.  Right now Hypervisors are driving change and shaping the future.  Addressing the challenges that Hyperviors bring are vital.  From my perspective Im not seeing a lot from Arista to help here

Management Big Hitters

Before finishing, it is worth quickly mentioning the very impressive list of networking legends that is heading up the Arista management team.  A quick scan of the management team listed on the Arista website reads like a who’s who of Data Center networking and is nicely capped off with Andy Bechtolsheim who has pedigree when it comes to picking winners.  These guys know their networking, oh ……. and they know the competition   So don’t write them off just because of what I say :-S

Finally

With the lack of a networking group at IBM, it would be remiss of me not to suggest that Arista may be a future acquisition target for Big Blue   I had to say it.

I'd love to have been more impressed with Arista.  Who knows, may be they have a game changer up their sleeve.

Nigel

You can follow me on Twitter where I talk about storage technologies (@nigelpoulton) and I am also available for hire as a consultant.

In my opinion the show was a real success with over 1,500 attendees, of which over 1,100 were end users and reseller delegates and the remainder made up of general riff-raff such as vendors, press and the likes….

One of the things I like to see at shows like these is hardware.  What can I say, Im just the kind of guy that gets a kick out of seeing hardware.  So for the rest of you out there like me – sit back and enjoy……..

First up, I was impressed to see a Symmetrix V-Max, blue strip light ablaze.  The only disappointment was that whenever I popped by to have a chat, somebody else was always being given an overview

Next up the IBM HS22 BladeCenter that was used to demo the new IBM Virtual Fabric, which is based on technology from IBM, BLADE Networks and Emulex.  A much needed addition to the IBM portfolio in my opinion.

Oh and while on the theme of IBM, here is another rack of IBM kit

As the IBM Virtual Fabric solution has an Emulex CNA in it, next up is Shawn Walsh from Emulex showing us that the Emulex UCNA is real and not a myth.

And a close up on a desk with a pen for lined up to give an idea of size

Then if you followed the girls with “Thin” written on their T-shirts you couldn’t miss the 3PAR InServe kit on show.  I plan on writing about 3PAR RAID MP and Persistent Cache, both of which are potentially very interesting technologies.  But seeing as 3PAR are attending the upcoming GestaltIT Field Day I might wait and see if I can glean some deep tech info from them.

Brocade also turned up with a rack load of kit, although hugely disappointing for me was the lack of an FCoE 10-24 blade in the DCX director.  Not to worry though, there was a B8000 top of the rack CEE/FCoE switch to keep me happy.

And a Brocade dual port CNA

I have some video footage from the Brocade booth that I will post some time next week.  So stay tuned.

Even the internet connected laptops were also of decent spec.  Below is a smart little HP laptop alongside my personal 11.1” Sony job – check out the grease marks on my trackpad and spacebar

And last but not least, the quality of freebies was good.  Aside from the standard pens and stress balls, I was particularly impressed with –

iPod Shuffle
Solio solar powered USB charger
3-in-1 pen/laser pen/1GB memory stick (James Bond style)
2GB micro SD with standard SD adapter 

The economy must be recovering!

As well as the above mentioned video footage from the Brocade booth, I also got some video footage from the Arista networks booth.  Keep an eye out for that as I plan to post them in the following week…

Nigel

You can follow me on Twitter where I talk about storage technologies (@nigelpoulton) and I am also available for hire as a consultant.

In the video, William Lloyd Scull Senior Network Architect at BLADE Network Technologies, demos a feature of the IBM BladeCenter Virtual Fabric (comprised of Emulex OneConnect UCNA, BLADE Network Technologies Blade Switches and IBM BladeCenter hardware).

In the video William dynamically reduces network bandwidth allocated to a vSwitch from 2500Mbps down to 1900Mbps, as well as talks us through some of the other features and some of the hardware involved in the Virtual Fabric solution.

Its an interesting video if you want quick and dirty overview of Virtual Fabric for IBM BladeCenter and a glimpse at what it looks like with the lid off.  The video was not rehearsed, and although Id gone through the same thing with William the day before he had no idea I would turn up again the next day with my video camera and ask him to perform to the world

So thanks to William for being a good sport.  Enjoy……..

First impressions for me are similar to my thoughts on HP Virtual Connect Flex-10…….  In fact this is very similar to VC Flex-10 but with a more clear roadmap for FCoE and is built on a solid Converged Network Adapter (OneConnect UCNA) from Emulex.  However, as good as it is, its very generation 1.  But that’s not a huge issue because these are early days in this area.  Everybody else is in a similar position.  To my knowledge there is nobody doing hairpin turns in silicon yet and we are all a long way off MR-IOV and the new rack area landscape which that makes possible……..

So…. an important technology for IBM, may be slightly better than Virtual Connect Flex-10 but is probably not quite as good as some of the stuff Cisco is doing with Palo… But a solid foundation to build on (see my previous post on the Emulex OneConnect Universal Converged Network Adapter for more info).

Thoughts and comments welcome as usual…..

Nigel

Oh and you can follow me on Twitter where I talk about storage technologies (@nigelpoulton) and I am also available for hire as a consultant.

Anyway, the interesting thing from my point of view is that it was just last week that I cheekily said to somebody at HP, via email, that I thought HP should buy Brocade.  So with the current rumour in mind I might as well take a minute to explain why I think this would be a great move, not only for HP and Brocade, but also the wider industry……

First up, there is no doubt that there is a huge push towards convergence in the industry.  One topic Im thinking a lot about at the moment is the converged network (aka unified fabric) bringing together the likes of IP, iSCSI, FC and more, all on to the same cable, PCI adapter, switch ports…..  In this area Cisco are flexing their muscle and driving much of both the changes as well as leading the standards bodies.  Brocade and others are at the table, especially in the designing of the standards etc, but struggle to compete with the might of Cisco when it comes to getting products to market and gaining market share.

Being Brutally Honest.

Question 1: Is it realistic to expect Brocade to compete with Cisco in the unified fabric arena?

My thoughts are heavily influenced by the fact that I have never seen a Brocade Foundry switch (IP) out in the wild.  Now if Ive never seen one in my professional life, and Ive seen hundreds and hundreds of Cisco switches, I have to wonder whether they can ever compete.

Sure Brocade have a great FC capabilities and are very strong in that area of the market.  But they are relatively poor in the IP arena. 

Honest answer – Probably not good enough to compete with Cisco.

Question 2: Is it realistic to expect HP to compete with Cisco, based solely on their existing portfolio in ProCurve?

While I HAVE seen HP ProCurve switches in the wild, I have not seen many. 

Based on the way things have gone and are going in IP networking, and the fact that HP ProCurve does not offer FC, FCoE or DCB then they will probably struggle to compete with Cisco.

Interestingly though, on the unified fabric strengths and weaknesses comparison chart, HP  ProCurve is almost the exact opposite of Brocade – it has relatively good Ethernet/IP capabilities but not FC or FCoE.  When you compare them, its almost as if they are screaming out to be married up.

Introducing the HP ProCade platform

Bring the two together though – the unquestioned market leading capabilities of Brocade in FC and FCoE, and the number 2 player in the IP networking space, merge the technologies together and suddenly a potential challenger to Cisco materialises. 

Im calling it HP ProCade!

May be the HP ProCade Unified Network System (HP PUNS)!  OK may be not, but ProCade definitely has potential.
 

NOTE: Based on existing market share, and the fact that it would be HP doing the buying (I must re-iterate that this is 110% pure speculation) and Brocade doing the dancing, I would expect ProCurve to win over Foundry.

Granted takeovers and technology mergers take time (dont even ask about the Oracle SUN thing which is seeing the SUN side of the bargain destroyed while the European competition people drag their heels) and of course it would not be easy – but then again not many things in life really worth while are easy.  But I think for the industry this would be extremely worthwhile.  Competition is is what drives the industry forward.

Sadly, aside from this potential, I dont see any other serious challenge to Cisco.

Interestingly though, Im willing to bet that Cisco would be worried if HP were to buy Brocade.  Some of the guys on Twitter seem to be coming across as if they feel HP purchasing Brocade would pose a threat to Cisco.  At least far more likely to threaten than anything else on the table at the moment.

In summary, I would LOVE to see HP and Brocade come together and I think it would really give Cisco something to think about.  And for any Brocaders out there reading this, I think HP StorageWorks is in good hands, and I dont think being taken over by HP would be the worst thing in the world.  Im sure we'd all love Brocade to stay independant, but in order to survive and live on and challenge, may be a change is needed!

Thoughts and comments encouraged.

Nigel

Now then, there is no easy way to put this, but to do justice to a topic like this will require a lot of words.  I’ll do my best to keep it succinct, but if you are looking for a high level overview in under 1,000 words then this is probably for you, but thanks for stopping by………  However, if you want a deep dive and opportunity for technical discussion, then this might be what you’re looking for.

Still here? Magic, let’s go…..

First things first, its best to have a quick look at what things look like without CNAs, in order to more fully appreciate some of the problems CNAs are resolving.

 
Before CNAs

In a traditional server with no CNA cards and not connected to a unified fabric it is not uncommon to see the following “network” related sprawl (sorry about the terrible diagram but I wrote most of this post while on an aeroplane) –
 

Diagram 1

 
The FC HBAs tend to run at either 2, 4 or 8Gbps with the NICs usually running at either 100Mbps or 1Gbps.

The multiplicity of NICs is common to meet the demands of multiple traffic types such as; production, backup, management etc.

It is also worth noting that most servers are built with two physical HBA cards installed to provide both redundancy and increased aggregate bandwidth/performance – commonly referred to as I/O multi-pathing and is a must in 99.99% of FC SANs.

But that was then and this is now….. or nearly now

Say hello to the Converged Network Adapter, or CNA for short.

The Converged Network Adapter is exactly what it says it is – an HBA and a NIC converged on to a single PCIe adapter –
 

Diagram 2 – Picture courtesy of Emulex

Digging into the technical detail, some CNAs have a single ASIC that performs both the HBA and the NIC functionality, whereas other have a separate ASIC for each of the distinct functions.  This really depends on the vendor and model.  The important point being – CNAs provide HBA and NIC functionality in hardware, making them fast and reducing CPU overhead.  Implementing functionality without thieving CPU cycles is a big plus-point in server virtualisation environments.

Offload EVERYTHING!!

Considering the fact that stealing CPU cycles is undesirable at the best of times and a federal offence punishable by death in virtual server environments, providing hardware offloads is vital.  To help out in this area, the latest raft of Generation 2 Emulex OneConnect Universal Converged Network Adapters, such as the LP2100x, provide full CPU offload for all protocols on a single chip design and are FIP compliant!  That’s not just FCoE offload, we’re talking about TOE and iSCSI as well – the iSCSI one is interesting and may be a chat for another day!
 

NOTE: I will point out that I had a chat recently with Emulex and was very impressed.  They are laser focussed (pun intended) on FCoE and really understand the market and value position of FCoE.  A real pleasure to chat with passionate like minded people, oh and they didn't hang up on me when I mentioned "Broadcom"   VERY unprofessional of me but I was unbelievably jetlagged when we spoke, so I thank them for overlooking my lack of tact and poor humour.

So with the fact that CNAs provide both HBA and NIC functionality, as well as connecting to 10Gbps Enhanced Ethernet fabrics, it shouldn’t take a rocket scientist to figure out that we could swap out our 6 PCI adapters from Diagram 1 and replace them with just 2 CNAs (2 for redundancy).  If you think about it, this has the potential to reduce PCI adapter count and cable sprawl by crazy%

Saving Space

Early generation CNAs are installed as PCIe adapters in servers and as PCIe mezzanine cards in blade servers.  However, the way forward is to eventually have them embedded on the motherboard – a la’ LOM – vendors currently have this on their roadmap.  But this is no biggy right?  Well actually this has the potential to hugely reduce the amount of space consumed by PCI adapters inside of servers paving the way for higher density in blade servers where real-estate is at a premium!

Now this brings up another interesting benefit.  Not only is FCoE and its associated technologies resolving todays problems, it is actually enabling a better future.  How good would it be to have 6 or more CNAs per blade server!  That’s 60Gbps+ of flexible bandwidth based on today’s current 10Gbps Enhanced Ethernet!

Now this becomes a real winner in light of the current raft virtualisation optimised CPUs on the market – think Intel Nehalem and associated  virtualisation technologies like Intel VT-x VT-c, VT-d VMDq – xpect the ratio of VMs per core to rocket skyward!  In order to keep pace with the advancements in chip technology the I/O subsystem needs to move in step.  CNAs and Lossless Enhanced Ethernet are vital to this.

This point aside, I recently had a hand in a design involving HP BladeSystem c7000 technology.  I remember during the design wishing that the system had CNAs instead of the Flex10 and Virtual Connect technologies.  Could have saved space and cabling as well as potentially given more flexibility.  However, this was prior to even FCoE being ratified by INCITS.

Obvious benefits summarised

So some obvious benefits that CNAs bring include – reduced number of PCI adapters, cables, space, power and cooling.  Nice!

Secondly, by supporting 10Gbps Enhanced Ethernet they provide greater bandwidth than existing adapters allowing more traffic types and volume to travel down the same stretch of cable.  Suddenly the wire once nirvana is now a reality (actually wire twice if you want true I/O multi-pathing).
 

NOTE: A quick heads up on throughput.  While CNAs provide access to 10Gbps CEE networks, not all services, including FCoE, are supported at full bandwidth.  For example, CNAs normally support Ethernet at 10Gbps but FCoE only runs at a max of either 4Gbps or 8Gbps.  This is in line with 4Gbps and 8Gbps FC – there is currently no standard for 10Gbps FC N-Port to F_Port connections.

<OK so this is about the half way point, so you might want to come up for some air here before discussing the interesting stuff>

Clever things with CNAs

Now to the deeper technical stuff………..

With 10Gbps Enhanced Ethernet, FCoE, CNAs and other associated technologies being relatively new, the standards bodies (such as INCITS T11 FC-BB-5 for FCoE) as well as the vendors are able to design with virtualisation in mind.  This is great!

Let’s mention a couple of technologies that bring a lot to the table in Hypervisor environments.

NPIV

The first technology worth mentioning is N_Port ID Virtualisation, otherwise known as NPIV.  NPIV is a T11 INCITS and ANSI standard initially developed by IBM and Emulex, and yes I know it’s not exactly new.

Because NPIV is not a new technology I wont spend much time on it here other than to say NPIV makes it possible for a single HBA to have several N_Port IDs and WWPNs, allowing virtual machines to be uniquely addressable on the SAN and therefore able to be managed on the SAN just like normal physical servers.
 

If it would be useful I can put something up on NPIV.  Just leave a comment at the bottom asking and if enough people ask I’ll post on it.

I/O Virtualisation (SRIOV)

So while NPIV allows multiple VMs to be uniquely addressable on the SAN side of an HBA, on the other side, the side of the servers PCI tree, there is still a 1:1 mapping between physical ports and addressable PCI devices.

The Middle-man

Early virtual server implementations see the hypervisor act as middleman, between the I/O adapter and the Virtual Machine (VM).  The VM never actually talks directly with the I/O adapter, always through a middle-man – the hypervisor.  But like any middleman scenario, it has its advantages and disadvantages.  The middleman would argue that he adds “value”, but is rarely so keen to point out that he also always takes a generous percentage of the spoils (in our case, I/O performance and CPU cycles).

Some of the advantages and disadvantages of having the hypervisor as the middleman might include –

Middle-man Advantages

• Hypervisors often provide snapshot capabilities
• Hypervisors often provide thin provisioning capabilities
• Many existing Hypervisor technologies are currently designed around this model

Middle-man Disadvantages

• Lower I/O performance.  Because the hypervisor handles all I/O to and from a VM, as well as the associated interrupt handling etc, this injects server-side latency in to the I/O path as well as stealing CPU cycles form the main role of the Hypervisor.
• Security concerns.  The fact that all I/O, to and from VMs, is seen and touched by the hypervisor may be a concern to some people :-S
• Limited feature set.  You are restricted to the features and functions provided by the Hypervisor and not exposed to the full feature set available from the manufacturers native driver….

A common example where having the Hypervisor act as the middle man is not seen as desirable is a high throughput OLTP type system.  These are rarely virtualised due to, among other things, the I/O performance impact associated with having the Hypervisor as the middle-man.

So what does SRIOV do to help?

NOTE: Single Root I/O Virtualisation (SRIOV) and Multi Root I/O Virtualisation (MRIOV) are extensions to PCIe brought to us courtesy of the electronics industry consortium known as the PCI Special Interests Group (PCI-SIG).

SRIOV implementations allow an I/O adapter, in co-operation with the Hypervisor, to be sliced up in to multiple virtual adapters.  Each of these uniquely addressable on the servers’ PCI tree.  Each virtual adapter can then be mapped directly to a virtual machine and in turn is directly addressable by that virtual machine – no more middle-man.  SRIOV based adapters may have dedicated I/O paths in silicon and work alongside other related technologies such as Intel VT-c, VT-d, VMDq…. to provide a huge variety of offloads and assists aimed at reducing the load on main CPU and memory systems and thus increasing system performance.  This mode of operation is sometimes referred to as “hypervisor bypass” mode (we should be thinking VMDirectPath by now) and offers close to full line rate, making virtualisation a more realistic option for transaction based and other I/O intensive systems like our example OLTP system.

This also opens the door to things like VMs running drivers provided by the manufacturers making new functionality immediately available without waiting for the Hypervisor middle-man to support it.  Obviously it all needs buy in from your Hypervisor …..

Of course SRIOV is a semi-open standard (I say semi-open because several months ago when I researched it you had to pay to get access to the standards docs) and like most standards each vendor is free to implement the specifics in their own unique and value-add way as well as to add more features and requirements around it.

So in a nutshell, NPIV enables a single port to have multiple discrete addresses on the SAN side, whereas SRIOV allows a single I/O adapter to have multiple discrete addresses internally on the servers PCI bus/tree.  The diagram below from a long time ago on the Cisco website shows an I/O card that can be partitioned in to 128 (0-127) virtual interfaces.
 

Diagram from the Cisco website

Say goodbye to hardware rip and replace

By implementing the above mentioned technologies it becomes possible to run a single cable to a server and then use management software to configure the virtual interfaces according to current requirements.  For example, removing a NIC function and adding an HBA function becomes just a software change – no requirement to physically swap out a PCI card or lift the floor and run new cables.  Simply make the change in software and the new device will be presented to the servers PCI tree, job done!  Sound good to anyone else?

There is more, but I doubt anyone would read more than this in one post.  Other topics can always be discussed in the comments section below….

So if you made it all the way to hear, thanks and I hope it was useful.  Please feel free to join the discussion either here on the blog site via the comments section below, or by following me on Twitter @nigelpoulton.  I only talk about storage and related technologies.

Nigel

PS.  I am an independent consultant and available for hire via the Contact Me page