According to ESG Research, most organizations host 5 to 10 VMs per physical server today. This is likely to increase moving forward as enterprise implement servers with multiple CPU cores, gigabytes of memory, network interconnects, and additional network bandwidth.
As this happens, we will also see more direct VM to VM traffic within each physical server.
At this point, industry agreement meets a huge fork in the road. Most of the virtualization technology vendors believe that VM to VM traffic will continue to flow through virtual switches. As a result, VMware is adding enterprise-class features into its vSwitch while Citrix pushes Open vSwitch (an Apache open source project) as an alternative to proprietary implementations. I'm not sure what Microsoft is doing, (I hear that its enterprise virtual switching efforts are more focused on Azure than Hyper-V) but is seems to be following VMware's lead, albeit at a slower pace. Cisco also supports virtual switching with its Nexus 1000v for the VMware platform. In this way, Cisco customers can align virtual and physical switching with common Nexus/UCS management and oversight.
While the server crowd is sold on virtual switching, networking vendors aren't so sure. Many networking experts claim that advanced switching will place too much processing overhead on vanilla Intel servers. Furthermore, why add a layer of switches in servers when you have plenty of switching horsepower (and tried-and-true network management) on the existing physical switching infrastructure already.
Seems like a good alternative but unfortunately the networking industry dissolves into a technology pissing contest at this point.
Many network providers (i.e. Avaya, Brocade, Enterasys, Extreme, HP, Juniper, etc.) advocate a technology called Virtual Edge Bridging (aka IEEE 802.1Qbg). This is also sometimes referred to as Virtual Ethernet Port Aggregator or VEPA but I'm told that this isn't entirely accurate. While many vendors support this direction, there is no ratified IEEE 802.1Qbg standard so the technology involves some proprietary work-arounds for now.
Cisco seems like the lone vendor backing a competing standard, Bridge Port Extension or IEEE 802.1Qbh. This standard also moves VM to VM switching to the physical switching infrastructure but since it requires VN-tagging, a Cisco technology, it is eschewed by most of the rest of the industry.
Okay, just so you are still with me: You can use virtual switches, but each virtualization vendor supplies its own technology and it works independently of the physical switching infrastructure (although most vendors provide methods to aggregate management and integration). You can replace VMware vSwitch with Cisco Nexus 1000v but as of now you can't replace virtual switches in other virtualization platforms. Finally, you can eliminate virtual switches entirely and shift VM to VM switching to physical switches but you have to sort through competing technology standards that seem to contain proprietary implementation "gotchas" either way.
For most enterprises with 5 VMs per physical server, this isn't a problem -- yet. In the future however as the ratio of VMs to physical servers steadily increases, this could become a major headache. We as an industry need to guide these users with reference architectures, pros and cons, performance testing, and best practices.
I hope we've sorted out this mess by then!