I was disapointed last week to see how cheap and over-hyped Cisco's "Validated Architecture for Long Distance VMotion" was. When I saw the report on NW I thought this would be a good reference on the real issue inter-data center VMotion: dealing with crossing Layer-3 boundaries.
But, to my chagrin, it turned out to be a total marketing effort. It took Cisco a 17-page whitepaper, blog entry, video, and presentation at VMworld to essentially tell us to do this for inter-data center VMotion:
WOW! Thanks! I would've never thought of that. And Cisco tested it at up to whole 200 km. Great! I was really worried about running VMotion over a Gigabit link with 4-5 ms of delay. Shew, glad we got that covered. This architecture is flawed in many ways.
One of the biggest problems is it doesn't cover Layer-3 impacts of this VLAN WAN extension (there's one paragraph about this problem with "active-active HSRP" as the fix...huh?). Let's say you do trunk all the VLANs to your other data center, but the subnet itself still has to be advertised via routing protocols to the rest of the network. This advertisement will bring traffic toward the source of the route advertisement. But what happens if the original data center is still advertising the subnet, but the destination VM has VMotion'd to the other data center? Well, then the user traffic still has to go to the old data center and ride across your new, fat WAN link to the VM. What benefit does this add? What if DC1 is going down? Are you going to make OSPF/BGP route advertisements manually in DC2 to keep the subnet advertised?
You could advertise the subnet out of both data centers to begin with since you have a L2-trunk between them, but how do you ensure traffic will enter the right data center? User traffic for VM1, which is still in DC1, could now enter DC2 and then have to flow across the fat WAN link. This is suboptimal routing and will affect user performance. The only way to fix this is to leak /32 routes into your global routing table, but that gets messy...FAST.
And the worst part of this marketing campaign is some senior IT manager could get a hold of it, not realize how silly this design is, and start asking when it can be implemented. This design doesn't solve real, long-term problems with inter-data center VMotion, but senior managers may want to invest in it now, wasting money on a solution that solves a short-term tactical problem without long-term strategic benefits. Cisco, because of their market size and clout, has a responsibility not to put out "reference architectures" that are nothing more than the obvious designs network engineers would probably shy away from.
What I really want to see from Cisco is a solution to the biggest problem on inter-data center VMotion: dealing with crossing Layer-3 boundaries. When a VM is VMotion'd to another data center, it's almost a certainty this other data center is going to have different IP subnets. Thus, to work properly, the IP address of the VM needs to change. This IP address change must be coordinated with all other parts of the infrastructure environment such as DNS, load balancing, authentication, and management platforms. That's tough, but it is the real problem limiting VMotion. Cisco's whitepaper does mention this problem, but punts the problem down the road:
Deploying VMware VMotion across data centers that are dispersed over very long distances (500 miles or more) potentially involves moving the virtual machine to an entirely new subnet, but the goal continues to be to help ensure that the IP address of the virtual machine as well as the existing client connections are not disrupted. This type of VMware VMotion migration is not possible with existing technologies. Special hardware and software features will be required to route the TCP connections to the virtual machine in its new location without terminating the sessions. This approach will require the redesign of the IP network between the data centers involving the Internet. Technologies are being developed by Cisco, VMware, and standards organizations to address this network scenario in the future.
Cisco should've waited to deploy a "reference architecture" until this problem is solved. A combination of tunneling, ACE loadbalancers, DNS updates, and NAT'ing will probably be needed. Or maybe something cool with a little internal MPLS/VPLS. F5 is tackling this issue with similar ideas. It doesn't appear perfect, but it's far beyond this Cisco "reference architecture":
This is a poor solution from Cisco.
More >From the Field blog entries:
Arista's New vEOS Providing Competition for the Cisco Nexus 1000V
It's One of Those Opinionated Days Again
A Private Extranet for Cloud Computing
It's Really Only Partly Cloudy Out There
Networking in the (Thunder) Clouds
Networking in the (Storm) Clouds
Go to Cisco Subnet for more Cisco news, blogs, discussion forums, security alerts, book giveaways, and more.
Michael Morris is a communications engineering manager at a $3-billion high-tech company. His background is in enterprise WANs working with telcos and developing large-scale routing designs. He has worked on networks at government and corporate organizations, including networks at two Fortune 10 companies. In his current role, he leads a team of 10 engineers responsible for large-scale IT networking projects and architectural standards for data networks, storage area networks, IP telephony, contact centers, and security. Michael is CCIE #11733 and recently became one of the first three Cisco Certified Design Experts (CCDE) ever (#20080002). He has 11 years experience in networking and communications, including four years as a paratrooper in the U.S. Army. He has a bachelor's degree in MIS from the University at Buffalo and is working on his MBA from NC State University. In 2008, he was awarded the Network Professional Association (NPA) Professional Excellence and Innovation Award for his work on network architecture, templates and enterprise MPLS design.
The Leader in Network Knowledge
Fair Feedback
Michael:
So your feedback that for some use cases a layer 3 component is necessary is fair. In the intro to my post, I say essentially the same thing--building a comprehensive solution is a multi-phase process and this is the first phase, with more to come. Since we demo-ed PoC at CiscoLive, we have gotten a number of requests for more details, so we published our work to date. For some customers it will meet their needs, for others, it will not..yet.
So, we are diligently working on fleshing out both the storage and layer 3 aspects of the solution. From a layer 3 perspective, next steps will include some of the things you mentioned as well as a couple of new technologies.
Regards,
Omar Sultan
Cisco
blogs.cisco.com/datacenter
I Agree 100%
Great post!
This solution falls short of what I expect from a "Validated" design. Perhaps using that term was the error. With the Cisco Validated Design Guide program (http://www.cisco.com/en/US/netsol/ns741/networking_solutions_program_home.html) there is a high expectation for the term "validated".
There's a great market opportunity for the first company to create a comprehensive VMotion solution. I'd rather see Cisco concentrate its efforts on producing that architecture, rather than an L2 interconnect hack. In five years we'll all look back and wonder why we ever went down that road.
One point on the eventual solution.. I wouldn't rule out /32 host routes (or IPv6 mobile IP) as a solution for smaller apps, where load-balancing and multiple server instances are not required.
Jeremy
Why Validated
Jeremy:
The "Validated" means that the design has been tested and will be supported if you call either company for support while using that particular configuration. That being said, since we already have CVDs, perhaps a different term might have been better.
Omar Sultan
Cisco
blogs.cisco.com/datacenter
Replace it with a Juniper
This goes to Juniper's vision for the Data Center. I believe spanning IP segments across multiple data centers is going to be the future for many reasons including VMotion between data centers. MPLS will soon break the boundary of a "carrier only" technology.
Virtualization requires large L2 domains
Vendors such as Arista are introducing Multichassis Link Aggregation Groups that allow large layer 2 domains across multiple 10GE switches to support VM sprawl. Cisco calls it VSS on Cat 6K and VPC on Nexus.
VMotion breaks MPLS and Layer 3 and the world does not need another proprietary tag!!
MC LACP does not help with VM spawl
Arista and Nexus are both limted by 16K MAC addresses...this is the true limitation for mass VM deployments. MC LACP only helps provide active/active uplinks for better bandwidth utilization...it does not increase L2 scalability.
Long Distance VM and Virtual Storage Extension
AFORE demonstrated long distance data center extension, using live cloud computing service provider networks, at VMWorld last week. The demonstrations involved live migration for both server VM environments as well as FC based storage over long distances, over IP wide area networks. Shared Storage, Active-Passive, and Active-Active solutions were demonstrated, including the first 'inter Cloud' migration between cloud computing providers based in Europe and the US over a transatlantic Internet link. AFORE's Virtual Wire and Virtual Fiber technology enables lossless data center extensions over existing wide area networks without altering any of the VMWare best practices when it comes to VMotion and storage networks. They key consideration for any approach is to minimize the operational workarounds for the user.
F5 solution
The F5 solution leverages the company's application delivery controllers for VM migration. It suggests that DNS entries will change to reflect the new location of the VM for future connections. If the VM is accessed by its original IP address, then the redirection still needs to happen from the original data center as indicated in the video.
As your post says, this is not perfect but a viable option. It still leaves the larger Layer 3 infrastructure update question open.
One of the comments to this post talks about a Mobile IP like scenario, where data traffic is redirected from the old to the new location (F5 does something similar).
Question: at what point does "live" migration become so complex that you would prefer to tear down connections and restart the VM at the new location? Especially if you need to do this redirection... (can you do such a redirection if the original data center is in a disaster zone, which is the reason for the migration in the first place?).
LeitM
No Single Use Case
LeitM:
You bring up some interesting points--the challenge is there a number of different use cases with different functional requirements, depending on the nature of the VMotion.
If you are looking at low frequency use case (say a DC move, evacuation from a disaster area) it might be OK to have sub-optimal network, since it is temporary state that you are trading for system availability and you will eventually return to the previous optimal state.
One the other hand, creating a truly fluid environment with VMs moving between data centers, with cloud import/export and no "reference state", there is a bit of work to be done to both simplify the mgmt and to ensure the underlying network does not end up being undermined.
So, the good news is there are a number of efforts underway to crack this nut so stay tuned.
Omar Sultan
Cisco
blogs.cisco.com/datacenter
Layer 2 Extension
Layer 2 extension is needed to ensure that the moved VM continues to have connectivity to the other hosts in the same subnet (broadcast domain). Just moving the VM and traffic to the VM is not going to work for everyone. Just looking at solving layer 3 is not enough.
Layer 2 solutions today need to do STP isolation between DC but in future one can do it without STP involved.
For details on our layer 2 extension solutions including L2 extensions using MPLS/IP, you can refer to this white paper
http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11_493718.html
There are enough flavors of customer requirements and our goal is to solve it in a comprehensive manner including more scalable Layer 3 solutions.
Balaji Sivasubramanian
Cisco
Post new comment