Set VMware vMotion into Fast Motion over High-Speed Interconnect

Cloud Computing, Virtualization, , , ,

Virtual Machine (VM) live migration is an important feature to enhance the high availability of applications, guarantee quality of service, and simplify infrastructure maintenance operations. Specifically, it can help to:


  • Automatically optimize VMs within resource pools to ensure that VM resource requirements are met, minimizing unexpected downtime.
  • Perform hardware maintenance without scheduling downtime or disrupting business operations, simplifying operations and reducing possibilities of hardware failure.
  • Move VMs away from failing or underperforming servers, guaranteeing application high availability.


As a result, it is almost a must-have feature for cloud infrastructure.


In VMware environment, the feature that supports VM live migration is vMotion, and it allows you to move an entire running VM from one physical server to another, without downtime, well, at least that is the goal. Because vMotion involves the transfer of the VM’s active memory and precise execution state across the network, the speed of vMotion network connecting these two hosts has a great impact on vMotion execution, and in some scenarios, even deciding whether vMotion can succeed or not.


In this blog, we compare the difference of performing vMotion over 40Gb/s network vs. 10Gb/s network. Our benchmark shows that:

  • 40Gb/s network can greatly accelerate vMotion process, cutting the vMotion execution time by 80-90% as compared to 10Gb/s network.
  • For VMs that are very active and performing frequent read and write operations, vMotion converges very slowly over 10Gb/s network, but it can succeed over 40Gb/s network with minimal impact on VM read/write operations.


We believe that the new generation of 25/50/100G interconnect networks will accelerate live migration in a similar fashion as 40G networks over 10G networks.


To understand the importance of a high-speed vMotion network, let’s first examine how vMotion works and the steps that vMotion takes to complete a live migration.


Chloe Ma 021116 Fig1

Figure 1 vMotion VM Live Migration Process


Out of the steps, Pre-copy, Iteration and Switchover all involves copying data from source server to target server and the network performance is critical in deciding whether iteration will ever converge, or switchover will fail or succeed.


In an ideal environment, the interation process converge in the following fashion (I borrowed the following table from an Opvisor blog):


Pre-Copy Iteration Main memory to be transferred Time needed for the transfer Change in memory during the transfer
1 2.048 MB 16 seconds 512 MB
2 512 MB 4 seconds 128 MB
3 128 MB 1 second 32 MB
4 32 MB 0,25 seconds 8 MB
5 8 MB vMotion cutoff, since the residual transmission takes only ~0.06 seconds


But if the application is dirtying the memory faster than the network ability to move the changes from source to destination server, this process may not converge.


In our benchmarking experiment, we ran ESXi6.0 over all-flash VSAN array. We looked at VMs with 128GB and 256GB memory configurations respectively. In these VMs, we run IOMeter to simulate applications with frequent read/write operations. We use the conventional network with 10Gb/s throughput as reference to examine the effect that Mellanox CloudX 40Gb/s high-speed interconnect can bring.


Chloe Ma 021116 Fig2

Figure 2 vMotion over High-speed Network Benchmark Configuration


For these two VMs, when we performed vMotion over 40Gb/s network compared to 10Gb/s network, the completion time was reduced by 88% and 82% respectively.


Chloe Ma 021116 Fig3

Figure 3 vMotion Performance Comparison between 40G and 10G Networks


Not surprisingly, when we monitor the bandwidth consumption during the vMotion process, we are seeing much higher bandwidth in the 40G vMotion network. Here we use the VM with 256GB memory to illustrate the network bandwidth consumption during vMotion precopy+interation phase, and during switchover phase.


Figure 4 Network Bandwidth Consumption During vMotion

Figure 4 Network Bandwidth Consumption During vMotion


As mentioned previously, in most cases, each pre-copy iteration should take less time to complete than the previous iteration, but if we get a very active VM that modifies memory contents faster than it can be transferred, the iterative copy might never converge. In that case, vMotion performs an action called Stun During Page-Send (SDPS) to ensure the memory modification rate is slower than the memory change-set transfer rate.


Upon activation, SDPS injects microsecond delays into the virtual machine execution and throttles its page dirty rate to a preferred rate, guaranteeing pre-copy convergence. The downside of SDPS is that it does run the risk of halting VM for an extended period of time and may cause read/write operations to fail. In our case, we run the IOMeter application in the VM, and we observed that during the SPDS phase of the vMotion, the I/O operations were halted, and resumed later. We ran OLTP workload with a VM with 256GB memory, and the duration of I/O halt is much longer on 10G vMotion networks compared to 40G vMotion networks.


Chloe Ma 021116 Fig5

Figure 5 Comparison of IOMeter Halt Time During vMotion Stun During Page-Send


As a conclusion, high-speed networks can really set vMotion into fast motion by transferring VM state data much more quickly from source server to destination server. In some special cases where we got active VMs with large memory, high-speed network is the only way to make vMotion converge and cause minimal downtime to applications.

Comments are closed.