File Level Slow Backup Performance on Xen Guest Windows Operating System

SUMMARY

File Level Slow Backup Performance on Xen Guest Windows Operating System

ISSUE

Summary

Slow backup performance of Windows guests operating systems on Citrix XenServer hosts because of network settings.

Symptoms

A dramatic loss in network performance after backup job begins.  Back speed slows to less that 1% of initial data throughput speeds.

For more detailed information on backup failures and performance issues see Unitrends KB 5062 - Backup Failures and Performance Issues

RESOLUTION

Resolution

Disable Large Send Offload inside the Windows guest operating system.

• Click Start > Run and type in devmgmt.msc and press Enter
• Expand Network adapters, right click on Citrix PV Ethernet Adapter and select Properties
• On the Advanced tab, click on Large Send Offload, and set the drop down to Disabled then click OK. 

• Reboot the VM, and you should see a significant difference!

More Information

Network Driver Interface Specification (NDIS) 5.0 lets TCP take full advantage of intelligence in network adapters by letting the adapter do some of the tasks that the processor normally performs. Offloading these tasks to the network adapter leaves the processor free for tasks that only it can perform.

The TCP Large Send Offload option allows the AIX TCP layer to build a TCP message up to 64 KB long and send it in one call down the stack through IP and the Ethernet device driver. The adapter then re-segments the message into multiple TCP frames to transmit on the wire. The TCP packets sent on the wire are either 1500-byte frames for a Maximum Transmission Unit (MTU) of 1500 or up to 9000-byte frames for a MTU of 9000 (jumbo frames).  Re-segmenting and storing up packets to send in large frames causes latency and timeouts.

https://support.citrix.com/article/CTX117374

https://support.citrix.com/article/ctx117491

https://support.microsoft.com/kb/904946/

CAUSE

Cause

On a physical machine, the overhead associated with packet receipt verification (ie: computing checksums), is usually handed off to be calculated by the physical network card itself, effectively relieving the Operating System of these duties. In a Virtual Machine, there is no real physical Network Adapter with a hardware chipset that the Operating System can use to pass off the task of calculating the checksums.

NOTES

Additionally see Slow agent-based backups of XEN guests if this does not resolve the issue.