SUMMARY
VMware has announced a bug related to ESXi 6.0.x whereby backing up a changed block tracking enabled virtual machine in ESXi 6.0.x returns incorrect changed sectors.
ISSUE
The Issue:
Virtual machine backups utilizing Changed Block Tracking (CBT) in ESXi 6.0 may experience the following symptoms:
- The CBT API call QueryDiskChangedAreas() API call can sometimes return incorrect changed sectors, which results in inconsistent incremental virtual machine backups.
- Inconsistent virtual machine backups
This affects all backup vendors backing up ESXi 6.0.x via snapshots or incremental backups. End users may experience corrupted or inconsistent backups, causing data loss on restore. VMware has announced a bug related to ESXi 6.0.x: Backing up a Changed Block Tracking enabled virtual machine in ESXi 6.0.x returns incorrect changed sectors (2136854).
Resolution:
This issue has been resolved in VMware ESXi 6.0 patch ESXi600-201511001. See VMware ESXi 6.0, Patch Release ESXi600-201511001 (2137545) for further details. Questions relating to this patch should be directed to VMware Support (see How to file a Support Request in My VMware).
Caution:
After applying the patch, a host reboot is required. In addition, you must reset CBT on your virtual machines (see: How to Reset Change Block Tracking (CBT) for VMware Backups) and run a full backup to ensure data is consistent going forward.
Optional Work Arounds:
In the event upgrading to the patch above is not possible in the near future, VMware has suggested using one of the following workarounds.
- Downgrade the affected ESXi hosts to version 5.5, and downgrade the virtual Hardware Version from 11 to 10, if necessary. For more information, see VMware Knowledge Base articles Reverting to a previous version of ESXi (1033604) and Downgrading the virtual machine hardware version in ESX/ESXi (1028019).
- Shutdown the virtual machine before doing an incremental backup.
- Do a full virtual machine backup rather than an incremental backup.
- Additionally, customers may utilize a Unitrends agent for backups within the VMs.
Important Considerations:
- If you are using Eager Zeroed thick virtual disks, just a full virtual machine backup is sufficient to resume incremental virtual machine backups.
- If you are using other types of virtual disks, a full reset of CBT data is required, see VMware KB 2139574, Resetting Changed Block Tracking for VMware vSphere virtual machines.
Additional Resources:
Unitrends has created a script which may be used to reset CBT, enable/disable CBT, and get current status across multiple VMs. See VMware CBT Reset PowerShell Cmdlet for instructions and to download the script.