How to Reset Change Block Tracking (CBT) for VMware Backups

SUMMARY

How to Reset Change Block Tracking (CBT) for VMware Backups

ISSUE

During the Backup of a VMware Virtual Machine (VM), the VMware APIs may have problems getting Change Block Tracking (CBT) information. Due to this, the entire VMDK(s) will be backed up, resulting in larger Incremental/Differential Backup sizes.

RESOLUTION

Warning: Before applying this procedure, the VM must be free from snapshots. For more information, see:

• Consolidating snapshots in vSphere 5.x/6.0 (2003638)
• Determining if there are leftover delta files or snapshots that VMware vSphere or Infrastructure Client cannot detect (1005049)

The following is a summary of information contained in VMWare's KB and will typically resolve most CBT issues.  Additional steps and alternate methods are presented in VMware KB 2139574 titled Resetting Changed Block Tracking for VMware vSphere virtual machines. 

Before following these steps, confirm that the VM hardware is version 7 or higher. If it is not hardware version 7 or higher,  these steps will not work,

The following steps should be followed in order to reset CBT information on the Virtual Machine. All of these steps should be followed on the vSphere Client:

1. Shut down the VM.  Do not proceed until the VM achieves a "powered down" state.
2. Right click the VM, click Edit Settings, then click on Options tab.
3. Under Advanced” settings, click General, then click the Configuration Parameters button.
4. In the “Configuration Parameters” dialogue box, locate the following two fields and set them to “false”.  NOTE: there will be a scsi entry for EACH disk on the VM.  
   - ctkEnabled = false
   - scsi0:0.ctkEnabled = false (do the same for any other scsi#:#.ctkEnabled)​
5. Go to the Datastore where the VM files are located. Remove any “*.ctk.vmdk” files. Remember a single VM may have disks in several different datastores, be sure to access each and remove the ctk files in all locations necessary.  
6. Power on the VM and login.
7. Shut down the VM a SECOND time.  Do not proceed until the VM achieves a "powered down" state.
8. Power on the VM and login.
9. Power cycle the VM a THIRD time. 2 full reboot cycles are required to properly reset CBT after powering on the VM in step 6.   NOTE: This step is needed to update the CTK Table for the VM.
10. Check the VM and confirm normal operation.  
11. Run a new 1-Time Full/Master backup for this VM.
12. Return to the configuration parameters and revert the CTK items altered in step 4 to TRUE.

From this point on, the Incrementals will work as expected receiving data from VMware based on VMware's CBT.  If your VMWare ESXi version is not current 

CAUSE

When Storage vMotion is used, or in some cases due to vmware software defects, the CBT information for a VM might get reset and hence, the Unitrends Software is unaware of the blocks that have changed since the last backup, which would trigger backups of the complete disks of that VM or may produce backup errors reporting CBT must be reset for a VM.  This is a vmware issue that impacts ALL backup vendors that use VMWare's storage APIs and vprotect backup engines and is not unique to Unitrends.  

This issue can be caused by VMWare CBT issues in multiple versions of their software, and can also be caused by unplanned ESXi or storage outages on running VMs, or in some cases by consolidation defects or failures.  Change Block Tracking is a critical VMWare component that makes incremental and differential backups possible and all backup vendors directly depend on it's stability.  VMWare has released numerous fixes for this system to improve it;s stability and solve critical defects.  Unitrends recommends customers always be on the latest VMWare Patch Release.  At the time of recent updates to this article, that is:
 

Note, the initial VMWare 6.7 build release ESXi 6.7 GA  build # 8169922 is known UNSAFE.  

Initial GA releases of VMWare 6.0 and 6.5 are known to have critical CBT defects that can lead to unrestorable machines and even production VM corruption.  The same is also true for 5.5 releases prior to 5.5 U2.  Customers running on GA releases or any release with a published CBT defect are recommended to disable VM backups and use agent-based protection until ESXi issues can be resolved by patching to later editions, and/or, to enable Unitrends CDM test automation technology present in Enterprise plus editions of unitrends licensing to validate backup integrity.