SUMMARY
You verified that the VM has ctkEnabled = true and scsi0:0.ctkEnabled = true. Yet, you continue to get the error: Change Block Tracking must be enabled
ISSUE
You verified that the VM has ctkEnabled = true and scsi0:0.ctkEnabled = false. Yet, you continue to get the error: Change Block Tracking must be enabled
RESOLUTION
To resolve this issue perform one or more of the following processes.
- Though the VMWare UI may report "you are here" with no snapshots present on the effected VM, inspection of the datastore itself may reveal orphaned snapshot files are in fact present. Snapshot files will have the same filename as the parent VMDK but contain *-00000#.vmdk. If you have orphaned snapshots that the VMWare UI does not reference, you will also likely note you are not on the latest ESXi release for your hosts. Consolidation failures and CBT issues are a common issue with releases of VMWare prior to 5.5 U3b P10, 6.0U3f EP15, 6.5 U2C EP10, and 6.7 U1. Unitrends strongly recommends customers be on the latest vmware patch or express patch release as quickly as possible to avoid various CBT and consolidation issues that could lead backups that are unrecoverable to appear otherwise successful. You can check here for current VMWare ESXi build numbers. This condition is not caused by backup vendors but is a latent VMWare defect present in numerous ESXi editions.
- To remove stale snapshots, use the UI option to "consolidate" which is typically successful even when no snapshots appear present. Check the datastore again to ensure orphans have been removed.
- If standard consolidation is unsuccessful, often the creation of a new snapshot followed by consolidation will otherwise work.
- If you still are unable to remove the snapshot orphans, see VMWare's documentation for manual offline consolidation. In some rare cases vmware support itself may be required to assist. VMWare may recommend using storage vmotion to migrate the VM to alternate storage to resolve this condition, and may in some cases also recommend cloning the VM.
- Additional VMWare Links are provided in How to Reset Change Block Tracking (CBT) for VMware Backups
- Ensure you are not directly protecting a vCenter Virtual Appliance using a backup schedule. vCetner's database and configurations must be protected using VMWare's own processes independent of 3rd party backup products. Please see VMware's documentation for your vCenter version for how to do this in 6.0 or prior releases. For 6.5 releases the backup option is exposed in the VMWare UI. Unitrends and VMWare do support vcetner VM direct backup for vCenter 6.0 and higher (not at all for prior releases), but it must be noted a vcetner backup must only perform full backups, and must occur at a time when no other VM backups are in process, and no vmotion, storage motion, or other pending vCenter tasks are in progress. It is only safe to perform such a backup manually, and never on a schedule. Any snapshot or stun event that impacts a vCetner virtual appliance directly may cause other vcetner orchestrated operations, including CBT tracking or snapshot consolidation or VM migration processes to fail.
- Follow the instructions in Unitrends KB 3170 titled "How to Reset Change Block Tracking (CBT) for VMware Backups"
- Then, perform a 1-Time Full/Master backup on the VM (see KB 5580).
CAUSE
VMware tracks changes using Change Block Tracking (CBT). In order to initially begin or reset the tracking process, the VM must be free from snapshots. CBT is not aware of the changes between the original VMDK and the Deltas. The snapshots prevent CBT from creating a baseline to track changes against. Numerous VMWare issues may cause this system to fail to successfully track change, or, may lead to orphaned snapshots.