SUMMARY
Failed while reading data for disk [LUN] /disk-0000xx.vmdk
ISSUE
Error in ui reports vmware backups are failing with an error similar to:
VMWare backups Failed while reading data for disk [LUN] <server>/disk-0000xx.vmdk
For more detailed information on backup failures and performance issues see Unitrends KB 5062 - Backup Failures and Performance Issues
RESOLUTION
**Update** - Customers should engage VMware if they would like confirmation from them this is a VMware issue. VMware will likely ask to migrate off of the affected host and reboot the host. VMware should also advise to update to the latest build for their VMware version.
Try to restart services on the ESX host where the VM resides: services.sh restart
-This may resolve the issue and allow the backups to proceed
When pre-existing snapshots exist for a VM, they have to be cleaned up.
The easy way that often (though not always) works, is to create a snapshot for the VM and then consolidate. Manually check the datastore after consolidation to confirm the snapshot files are removed. Perform a new full backup of the VM, and if the issue returns, contact VMware support directly, not Unitrends support, to diagnose snapshot issues with the VM host.
If UI consolidation does not remove the files, manual consolidation is required. VMware has a knowledge article on this:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002310
A specific article for 5.5 can be found at:
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2003638
If manual consolidation does not work, please contact VMware for assistance. Unitrends Support does not have the capability to review VMware's logs related to snapshot consolidation failures.
CAUSE
Possible Cause 1) known defects in VMWare base and early versions may fail to properly consolidate snapshots. Please ensure you are on the latest release of ESXi to avoid this issue. All VMWare editions supported with build numbers lower than update 1 exhibit this behavior. Many will exhibit this on lower than update 3a/3b). Once snapshots are consolidated safely, ensure all ESXi hosts are updated to at least the most recent ISO edition if not latest vmware patch release.
Possible Cause 2) a VMware memory-related condition.
2017-03-31 12:23:09.860 -0400 : I vix_warning: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 4194328 bytes
2017-03-31 12:23:09.861 -0400 : D vix_debug: DISKLIB-LIB : RWv failed ioId: #2 (802) (34) .
2017-03-31 12:23:09.861 -0400 : D vix_debug: VixDiskLib: Detected DiskLib error 802 (NBD_ERR_INSUFFICIENT_RESOURCES).
2017-03-31 12:23:09.861 -0400 : D vix_debug: VixDiskLib: VixDiskLib_Read: Read 8192 sectors at 14336 failed. Error 2 (Memory allocation failed. Out of memory.) (DiskLib error 802: NBD_ERR_INSUFFICIENT_RESOURCES) at 5178.
2017-03-31 12:23:09.861 -0400 : D vpt_backup_get_chunk:471: Breaking out of the retry logic when change tracking is off.
2017-03-31 12:23:09.861 -0400 : E VixDiskLib_Read error: Memory allocation failed. Out of memory.
The [LUN] will be the name of the VMware datastore trying to be accessed.
Within the LUN is a guest-name folder. This should typically match the name of the guest being backed up (though in some cases for VMs migrated from older VMware versions it may match an older VM name if the name was changed).
The disk referenced should be the original VMDK base disk, but in the example above there's a *-00000x.vmdk. This is not a base disk but is a snapshot.
This is also most commonly related to running older unpatched builds of ESXi.
Possible Cause 3) Another common cause is a snapshot of the vCenter VM itself was made, usually by a backup process, while VMWare was busy creating, or removing snapshots or migrating VMs with vmotion or storage vmotion. Never back up the vCetner VM directly except manually when VMWare is idle to all snapshot. vCenter is designed to be protected only by VMWare's native backup process and not by VM backup and does not support incremental snpashots. If you are protecting VMWare vCenter using Unitrends backups remove it immediately from schedules and check VMWare documentation to produce vcetner backups natively. unitrends can back up the exports vCetner creates to move them offsite safely later.