SUMMARY
Failed while reading data for disk [LUN]
ISSUE
Error in ui reports vmware backups are failing with an error similar to:
VMWare backups Failed while reading data for disk [LUN] <server>/disk-0000xx.vmdk
For more detailed information on backup failures and performance issues see Unitrends KB 5062 - Backup Failures and Performance Issues
RESOLUTION
**Update** - Customers should engage VMware if they would like confirmation from them this is a VMware issue. VMware will likely ask to migrate off of the affected host and reboot the host. VMware should also advise to update to the latest build for their VMware version.
Try to restart services on the ESX host where the VM resides: services.sh restart
-This may resolve the issue and allow the backups to proceed
When pre-existing snapshots exist for a VM, they have to be cleaned up.
The easy way that often (though not always) works, is to create a snapshot for the VM and then consolidate. Manually check the datastore after consolidation to confirm the snapshot files are removed. Perform a new full backup of the VM, and if the issue returns, contact VMware support directly, not Unitrends support, to diagnose snapshot issues with the VM host.
If UI consolidation does not remove the files, manual consolidation is required. VMware has a knowledge article on this:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002310
A specific article for 5.5 can be found at:
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2003638
If manual consolidation does not work, please contact VMware for assistance. Unitrends Support does not have the capability to review VMware's logs related to snapshot consolidation failures.
CAUSE
**Update**- The most common root cause as of this update, 4/5/17, is a VMware memory-related condition.
2017-03-31 12:23:09.860 -0400 : I vix_warning: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 4194328 bytes
2017-03-31 12:23:09.861 -0400 : D vix_debug: DISKLIB-LIB : RWv failed ioId: #2 (802) (34) .
2017-03-31 12:23:09.861 -0400 : D vix_debug: VixDiskLib: Detected DiskLib error 802 (NBD_ERR_INSUFFICIENT_RESOURCES).
2017-03-31 12:23:09.861 -0400 : D vix_debug: VixDiskLib: VixDiskLib_Read: Read 8192 sectors at 14336 failed. Error 2 (Memory allocation failed. Out of memory.) (DiskLib error 802: NBD_ERR_INSUFFICIENT_RESOURCES) at 5178.
2017-03-31 12:23:09.861 -0400 : D vpt_backup_get_chunk:471: Breaking out of the retry logic when change tracking is off.
2017-03-31 12:23:09.861 -0400 : E VixDiskLib_Read error: Memory allocation failed. Out of memory.
The [LUN] will be the name of the VMware datastore trying to be accessed.
Within the LUN is a guest-name folder. This should typically match the name of the guest being backed up (though in some cases for VMs migrated from older VMware versions it may match an older VM name if the name was changed).
The disk referenced should be the original VMDK base disk, but in the example above there's a *-00000x.vmdk. This is not a base disk but is a snapshot.
This indicated VMware at some prior point failed a snapshot consolidation. Further it will likely not show the VM running from a snapshot when reviewing snapshot manager, but multiple such snapshot files will likely exist in the datastore. This is a VMware defect, and can be related to older VMware releases, use of VMware tools versions that are not current, interruption of vCenter during migration or backup operations, or unplanned termination of a live backup job. In the latter case, Unitrends should clean up the snapshot on the next backup, but if that consolidation cannot complete within 10 minutes, backups may still fail.
One common cause for a snapshot to be left behind is the direct backup of vSphere as a VM while other backups are in progress. NEVER back up a vCenter virtual appliance directly. They also must not be deployed in HA pools.