VMware out of memory error (VixDiskLib_Read error: Memory allocation failed. Out of memory.)

ISSUE

VMware backups fail or time out after running for a few minutes. In the vprotect logs you can see the error:
 

vix_debug: VixDiskLib: Detected DiskLib error 802 (NBD_ERR_INSUFFICIENT_RESOURCES).
VixDiskLib_Read error: Memory allocation failed. Out of memory.

RESOLUTION

1. Add the ESXi hosts to the Unitrends Client/Protected Assets page.
2. Restart the hosts related to the VMs with the errors so that the current NFC connections are cleared
3. If the issue continues, contact VMware as there may be patch required to correct the NBD/NFC related issue.

For more information review VMware's document on Virtual Disk Transport Methods.

 
 

CAUSE

How vCenter backups affect the VMware Ecosystem.

Backing up vCenter using snapshots is not a good idea. The reason is directly related to the PostgreSQL database in vCenter and its internal buffering. The database server must be shut down in order to get a usable backup. Half-way measures such as disallowing all connections will not work. You might be able to use the new Crash Consistent option in 9.1.1, however, a backup created in this way saves the database files in a state as if the database server was not properly shut down; therefore, when you start the database server on the backed-up data, it will think the previous server instance crashed and will replay the WAL log. VMware recognizes this limitation and create a new tool in vCenter 6.5 to allow backups to be performed, were before there was only a KB on how to perform the backup manually. For more information on this, please look at PosgreSQL's documentation: https://www.postgresql.org/docs/9.0/static/backup-file.html

vCenter: Backup and Restore:
6.5 and newer: What’s New in vSphere 6.5: vCenter Server​
6.0 and older: 
Back up and restore vCenter Server Appliance/vCenter Server 6.0 vPostgres database (2091961)

Improving VMware Protection Performance and Resilience

When using vCenter, please add the ESXi hosts to the Unitrends as well. This adds a second level of protection (in case your vCenter goes down) but also improves backup performance, as the data does not need to traverse the vCenter on its way to/from the Unitrends system from the ESXi host. This also bypasses a VMware limitation that limits the NFC connections to 52 or less. Each disk is listed as a connection, and this value can be easily reached if the ESXi hosts have not been added to the Unitrends system. If you suspect this, restart the hosts related to the backup errors to clear the NFC connections and free up the buffers. For more details on Virtual Disk Transport Methods and this VMware limitation, please view VMware documentation:

VMware issues with Change Block Tracking and Snapshots

VMware has acknowledged and corrected several issues directly related to Snapshots and Change Block Tracking on VMware 5.5 and Vmware 6. To correct these issues on VMware 5.5, deploy Update 3b build 3248547 or newer. To correct these issues on VMware 6, deploy Update 1b build 3380124 or newer. Questions relating to these patches should be directed to VMware Support (see How to file a Support Request in My VMware).