Unexpected virtual backup failures on VMware 6.x: "Cannot complete login due to an incorrect username or password"

ISSUE

An unexpected backup failure occurs on VMs that were backing up previously, with no changes in the VM or Hypervisor envrionment. When the error occurs, you are unable to perform a VMware backup for hours.

RESOLUTION

Ensure that there have been no changes to password or computer names, or any vMotion of VMs during the backup process. Once confirmed, perform the following:

1. Do not expose vCenter or ESXi ports 22, 443, and 902 outside of your internal management network. The Unitrends Appliance will require access to vCenter and ESXi via port 443 and 902.
    
2. The user that Unitrends uses for the vCenter should be similar administrator@vsphere.local or you can create a similar user with the same permission but without the ability to create users.
    
3. The user that Unitrends uses on the ESXi should be root or you can create a user with the same permission but without the ability to create users.
    
4. If you see latency errors or vMotion occuring during backups, check with your storage vendor to see if you can increase performance. You can try to reduce the number of backups at a time if there is no way of changing this threshold. If you have multiple hosts, choose a few VMs from each host per Backup Job and set each Backup Job to run only after the previous backup job has completed. (For example, if you have 3 hosts with 10 VMs per host, Backup Job 1 can have 3 VMs for Host1, 3 VMs from Host2, and  3 VMs from Host 3 for a total of 9 VMs being backed up at one time across 3 hosts.)
   ​​​​​​​ 
5. Disable SSH when not in use.

CAUSE

VMware is blocking the user from authenticating. If this happens, you will not be able to use use that user for 120 seconds from the time it is locked.

1. Your ESXi or vCenter may be getting accessed via SSH or the Web interface. Look at the logs on the ESXI host and the vCenter (example: /var/log/messages) and look for anything unusual. Check the security logs and look for a for a user or application that is logging in incorrectly (e.g: an MSP or Monitoring Agent) or for brute force attempts.
2. If you are not using administrator@vsphere.local for the vCenter entry Protected Asset tab, please try this user instead. If the user you create for the vCenter is not similar to administrator@vsphere.local then certain backup or recovery options will fail.
3. If you are not using the root for the ESXi host entries in the Protected Asset tab, please try this user instead. If the user create does not have the same permissions as root then certain backup or recovery options will fail.
4. There may be to many jobs being conducted at one time. Check the VMware Tasks and Errors and look for any latency issues.