SUMMARY
VMware backup complete with warnings, inventory sync errors on vCenter, VMs are unrestorable due to vCenter connection issue
ISSUE
When navigating to Configure > Protected Assets and editing the vCenter instance an error occurs, even though the credentials and IP address are confirmed to be correct
Inventory sync fails, the inventory sync logs show an error when connecting to vCenter or an ESXi host.
inventory_sync-9.log:2018-06-13 20:00:06.033 -0700 : [LOG0] vmware.c:6008: bpGetVMInfo: Error: Specified ESX server not found in vProtect output inventory_sync-9.log:2018-06-13 20:00:06.033 -0700 : [LOG0] rest_inventory.c:1167: bpDoInventorySync: bpGetVMInfo failed for server vcenter.domain.local with uuid 12345678-1234-1ab2-a123-1a23bcdf456e inventory_sync-9.log:=== Error: Specified ESX server not found in vProtect output
Locating the inventory sync logs
Use an SSH client such as PuTTY to access the Unitrends system at the command line level. Note: Ensure you have the OS password to access the Unitrends system’s command line. The OS password may differ from the password used to access the User Interface.Navigate to the logs directory and open the most recent inventory sync log.
[root@Recovery-Series ~]# cd /usr/bp/logs.dir/ [root@Recovery-Series logs.dir]# ls -rlath in* [root@Recovery-Series logs.dir]# less inventory_sync-9.log
RESOLUTION
For the processes below, the following must be true:
vCenter is of the same or higher edition as the prior version deployed
NO OTHER CHANGES in the environment have occurred outside of updating vCenter (no new or changed esxi hosts)
Option 1) Automatic Resolution:
Customers who update their backup appliances to Unitrends Release 10.3.0 or higher may perform this operation via the UI.
- Ensure the new vCenter uses the same IP and DNS name as the prior vCenter and is of the same or higher edition release
- Log into the unitrends UI and Navigate to Configure > Protected Assets Tab
- Select your vCenter from the list of systems and click the Edit button
- Enter the vCenter credentials and save.
Option 2) Manual resolution
*** Caution: For advanced users only. ***
*** May cause un-repairable damage including loss of historical backups! ***
Run the following vcenterfixup script on the Unitrends appliance to locate the new or updated vCenter:
Use an SSH client such as PuTTY to access the Unitrends system at the command line level. Note: Ensure you have the OS password to access the Unitrends system’s command line. The OS password may differ from the password used to access the User Interface.
- Run this command in a SSH session:
wget ftp://ftp.unitrends.com/support/scripts/vcenterfixup -P /usr/bp/bin/vprotect ; chmod +x /usr/bp/bin/vprotect/vcenterfixup;
- The vcenterfixup script has two options. Be sure you read and understand the commands required and the changes being made prior to committing any changes. Improper use of these commands may cause un-repairable damage
- Option 0 shows the changes that will be made but does not commit any changes.
- Option 1 commits the changes on the first pass and should be run with caution.
- Use single or double quotes for the username, depending on the domain syntax.
- Now run the following script to test:
/usr/bp/bin/vprotect/vcenterfixup --server=<vCenter IP Address> --username=<username> --password=<password> --dbfixup=0
- Now run the following script to make and commit the changes shown in step 3:
/usr/bp/bin/vprotect/vcenterfixup --server=<vCenter IP Address> --username=<username> --password=<password> --dbfixup=1
- Now refresh the UI. VM's will now refresh normally.
CAUSE
vCenter's UUID is a key item that must be tracked independent of it;s name and IP to understand VM relationships in a cluster. When following proper vCenter upgrade processes, vCenter imports the prior vCenter's configuration and identity, either by communicating with the existing vcenter during replacement, or by importing VMWare's vcetner backup created by the end user.
If a proper process to update or replace vCenter per VMWare's documentation is not followed the new vCetner will have a unique ID and be incompatible with being added to our database. Manual action is necessary to update the database records to assume the new ID as opposed to adding it in conflict.