What is the best way to backup our vCenter?

SUMMARY

We'll explain the best way to backup our vCenter virtual appliance.

ISSUE

Question: What is the best way to backup our vCenter?

Answer: Use VMWare's native tools included in vCenter itself to export vCenter configuration to a network path and allow your backup system to protect that path. 
Not Recommended (and in most cases not supported): leveraging a backup solution to protect the vCenter VM directly.  

Following is a list of problems that may occur when the vCenter is being backed up virtually as a VM: 

• A message in vSphere that it has lost connectivity.
• The post-backup process is encountering a storage error.
• If using UB Virtual Appliance, the UB may experience a loss of communication with the Hypervisor/vCenter resulting in anomalies in backups.
• Other Guest VMs that are being backed up at the same time, may encounter a locked snapshot that will restrict access to the virtual machine
• Snapshot consolidation may fail.
• DRS may fail to respond and may not appropriately move VMs or resources when needed.
• vCenter PSQL database changes may be lost which can result in recovered vCenter VMs requiring manual database repair or restoring corrupted.  
• VM migrations or snapshot operations may fail.  

RESOLUTION

Where you have multiple hosts and independent storage systems, the use of vCenter HA is strongly encouraged to avoid vCenter-related system outages.  Depending on vCenter manual restoration is not ideal. 

To account for environment-level failures that necessitate full vCenter restore, backup of vCenter typically requires the use of VMWare's native tools to protect it's PSQL database engine.  

To avoid additional complications in the backup and restore of VMWare environments, ensure you have registered not only your vCenter server as a protected asset to Unitrends but EACH ESXi host as well.   Catastrophic recovery of vCenter using any method other than VMWare's own native backups will require this to have been done in advance of any vCenter outage. 

In addition to leveraging VMWare's native vCenter backup tools as noted above, VMWare supports vCenter 6.0 and higher to be protected by VM image backup, HOWEVER, this has inconvenient restrictions and is typically not viable for routine or scheduled backups that will be discussed below.

For All vCenter Versions
The safest method is to use vCenter's best practice for backing up itself, which uses native tools.  These tools can be scheduled to run exports safely and periodically to network resources with vCenter 6.5 and newer.  

vCenter 6.5 and Above
Starting with vCenter 6.5, customers should become familiar with the new backup utility inside vCenter through the vCenter Server Appliance Management Interface.   This is a simple, schedulable export that can be done without production impact to vCenter itself.  

This is done by:

1. Create a network share in your environment on a system Unitrends will protect to house vCenter backups.  Each backup typically takes only a couple of GB of space.  VMWare supports SMB and NFS shares and can also leverage SFTP and some other types of targets.  
2. Log into the vCenter Server management UI at https://<vcenterip>:5480 as a local vCenter administrator
3. Access the "backup" option on the navigation pane and select to edit a backup schedule.
4. Complete the fields in this form based on your options.   
   1. The Backup location must use the format <protocol>://<IP or hostname>/<share_name>/[folder]
   2. The username for windows systems will typically need to be in the format user@fqdn.  domain\user will usually not work.   
   3. The recommended schedule is daily at a time VMware itself is most likely to be idle (a time other than peak backup times).  A Daily schedule is recommended for larger environments, weekly can be OK for relatively static or small environments.  
   4. As vCenter backups are small it is recommended to retain a reasonable number of them in the share, at least 7 days worth or several RPOs if performing these backups weekly.   
   5. On successful save, click the "backup now" option to ensure vCenter completes a backup successfully to the chosen share
   6. Configure Unitrends to protect the system containing the share at least as frequently as your VMWare backup schedule setting using an incremental forever backup strategy.  

vCenter 6.0 and older

Unfortunately, if you are still running VMWare 6.0 or older, protection of vCenter requires command line processing and must be done manually.  This process will result in vCenter being unavailable during the backup process.  The lack of an online backup tool to protect vCenter is one of the biggest reasons to upgrade off of older VMWare editions (not to mention all versions this old are unsupported and are no longer receiving critical security updates).  

Virtual Backups of a vCenter are not supported for VMWare 5.5 or older at all and can result in production database damage if attempted.  You must use VMware's database export best practices and provide that to Unitrends for collection in another VM or as a Client Agent FLR. Undesired results can occur if you backup these older vCenter VMs when the database is still running.

VMWare 5.5 process is here.  The same process for 6.0 can be found here.  

vCenter 6.0 and higher
You can also optionally perform a virtual backup of the vCenter by running an OnDemand Full of vCenter.  This must be done only after confirming that there are no other backups occurring, and no snapshots or vmotion in progress. You should also perform the work indicated in VMware's document "Back up and restore vCenter Server Appliance/vCenter Server 6.0 vPostgres database".  VMWare recomends this in addition to native vCenter backups, not in place of them.  

Warning: VMware only supports the virtual protection of vCenter 6.0+ using FULL BACKUPS. Incremental forever backup, though generally recommended for all common VMs, are not supported for vCenter itself. This is a VMware limitation, not a Unitrends limitation. Further, such backups should only be done MANUALLY and never via a scheduled operation as it is critical that other VMware operations managed by vCenter including other backups, snapshots, consolidations, migrations, and more not be happening during this operation. If you have been performing incremental or differential backups of a vCenter virtual appliance recovery may result in corruption of the vCenter database. See https://kb.vmware.com/s/article/2091961 for more information.

CAUSE

vCenter uses a high performance database called PostgreSQL which stores most of the database and changes in RAM. This database engine is not virtualization-aware and does not handle the snapshot request like a Windows computer can with MSSQL. When VMware performs the snapshot stun, data in RAM is at risk for loss affecting current snapshots or vmotion processes.  This will typically result in a recovered vCenter that will not start it's database without manual repair.  The only safe way to protect PSQL engines is with a native tool.