SUMMARY

Explanation of how backup and restore of version > 5.0 VMs in Hyper-V 2016 work. Limitations and restrictions Common problems and solutions

ISSUE

Overview:

With Hyper-V 2016, Microsoft introduced a new group of VM versions and completely revamped the backup and recovery model for those versions.  The Unitrends Hyper-V agent has been modified to distinguish between the old and new backup paradigms so each can be handled appropriately.  This article explains the new model and how the Hyper-V agent has been modified to support it.

Definitions:

VM Version:

Windows 2008:  version 1.0 (not supported by Unitrends)
Windows 2008 SP1:  version 2.0 (not supported by Unitrends)
Windows 2008 R2:  version 3.0
Windows 2012:  version 4.0
Windows 2012 R2:  version 5.0
Windows 2016:  version 5.0, 6.2, 7.0, 7.1, 8.0

VM Generation:

The Generation determines what types of hardware and functionality is supported by a VM.  In Hyper-V there are two supported virtual machine generations, generation 1 and generation 2. Generation 2 virtual machines have a simplified virtual hardware model, and support Unified Extensible Firmware Interface (UEFI) firmware instead of BIOS-based firmware. The majority of legacy devices have also been removed from generation 2 virtual machines.

Resilient Change Tracking (RCT):

RCT is a new method of maintaining changed block virtual disk data that Microsoft introduced with Hyper-V 2016.  The changed block data is maintained from the VM guest operating system's point of view, i.e. changes to the disk as seen from within the guest operating system.  Unitrends uses a similar method of maintaining change block data known as Changed Block Tracking (CBT).  CBT views changed block data from the host's point of view, i.e. changes to the actual virtual disk vhd(x) file on the host.  Since RCT data is not compatible with CBT data, Unitrends does not use RCT for retrieving changed block data for incremental backups.

App-Aware (Production) Checkpoints:

Backup/Restore Methodologies:

Before Hyper-V 2016, the backup and recovery model for Microsoft was through the Hyper-V VSS writer.  For backups, the VSS writer provides the list of files required to restore a VM, takes care of quiescing the VM while the VSS volume snapshots are created, then unquiescing them when the VSS volume snapshots are ready for files to be extracted and backed up.  For restores, the VSS writer takes care of the details of powering down and deleting the VM if it already exists, and creating the new VM after the VM's files have been restored.

Beginning with Windows 2016, which includes new version 6.2 through 8.0 VMs, Microsoft has begun a move away from the VSS writer, and towards a WMI based approach using existing and enhanced Hyper-V functionality.  Operations formerly handled by VSS are now handled by the Unitrends Hyper-V agent. The agent creates the list of files required to back up a VM, creates a temporary checkpoint of the VM to put the VM in a state suitable for backup, and deletes the checkpoint when it is no longer needed.  For restores, the agent restores the VM's files, then uses the Import-VM cmdlet to create the VM.

Version Handling in Core and Agent:

All VMs with version less than or equal to 5.0 are handled the traditional way using the Hyper-V VSS writer.  Version > 5.0 VMs are handled using the newer method described.  It is important, therefore for both appliance and agent to keep up with the version of the VM when the last backup was taken, and the current version of the VM on the Hyper-V host.  Beginning with release 10.0.0-1, new database fields for the VM version have been created for both VM instances and VM backups.

Older VM Backups:

The VM version tracking is new in release 10.0.0-1, so older VMs and older backups will not have a VM version value associated with them.  Any backup that does not have an associated VM version will be considered version 5.0 or lower.  For Windows 2008 R2 and Windows 2012, it is difficult to detect the version of a VM, so all VMs on these older operating systems are considered version 3.0.

VM Host vs VM Backup Version:

The current VM version on the host will determine which method is used for backups.  The VSS writer will be used for VMs with version 5.0 or earlier.  WMI will be used for VMs with version > 5.0.
The VM version when a backup was taken will determine which method is used for restoring the backup.  As with backups, the VSS writer will be used to restore VMs with version 5.0 or earlier.  WMI will be used for VMs with version > 5.0.

VM  Version Upgrades:

It is possible to upgrade a version <= 5.0 VM to a version > 5.0 VM.  The upgrade path is one way, so it is not possible to downgrade a VM's version.  This is important because it is not possible to mix VSS based backups with WMI based backups in an incremental backup chain.  When a VM is upgraded from 5.0 to 8.0, a new full WMI based backup must be taken before any additional incremental backups can be run.  For VMs with backup schedules, this will happen automatically in a process known as Autopromotion.  There are several places where a version upgrade can be detected:

• During routine inventory sync:  Beginning with release 10.0.0-1, the list of VMs returned during a host inventory sync will include each VM's version.  If the appliance detects a VM version has changed since the previous backup, the next regularly scheduled incremental backup will be run as a full instead of an incremental.
• During an incremental backup request:  Beginning with release 10.0.0-1, the Hyper-V agent will check a VM's version at the beginning of every backup.  If the backup request is for an incremental backup, and the VM's version has changed, the backup will fail with a return value informing the appliance that the next regularly scheduled incremental backup request should be a full, and will include in the backup summary the VM version so the appliance can update the database version value for that VM.
• During a full backup request.  If the backup request is for a full backup, the full backup will be run, and the new VM version will be returned in the backup summary so the appliance can update the database version value for that VM.

VM Backups (VM version > 5.0):

Full Backups

1. Save a copy of the VM's config file in the VM's config directory which is the name of the binary config file with ".preCheckpointCopy" appended to it.  This file is included in the backup. 
2. Creates an XML based config file that contains information required for IR and FLR.  This file is also saved in the VM's config file directory with filename <VM GUID>.xml, and is included in the backup.
3. Creates a temporary checkpoint that is used to quiesce the VM for backup.
4. Creates a list of files to be backed up for the VM.
5. Creates a shadow copy of each volume containing files in the list.
6. Deletes the temporary checkpoint that is no longer needed.
7. Backs up all files for the VM
8. Removes temporary files created for the backup, including the copy of the config file and the xml config file.  Also removes any reference points for the VM.

Incremental Backups

WMI based incremental VM backups are identical to full backups, except that block based disk differences are included in the backup in lieu of the actual disk files.These are obtained from the CBT driver.

Checkpoints

Backup requests include taking a temporary, short lived checkpoint of the VM in order to quiesce the VM long enough to take VSS volume snapshots of volumes containing the VM's files.  The checkpoint is then deleted since it is no longer needed.  The temporary checkpoints will have the name, "Unitrends: Temporary Checkpoint".  Although this should not happen, if any of these checkpoints are left in place after a backup has completed, they are no longer necessary and can be manually deleted.

The checkpoint can be either an App-Aware, "Production" checkpoint, or a standard checkpoint.  The agent takes an App-Aware checkpoint unless the user has specifically disabled these in the VM's settings.  The type of checkpoint taken for each of the possible VM's settings is as follows:

• Checkpoints disabled:  The agent will take a Standard Checkpoint
• Standard checkpoints enabled:  The agent will take a Standard Checkpoint
• Production checkpoints enabled:  The agent will take a Production Checkpoint, and will fail the backup if the checkpoint creation fails
• Production checkpoints enabled/create standard checkpoint if production fails:  The agent will first attempt a Production checkpoint, then will attempt a standard one if that fails.

App-aware/Production checkpoints will fail for non-Windows based guest operating systems like Linux, so it is important to set the checkpoint strategy for these types of VMs correctly.

VM Restores (VM version > 5.0):

Full Backup Restores

WMI based VM restores from full backups are accomplished by the Windows Hyper-V agent by executing the following operations, in the order listed:

1. If the VM exists on the host (a VM with the same GUID), it is powered off and deleted.
2. For each file that is to be restored, if the restore is to an alternate location (same or alternate server), the path the file is to be restored to is modified as required.
3. For the ".preCheckpointCopy" file, changes the name back to the name of the original config file.  This is the version of the VM's config file used for the import.
4. Deletes the XML config file that was created during the backup.  It is not needed for restore.
5. After all files are restored, uses the Import-VM cmdlet to import the VM.

Incremental Backup Restores

WMI based VM restores from incremental backups are nearly identical to full backup restores, except for the step where the VM's files are restored.  For incremental backup restores, the files from the most recent full backup are restored first, followed by the files from each subsequent incremental backup.  For each incremental backup, the block changes are merged into the disk file from the full backup.  Additionally, the configuration files are restored from each incremental backup, one at a time, with each file version overwriting the previously restored one.

Restored File Locations

For all restores to the original location on the original host, all files are restored to their original location.  There is one exception for VMs with non-default checkpoint paths, as described below.

For all restores to an alternate location (same server or alternate server), the following rules apply:

• The base directory for the restore will be the path that is given in the UI.  For the new UI, only volume root directories are allowed, i.e. E:\
• The restored config and runtime files are restored into a temporary location, which is their original path with the restore path substituted for the original volume designation.  For example, a VM whose original config files were in 'C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines', that is restored to alternate path 'E:\', will be restored to 'E:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines'.  Note that these files are used to do the VM import and will be deleted after the import is successfully execited.  The empty folders can be manually deleted after the restore if they are still there.
• Regardless of their original location on the original host, the restored disk files are restored into a new path '<ALT PATH>\<VM NAME>\Virtual Hard Disks'.  All disk files will therefore be restored to the same folder.
• For a VM with differencing disks, the differencing disk parent disks are also restored into this disk path.  This means that for alternate location restores, differencing disks will lose their relationship to the original parent disk.  If you want a VM restored to an alternate location to maintain its child/parent relationship to the original parent disk, you will have to manually change this using powershell after the restore.
• Once all files are restored, the Import-VM cmdlet is used to import the VM.  The VM is imported into a new VM path '<ALT PATH>\<VM NAME>\Virtual Machines'.
• If the VM has a non-default checkpoint path set, the checkpoint files will be moved to a different location during the restore as explained below.

Once the restore is complete, the VM's path structure will look something like this (in this example, VM with name 'SQLServerVM' with one disk file Disk1.vhdx, is restored to 'E:\':

The Import-VM command during restore

The Import-VM cmdlet is used to import restored VMs after all files are restored. All VMs are restored using their  original GUID regardless which host they are restored to.

Original location restores

An "in-place" Import-VM is performed when restoring a VM to its original location on the original host.  Using the command in this mode uses all of the restored VM's configuration and disk files in their restored locations.  All parent/child differencing disk relations are kept as-is.

Alternate location restores

All other restores to alternate locations are performed using the Import-VM command in copy mode.  The command in this mode makes a copy of the restored configuration files into the new VM file location.  The same disk path for source and destination disks is given to the command, so the disk files are not copied, but are used in-place.

There can be issues when importing a VM that was backed up on one host to a different host.  The Import-VM command can fail if a switch name used in the original VM on the original host is not available on the new host.  Other issues can include insufficient resources like available RAM, # cores, etc.  The switch name difference is a fairly common problem and is handled by the agent by disabling adapters that don't have a comparable switch name on the new host.  For this case, the user will have to manually reconfigure switches for the restored VM after the restore completes.  For problems with insufficient resources on the new host, those issues must be addresses separately before the restore is started.

When an Import-VM command fails, all of the restored files are left in place so the user can debug the cause of the failure and possibly atteEmpt to import the VM manually.  The entire command with arguments that caused the failure is logged in the backup messages.  The command can be copied into a Powershell window on the restore host and executed to see what errors are displayed.  Also, the Compare-VM command can be used with the exact same arguments as the Import-VM command to get a report of any incompatibilities that might be preventing the Import from succeeding.

VMs with non-default checkpoint paths

Limitations

Please note the following limitations for protection of VMs with version > 5.0:

Duplicate disk file names

If you have a VM that has multiple disk files with the same name, that are located in different paths, you will not be able to restore that VM to an alternate location.  This is because alternate location restores put all disk files into the same directory.

Clustered VMs

Hosts allowed for restores

There are several rules used to restrict which VM backups can be restored to which hosts:

• VM restores: the host O/S version must support the version of the VM being restored at the time the backup being restored was taken.  For example, a version 4.0 VM backup cannot be restored to a Hyper-V 2016 host and a version 8.0 VM backup cannot be restored to a Hyper-V 2012 R2 host.
• Hyper-VInstant Recovery (IR): the IR host must be a Hyper-V version or higher than the host where the backup was taken.  A VM from a Hyper-V 2016 host therefore cannot be restored to a Hyper-V 2012 R2 host.
• Windows Instant Recovery (WIR): the host must support the guest O/S that is being restored.  Refer to Microsoft's list of guest operating systems supported for each version of Hyper-V.  Additionally, a backup of a UEFI system can only be restored to a Hyper-V host that supports generation 2 VMs, i.e. 2012 R2 and 2016.

VM types not supported

The following types of VMs should not be backed up. It is not always possible to detect these types of VMs to filter them out of the list of available VMs for backup, so VMs of these types may show up in the inventory list.  It is important therefore to know which VMs fall into these categories so they are not backed up or added to backup schedules.

• Backup of VMs containing shared disks (.vhds) is not supported
• Backup of nanoserver VMs is not supported
• Backup or restore of VMs in a heterogeneous Hyper-V 2012 R2 / Hyper-V 2016 cluster that is in the process of a rolling upgrade is not supported
• Backup of VMs in containers is not supported

Nano-Server hosts