Client Backup is Running Slower Than Expected

SUMMARY

Possible causes for a single client’s backup to run slower than expected.

ISSUE

A single client’s backup is running slower than expected.
The speed of file-level, application-level and Bare Metal backups are running slower than expected.

For more detailed information on backup failures and performance issues see Unitrends KB 5062 - Backup Failures and Performance Issues

RESOLUTION

Typically, it is best to use the following test procedures in order to find the bottleneck causing the slow backup speed.  Alternatively, backups can be completed using multi-threaded jobs.  Multi-threaded jobs are particularly useful for large servers such as file servers.

1. Ensure you are using incremental forever backups if they are supported for your OS or NAS.  
2. Confirm if the backups are all universally slow or if only select systems are slow.  
   1. if universally slow, try running fewer backups (or just one) at once to see if average performance changes.  Engage Unitrends support to further review to determine the expectations of your unit or what may be slowing services if backups still do not meet expectations.  
   2. if one or only a few are slow, check to ensure the appliance and client connection contains no hops through gateways or firewalls.  Unitrends only supports backups to an appliance across a flat VLAN.  You may need to add additional network connections to your appliance to have direct VLAN connectivity.  
3. Temporarily disable any software anti-virus software.  Some anti-virus software needs to be configured to NOT scan files on backup or when files are read.  see Recommended Anti-Virus exclusions for Unitrends backup
4. Verify that the client is protected and the backup appliance are both registering at full network speed on the switch to which they are connected to.  Note, NIC speeds in ifconfig or other tools only display the negotiated speed, which may not be the NIC's current speed.  You will typically need to log into your switch to confirm actual port speeds at the current time.  10Gbit ports from most vendors require fixed speed settings.  it is not recommended for servers or other statically deployed systems to use auto-negotiate.  ensure you are using the correct spec cabling for all ports, typically a minimum of cat6 on gigabit and 6a for 10gbit.  
5. Servers containing a large number of very small files using file agent backups can be problematic when these files are presented for backup.  NTFS is single-threaded and a busy server or one with small file counts will make backups excessively slow.  Improving server hardware, ensuring the NTFS metabase is not being paged (file servers need more RAM than most believe), or improving disk IO can help.  Switching from a file backup to an image backup where supported should eliminate this bottleneck as long as the server has free IO to spare.   In very rare cases L2 may advise a customer to use aliases for file backups.   
   1. Note, a change from file backup to image backup will count as a new backup type for local storage capacity reporting as well as cloud reporting. we recommend reviewing your retention requirements for data before changing types and ensure sufficient local free space exists before performing backup type changes.  Discuss capacity with support before making this change if you are unsure.  
6. As a backup is running, view the task manager or system performance monitor to see what hardware components are being taxed.  You may also verify that all raid arrays and hard drives on the client-server are healthy.  Slow backup speeds may be an indication of a greater hardware problem.
7. for slow NAS backups, configure your chares to also support NFS and register your NAS to Unitrends using NFS instead of CIFS/SMB which should show a marked performance improvement.  Ensure your NAS vendor is not throttling per-user connection throughput.  Optionally, make smaller share mount points and back several small shares up in parallel instead of one large mount point. 
8. Ensure your backup appliance storage is not completely full.  Performing backups while the system is actively purging data to make room will be slower.  This will be exacerbated when unplanned full backups or unusually large incremental backups occur.  Use retention policies to hold your unit to a recommended 80-90% storage utilization to avoid purging at the same time as backups.  

CAUSE

The speed of backup is affected by many variables within the network, which may or may not include the following:

1. Anti-virus software actively scanning files being backed up
2. Available network bandwidth
3. network port speeds being downgraded by switch infrastructure
4. Client OS ability to seek and present files for backup (IO performance/NTFS limitations)
5. Client hardware issues causing performance limitations
6. appliance purging in parallel with incoming backups
7. under configured resources in client systems, VM hosts, or a unit ends UB virtual appliance. 

Other potential issues:  

If you are running a virtual appliance (UB) check to make sure that your appliance is deployed on storage that can operate at a recommended sustained write I/O of 500 IOPS, for large scale performance UBs protecting 20TB or more it is recommended to scale up IOPS, our largest hardware appliances have sustained database IOPS in excess of 2000.  migration of the unitrends database to higher performance storage may be recommended for larger UB units to separate IO from backup storage.  

Confirm that your switch port is hard set to gig full, or 10G where applicable.  It is common when using auto-negotiate settings on your switch for the connection to be downgraded in times of high network utilization, it is recommended for all high traffic servers to use fixed switch speeds.