SUMMARY

Steps to improve archive performance

ISSUE

Unitrends archiving performance on a next gen appliance or UEB using Inline Deduplication is noted to be slower than older generation appliances that did not have that feature in some cases.
Archive jobs are extending beyond an acceptable deadline.

Archive speed is a factor of several things.

First to understand is individual archive sets will top out between 80MB/Sec and 140MB/Sec depending on drive model and method of connectivity. USB2 will top out around 20-30MB/sec. This is the max theoretical performance. Archives should not be expected to exceed these speeds when exporting to single disk. The typical disk will be around 90MB/Sec.
Archive sets prepared prior to release 8.2 will be LVM stripes and are limited to the performance of a single disk. Archive sets prepared after 8.2 are RAID 0 stripes and will have higher aggregate performance.
Typical performance seen is 30MB/sec to 90MB/sec per disk with rare cases of faster performance. "Good" performance would typically be above 60MB/sec average for the duration of the archive. A RAID 0 stripe of 3 drives will not have 3X performance as there is some overhead, but will be substantially faster than a single drive. Unitrends cannot directly rate NAS archive speeds, customers questioning NAS performance should archive to disk or samba and compare vs teir NAS and engage their NAS vendor if it is dramatically slower than our internal data processing rate.

The above is the maximum speeds expected. However, other factors may render the appliance unable to export data at those speeds.
- Hashed backups will be slightly slower to archive vs unhashed backups. This will alone usually not impact archive speed except on underconfigured UBs or where DB performance is a serious concern.
- fully deduplicated backups, including those from inline dedupe or post process dedupe will archive significantly slower than simple compressed backups. Lower end units and some UBs may have trouble exporting fully deduplicated backups at disk IO speed. Customers that must archive frequently during tight windows may need to revert to post-process dedupe level 2 in order to achieve their window performance. The best way to get data reliably offsite in 24 hours is with hot Copy replication. Cold copies are typically expected for long term compliance based on monthly rotations, not for RPO/RTO operations and the platform is not optimized to accomodate legacy models for data rotation. To accomodate daily or weekly cold copies of large data sets, dedupe level 2 may be required. Archive mode 2, enabled by default after 10.1.0 primarily mitigates this issue.
- Active backup deduplication for running inline deduped backups, replication processes, database maintenance, or post-process dedupe (including unhashing operations) may cause temporary table locking causing archive threads to pause waiting on database access. Each of these processes uses a processor thread, some RAM, and internal IO. Underconfigured units with too little CPU or RAM, or systems with strained DB or backup device IO may bottleneck when too many concurrent processes are active. Locating the database to higher performance storage may be needed in many UB deployments and is a best practice when deduplication is enabled. Database health also of course plays a role.
- Active synthesis of a client asset will temporarily prevent export of that client's backups in the same chain causing archive to pause and wait for synthesis to complete. Use of inline synthesis and inline deduplication can reduce this impact. Ensuring synthesis runs on clients every 14 days and not more often can help with archive operations.
- If an encryption key is enabled on-appliance, the SIS is encrypted, which will add performance overhead to rehydration of backups. Hardware appliances in Gen 6 or higher will have little impact from this but older systems lacking hardware encryption processors and all UBs will have notable performance impact when using encryption.
- Compress with Encrypt master.ini setting added in 8.2 is disabled by default for performance. Enabling this will make archives 10-30% smaller when encryption is used, but at a great cost in time.
- When archiving hashed or deduplicated backups, database bloat health, index health, and database IO performance (especially in UBs) can negatively impact archive performance.
- Encrypted SIS data to an encrypted archive device will have the worst case performance and will most commonly fail to attain disk spindle performance for export.

Export speeds of individual jobs can be found in the uarchive log output. total time to complete an archive not only includes the individual archive times and speeds, but is also impacted as noted by times the archive engine is paused waiting on processes related to an asset's backup chain or hashes to complete. This will appear as gaps in time in log output where no operations appear to have been running. This total time can extend archive windows. If these gaps are appearing, it may indicate changes are needed to a customer's scheduling, retention model, or a need for increased system resources.

RESOLUTION

For cases where archive performance is more important than the storage footprint, disabling inline deduplication is an option. Note: doing so will cause appliance retention to be reduced dramatically.

Before resorting to these steps, upgrading your appliance to release 10.1 or greater is first recommended as significant performance enhancements are included. Release 9.1, 9.2 and subsequent update up to 10.1 contain improvements to archive thread performance for deduplicated backups and future releases will continue to improve this performance.

If that does not resolve your issue, you may also seek to add the following configuration change in /usr/bp/bpinit/master.ini: add SuspendPurge=1 in the [Archive] section. This must be done through an SSH client such as PuTTY using a command line editor. This option can cause space contention issues on appliances with limited storage space, use this option only if you can reliably maintain 2 or more masters of all clients at all times.
Note: When accessing the system through an SSH client or PuTTY, ensure you have the OS password to access the Unitrends system’s command line. The OS password may differ from the password used to access the User Interface.

[Archive]
SuspendPurge=1
LimitSynth=0 prevents, 1 or higher sets max concurrent synth. Default value is -1, and allows the appliance to determine how many parallel synthetic backups to use

For UB Virtual Appliances: Before following the below steps, you should review moving the Unitrends Database to higher performance storage. This is discussed in this article.

Further, ensuring your cold copy jobs are not concurrently running along with other large new backups sets may also improve performance in some cases, especially on low end units or UB systems deployed with limited resources of slow IO.

If you have tried the above and still cannot meet your archive windows for cold copy, and the best practice for having daily offsite data of using Hot Copy replication cannot be used, then the below steps will allow deduplication to be disabled on 9.0 systems. On 9.1 systems, see the instructions for changing the Deduplication level using the UI located in our Admin Guide.

WARNING: The steps below will cause a significant and immediate increase in new backup landing zone requirements and will substantially reduce overall appliance retention. If this change is made on an appliance with little active free space this may lead to backup failure. PRIOR to enabling these setting changes, manually reduce retention settings, or, manually delete older backups and ensure the appliance stabilizes with free space equal to at least 30% of your protected data set size. Appliances are sold under the expectation Inline Dedup will be used, and disabling this feature may not only result in lowered retention, but the inability to maintain minimum backup retentions in some cases necesitating appliance chassis to be upgraded to a larger configuration. These changes are intended to be used ONLY where Hot Copy Replication cannot be done to achieve daily offsite data rotation, and where cold copies must be done within specific RPOs.

Below process should only be performed under extreme cases following update to 10.1.

Disable Inline Deduplication

In 9.1 systems, in the Unitrends UI,

access Global options at the top of the UI, select Options > Deduplication Settings.
Change the Dedupe level from 3 to either 2 or 1.
If not there, you will need to check if this is a dedupdb system. SSH to the appliance and run sys_tune to verify the database type.

In 9.0 systems in the Unitrends user interface:

Go to Configure > Appliances
Select Edit > Advanced tab and click on "General Configuration"
Scroll down to the fileDedup section.
Under the Name column, find the entry enableInlineDedup, then click on it to pull up the values section.
In the Value box, change the Yes to a No, then press Confirm.
Press the Close button, then go back into the same values to make sure that the change was registered.
Start a new backup to verify that all systems are running correctly.

Alternate method using an SSH session:

Log in to an SSH session with root access.
You will need to change the master.ini file on the appliance in order to disable inline deduplication. To do this, issue the command: vi /usr/bp/bpinit/master.ini
Once in the master.ini file, find the section [fileDedup].
Change the entry enableInlineDedup=Yes to enableInlineDedup=No
Save the changes, then go into the master.ini file again to verify that the change was saved.
Start a new backup to verify that all systems are running correctly.

CAUSE

Unitrends introduced inline deduplication targeting a reduced storage footprint for backups. As backups are ingested into the backup appliance, they are also deduplicated thus minimizing the storage required for storing backups. Archiving is impacted in that backup must be rehydrated in order to be stored in the full, rehydrated state on the archive media. The additional overhead of rehydrating the deduplicated backup increased the time required for the archive to complete. This is a trade off of storage space vs speed.

Faulty archive disk or archive hardware can greatly impact archive success and/or performance. Archiving performance may also be impacted by other processes on the Unitrends system such as the Unitrends Autosynth process as it is competing for processing power and storage I/O. UEB units deployed to slower IO subsystems, especially when using NAS disk for backup and datatabse storage are often recommended to disable deduplication if the database cannot be migrated to high performance SAN or local disk.

NOTES

When inline deduplication is disabled, the Unitrends appliance will continue to deduplicate backups using a post-backup process. Post-backup deduplication preserves the most recent backups in full while deduplicated older backups to minimize the storage footprint of older backups. As a result, backups which are not the most recent backups, classified as “last backup”, will require more processing to be duplicated during the archive process. The oldest the backups will be impacted the greatest as they are subject to the most deduplication.

Archive performance is slow