SUMMARY

Archive Troubleshooting

ISSUE

Archive Troubleshooting

Supported configurations

Depending on the model of Unitrends Appliance or UB host configuration, archiving may be performed via one of several devices.  Content in this section has been moved and expanded upon in the following KB: Disk Archive Compatibility

USB Troubleshooting:

USB2 is supported only on appliances that do not have eSATA or SAS ports and lack USB3.  This is currently limited to the ION1 and ION2 for devices covered by Unitrends SUpport.  All other systems must use USB3 for single bay docking stations.  
USB3 is supported only with iStar docks shipped after mid 2016 having clearly labeled USB3 blue ports and is for use only on Gen 6 and newer hardware.  USB3 support on VMWare is limited to specific VMWare-supported chipsets (contact VMWare support for a list of supported chips).  
 

Basic Dock Troubleshooting

Docking Station:
1. Is the Docking station plugged in?
2. Did you push power button (located on the rear of the docking station) ON? If you did, the dock power indicator should be illuminated.  You should also hear/feel the drive spin up if it is inserted.  

eSATA Cable “Connectivity” for 4-1 RXDA breakout cables on legacy RXDA:
1. Ensure that one end of the eSATA cable is firmly and squarely connected to each port on the back of the docking station. The order of the ports is not relevant.  Ensure the other end is firmly connected to the Appliance. the SAS connector should click or lock in place.  

If USB3.0 Cable Connectivity:
1. Ensure firm connections between the Docking station and Appliance and that the dock is connected to a BLUE USB 3.0 port.  USB 2.0 docking is not supported on USB3-equipped appliances.  Older docks that include USB2 also include eSATA are no longer supported on any current generation appliance.  If you are still using an older USB2 or eSATA dock contact your Account manager to purchase a USB3 dock. 
2. I can take 10 to 15 seconds after insertion of the disk into the dock to spin up and/or be responsive to software commands.  The dock should be powered off when drives are being changed and powered on after.  The Unitrends UI should be able to scan for and detect the drive about 15 seconds later.  

Rotational Drive considerations:
1. Ensure Drive is correctly orientated before inserting into the docking station. Not doing so can easily damage the data and power connection inside the docking station. Use the button (front center) to release the drive from the docking station.
2. Unitrends QA and development will officially support drives that we internally test and sell.  These models are listed in our Disk Archive Compatibility KB and Unitrends fully supports for these models. These models may be purchased from 3rd parties and do not need to be acquired directly from Unitrends or its partners, however, note that drives purchased elsewhere and replaced under warranty by Unitrends mistakenly may result in full Unitrends retail charge assessments. 
3. Devices of differing models or size must not be mixed in the same disk set in RXDA units.  
4. when using RXDA on generation 7 and higher Unitrends Applainces, at least 2 drives must be used with RXDA.  For single-drive use acquire a USB3 docking station.  

NOTE: Do not use drives capable of entering into a low power state on its own (e.g. Green Drives). Data loss and or corruption has been observed on these drives as their lower power state conflicts with always ready requirement needed by the Linux Operating System and the RecoveryOS.  Unitrends Support will not troubleshoot issues with "green" drives and will ask for their immediate disconnection from Unitrends Appliances.  
For drive models supported: Drives that are not noted as certified may appear to function, but many drives, even enterprise class drives, may have odd behavioral characteristics, may not present serial or model IDs properly, and we have seen many models result in SATA/SCSI feedback issues that have resulted in appliance crashes resulting in complete data loss.  Devices not on this list must be considered Use At Your Own Risk.  Support effort will be provided only such to confirm issues are software specific and would equally apply to supported devices.  Unitrends STRONGLY recommends customers only purchase certified drive models.  

Archive Speed Issues

Archive speed is a factor of several things.  

First to understand is individual archive disks will top out between 80MB/Sec and 140MB/Sec depending on drive model and method of connectivity. USB2 will top out around 20-30MB/sec.  This is the max theoretical performance of a single disk, and realistic archive speeds will likely be below these numbers.  
Archive sets prepared prior to release 8.2 will be LVM stripes and are limited to the performance of a single disk regardless of number of disks in the set.  Archive sets prepared after 8.2 are RAID 0 stripes and will have higher aggregate performance.  However, performance is still not expected to scale linearly with more drives.
Typical performance seen is 30MB/sec to 90MB/sec per disk with rare cases of faster performance. "Good" performance would typically be above 60MB/sec average for the duration of the archive.  A RAID 0 stripe of 3 drives will not have 3X performance as there is some overhead, but will be substantially faster than a single drive.   Unitrends cannot directly rate NAS archive speeds, customers questioning NAS performance should archive to disk or samba and compare vs their NAS and engage their NAS vendor if it is dramatically slower than our internal data processing rate. 

The above is the maximum speeds expected. However, other factors may render the appliance unable to export data at those speeds.  
- In order to archive data backups must be rehydrated and converted to a portable data format.  This process is slower than simply copying data to a drive and will be limited by the appliance database performance vs other appliance activity happening concurrently with the archive.   
- Active backup deduplication for running inline deduped backups, replication processes, database maintenance, or backup deletion (operations that include unhashing/deletion operations) may cause temporary table locking causing archive threads to pause waiting on database access.   Each of these processes uses a processor thread, some RAM, and internal IO.  Options exist to disable some of these services during cold copy, however this should only be done under support advisement as there may be other factors to consider.  
- Under-configured UB units with too little CPU or RAM, or systems with strained DB or backup device IO may bottleneck when too many concurrent processes are active. Locating the database in virtual appliances to higher performance storage may be needed in many deployments and is a best practice when deduplication is enabled.  Database health also of course plays a role.  Additionally, see Deployment Best Practices for Unitrends Backup to determine your CPU, RAM, and IO needs for a UB for optimal performance.  
- Active synthesis of a client asset will temporarily prevent the export of that client's backups in the same chain causing archive to pause and wait for synthesis to complete.  This will typically only be a concern on UBs where Dedupe level 1 is set in virtual appliances.   Unitrends recommends all systems use Dedupe level 3.  This setting should only be changed at time of unit deployment, changing this setting later can result in storage space contention.  
- If an encryption key is enabled on-appliance, the SIS is encrypted, and all archives will also be encrypted regardless of the setting in the archive job.  The rehydration and export which will add performance overhead to rehydration of backups.  This overhead is limited in newer generation applainces but can be an issue in UBs where hosts do not provide AES-NI hardware passthru support.  
- Compress with Encrypt master.ini setting added in 8.2 is disabled by default for performance.  Enabling this will make archives 10-30% smaller when encryption is used, but at a great cost (often a doubling) in cold copy time. 


Export speeds of individual jobs can be found in the uarchive log output.  Total time to complete an archive not only includes the individual archive times and speeds, but is also impacted as noted by times the archive engine is paused waiting on processes related to an asset's backup chain or hashes to complete. This will appear as gaps in time in log output where no operations appear to have been running.  This total time can extend archive windows.  If these gaps are appearing, it may indicate changes are needed to a customer's scheduling, retention model, or a need for increased system resources.  
 

Archive Format Types

Release 10.1 introduced Archive Format 2 which was later enhanced with Format 3 and is our default and is auto-enabled when updating to a supporting release.  This new media format uses a block dump format called an SIS within which only 1 unique block object for each exported backup is contained.  Archive format 3 additionally leverage offset databases to enhance SIS block object read performance This is similar to but not to be confused with deduplication as the uniqueness set is limited to a single backup only.  By using this format combined with index files, export of deduplicated backups can be done in a database linear operation which can up to double the speed of exporting deduplicated backups.  This archive mode may create slightly smaller archive sizes, especially for virtual machines with substantial unused disk space in the backups, but it's design intent is not data reduction but speed improvement.  

Performance Implications of 10.3+ use of Archive mode 3:  Typical performance increase on Unitrends Physical appliances is 5-20% when using mode 3 vs mode 2.  However, on some systems, especially UBs or Older On-Metal implementations where the Unitrends Database already resides on high performance storage but the backup device used for bulk storage itself has substantially poorer IO response, for example when /_Stateless in a UB is mounted to a low end NAS device, this setting may actually reduce archive performance.  To resolve, simply change the master.ini to archiveFormat=2 to revert to 10.1 native archiving.  

Supported settings for archiveFormat in 10.3+ releases are: 
1 = rehydrate deduplicated backups to cold media.  (recommended when using tape media)
2 = use Archive Format 2 partially deduplicated archives with index.  (default in 10.1/10.2)
3 = use SIS Offset Cache to improve Archive Format 2 speeds (default in 10.3+)

Note, Archive Format 2 is not used for backup types that are not already deduplicated, which is the case for RAE backups like Oracle and Sharepoint as well as NDMP, or where deduplication level is set in a UB to 2 or less.  Only deduplicated backups use Format 2, other backups are exported using Format 1, and a single archive job can create a mix of format types.  This is by design for optimal performance.   Packed CTAR format backups created by default for most backup types on 10.3+ will also always use archiveFormat=2, even when set to format 1 as this is most efficient.  

When using archiveFormat=2 on Tape media, be aware recovery of Format 2 backups will require up to 2X the original backup raw size on import to the unitrends appliance, and granular recovery of individual files from Format 2 backups on tape is not supported.  Customers using Tape archive media are encouraged to understand their recovery needs, and if appliance storage space is constrained, or if granular recovery is common from tape cold copies, changing to archiveFormat=1 is recommended.  

RESOLUTION

Advanced Troubleshooting

Unitrends Support leverages multiple command line tools for archive troubleshooting.  These scripts can be used with the assistance of Unitrends support and are not resident to the OS natively.  They must be downloaded as required.  

NOTES

The best way to get data reliably offsite in 24 hours is with Hot Copy replication.  Cold copies are best targeted for long-term compliance retention based on weekly or monthly rotations for large data sets, not for RPO/RTO operations on a daily level.  When archiving very large sets, the time to export to reasonable media must be considered, and better technologies exist than legacy archives to move data offsite for DR purposes as opposed to long-term compliance needs.