SUMMARY

Archive Troubleshooting

ISSUE

Archive Troubleshooting

Helper script: hdware_checker, config_checker, disk_monitoring.py, and cb_syscheck
These scripts can be used with the assistance of unitrends support. Functions of these scripts can help identify SAS or esata card configuration or driver issues and determine disk status.

Due to customer firewall set up, you may experience one of two scenarios:

(1) wget hanging or not made: suggested workaround is to wget in tunnel server and scp to customer dpu

(2) execution of the script hanging: suggested workaround is to create /root/tmp directory and then copy the script to it. Then, execute the script in tmp directory.

Supported configurations

Depending on the model of DPU or UB host configuration, archiving may be performed via one of several devices. Content in this section has been moved and expanded upon in the following KB: Disk Archive Compatibility

USB Troubleshooting:

USB2 is supported only on appliances that do not have eSATA or SAS ports, and only on VMWare, not Hyper-V or Xen.
USB3 is supported only with iStar docks shipped after mid 2016 having clearly labeled USB3 blue ports and is for use only on Gen 6, 7, and 8 hardware. USB3 support on VMWare is limited to specific VMWare supported chipsets (contact VMWare support).

Basic Dock Troubleshooting

Docking Station:
1. Is the Docking station plugged in?
2. Did you push power button (located on the rear of the docking station) ON? If you did, the dock power indicator should be illuminated. You should also hear/feel the drive spin up if it is inserted.

eSATA Cable “Connectivity”:
1. Ensure that one end of the eSATA cable is firmly and squarely connected to the back of the docking station. Ensure the other end is firmly connected to the Appliance. In October 2009, Unitrends released an improved eSATA Cable with longer connector and more pliable cable.
a. Could your customer possibly be using a previously supplied eSATA cable, or one in which they purchased elsewhere? Confirm when purchased from Unitrends.
2. If the Drive has already been determined good through some other means, ensure they have the most current version of cable:

Part # Manufacturer Description
ESAC01 KINGWIN 1 METER ESATA CABLE - ESATA TO ESATA 26AWG BLK (Style20276, E162747)
ESATA6 STAR TECH 2 METER ESATA CABLE - ESATA TO ESATA 26AWG BLK (Style2725, E74020-C)

3. if using a 4-to-1 breakout cable with a SAS port, ensure the cable is connected securely to the appliance. Note, single-bay docks are not supported on SAS ports on Generation 7 or newer appliances, USB3 dock must be acquired and used instead on these systems.

4. Is the dock connected with the correct power supply? In March, 2011 Unitrends switched from the Thermaltake to the Azio. The Thermaltake does not fit securely in the Azio. Verify the customer is using the correct power supply for the dock.
Azio docks use MODEL # PA-227SA, power supply with straight lines on top.
Thermaltake docks use MODEL # CS-1202000, power supply with wavy lines on top.

If USB3.0 Cable Connectivity:
1. Ensure firm connections between the Docking station and Appliance and that the dock is connected to a BLUE USB 3.0 port. USB 2.0 docking is not supported on USB3 equipped appliances. Older docks that include USB2 also include eSATA and must be connected to your appliance using eSATA instead, or, the old dock must be replaced with a USB3 dock. Contact your sales rep to acquire the appropriate new dock for a new appliance.
2. I can take 10 to 15 seconds after insertion of the disk into the dock to spin up and/or be responsive to software commands.

Rotational Drive considerations:
1. Ensure Drive is correctly orientated before inserting into the docking station. Not doing so can easily damage the data and power connection inside the docking station. Use the button (front center) to release the drive from the docking station.
2. Unitrends QA and development will officially support drives that we internally test and sell. These models are listed here and full support for these models will be provided by Unitrends. These models may be purchased from 3rd parties and do not need to be acquired directly from Unitrends or it's partners, however note that drives purchased elsewhere and replaced under warranty by Unitrends mistakenly may result in full Unitrends retail charge assessments.
3. devices of differing model or size must not be mixed in the same disk set in RXDA units
4. when using RXDA on generation 7 and higher units, at least 2 drives must be used with RXDA. For single-drive use acquire a USB3 docking station.

Power Supply

Part # Manufacturer Description
ST3500418AS SEAGATE 500GB DRIVE (EOL)
ST3750528AS SEAGATE 750GB DRIVE (EOL)
ST31000528AS SEAGATE 1 TB DRIVE (EOL)
ST31000524AS SEAGATE 1 TB DRIVE (EOL)
ST1000DM003 SEAGATE 1 TB DRIVE (EOL)
WD1003FBYX and YZ WESTERN DIGITAL 1TB DRIVE (EOL)
ST31500341AS SEAGATE 1.5 TB Drive (EOL)
ST32000641AS SEAGATE 2TB DRIVE (EOL)
ST2000DM001 SEAGATE 2TB DRIVE (EOL)
ST2000DM006 SEAGATE 2TB DRIVE
ST33000651AS SEAGATE 3TB DRIVE (EOL)
ST3000DM001 SEAGATE 3TB DRIVE (EOL)
WD4003FZEX WESTERN DIGITAL 4TB DRIVE (EOL)
WD4004FZWX WESTERN DIGITAL 4TB DRIVE
WD4005FZBX WESTERN DIGITAL 4TB DRIVE
ST6000AS0002 SEAGATE 6TB DRIVE (EOL)
ST6000NM0015 SEAGATE 6TB DRIVE
ST6000DM001 SEAGATE 6TB DRIVE

NOTE: Do not use drives capable of entering into a low power state on its own (e.g. Green Drives). Data loss and or corruption has been observed on these drives as their lower power state conflicts with always ready requirement needed by the Linux Operating System and the RecoveryOS. Unitrends Support will not troubleshoot issues with "green" drives and will ask for their immediate disconnection from Unitrends Appliances.
For drive models not listed above: Drives that are not noted in this list may appear to function, but many drives, even enterprise class drives, may have odd behavioral characteristics, may not present serial or model IDs properly, and we have seen many models result in SATA/SCSI feedback issues that have resulted in appliance crashes resulting in complete data loss. Devices not on this list must be considered Use At Your Own Risk. Support effort will be provided only such to confirm issues are software specific and would equally apply to supported devices. Unitrends STRONGLY recommends customers only purchase drives from this list.

Archive Speed Issues

Archive speed is a factor of several things.

First to understand is individual archive disks will top out between 80MB/Sec and 140MB/Sec depending on drive model and method of connectivity. USB2 will top out around 20-30MB/sec. This is the max theoretical performance of a single disk, and realistic archive speeds will likely be below these numbers.
Archive sets prepared prior to release 8.2 will be LVM stripes and are limited to the performance of a single disk regardless of number of disks in the set. Archive sets prepared after 8.2 are RAID 0 stripes and will have higher aggregate performance. However, performance is still not expected to scale linearly with more drives.
Typical performance seen is 30MB/sec to 90MB/sec per disk with rare cases of faster performance. "Good" performance would typically be above 60MB/sec average for the duration of the archive. A RAID 0 stripe of 3 drives will not have 3X performance as there is some overhead, but will be substantially faster than a single drive. Unitrends cannot directly rate NAS archive speeds, customers questioning NAS performance should archive to disk or samba and compare vs their NAS and engage their NAS vendor if it is dramatically slower than our internal data processing rate.

The above is the maximum speeds expected. However, other factors may render the appliance unable to export data at those speeds.
- Hashed backups will be slightly slower to archive vs unhashed backups.   This will alone usually not impact archive speed except on under-configured UBs or where DB performance is a serious concern.
- fully deduplicated backups, including those from inline dedupe or post process dedupe will archive significantly slower than simple compressed backups. Lower end units and some UBs may have trouble exporting fully deduplicated backups at disk IO speed. Customers that must archive frequently during tight windows may need to revert to post-process dedupe level 2 in order to achieve their window performance. Upgrading to release 10.1 and higher defaults to a new archive data format that will mostly eliminate this performance disadvantage, though some speed impact is still seen on lower end systems lacking "S" configurations.
- Active backup deduplication for running inline deduped backups, replication processes, database maintenance, or post-process dedupe (including unhashing/deletion operations) may cause temporary table locking causing archive threads to pause waiting on database access. Each of these processes uses a processor thread, some RAM, and internal IO. Options exist to disable some of these services during cold copy, however this should only be done under support advisement as there may be other factors to consider.
- Under-configured UB units with too little CPU or RAM, or systems with strained DB or backup device IO may bottleneck when too many concurrent processes are active. Locating the database to higher performance storage may be needed in many UB deployments and is a best practice when deduplication is enabled. Database health also of course plays a role. Additionally, unitrends recommends 4CPU and 16GB RAM minimum when using deduplication, see Deployment Best Practices for Unitrends Backup to determine your CPU, RAM, and IO needs for a UB for optimal performance.
- Active synthesis of a client asset will temporarily prevent export of that client's backups in the same chain causing archive to pause and wait for synthesis to complete. Use of inline synthesis and inline deduplication can reduce this impact. Ensuring synthesis runs on clients every 14 days and not more often can help with archive operations. Options to disable synthesis during archive exist and can be discussed with support staff.
- If an encryption key is enabled on-appliance, the SIS is encrypted, which will add performance overhead to rehydration of backups. Hardware appliances in Gen 6 or higher will have little impact from this but older systems lacking hardware encryption processors and all UBs will have notable performance impact when using encryption.
- Compress with Encrypt master.ini setting added in 8.2 is disabled by default for performance. Enabling this will make archives 10-30% smaller when encryption is used, but at a great cost in time.
- When archiving hashed or deduplicated backups, database bloat health, index health, and database IO performance (especially in UBs) can negatively impact archive performance, as can the IO performance of the volume the Unitrends Database resides on. Ensure your appliance is on release 10.2 or higher for the best available database management and maintenance automation. See
- Encrypted SIS data to an encrypted archive device requires decryption to read from SIS and re-encryption on hydration during export. This method introduces the highest overhead and produces the slowest archives, in many cases failing to achieve disk IO speed, especially in slow UBs or appliances without "S" enhancements.

Export speeds of individual jobs can be found in the uarchive log output. total time to complete an archive not only includes the individual archive times and speeds, but is also impacted as noted by times the archive engine is paused waiting on processes related to an asset's backup chain or hashes to complete. This will appear as gaps in time in log output where no operations appear to have been running. This total time can extend archive windows. If these gaps are appearing, it may indicate changes are needed to a customer's scheduling, retention model, or a need for increased system resources.

Archive Format Types

Release 10.1 introduced Archive Format 2 which is default enabled after update. This new media format uses a block dump format called an SIS within which only 1 unique block object for each exported backup is contained. This is similar to but not to be confused with deduplication as the uniqueness set is limited to a single backup only. By using this format combined with index files, export of deduplicated backups can be done in a database linear operation which can up to double the speed of exporting deduplicated backups. This archive mode may create slightly smaller archive sizes especially for virtual machines with substantial unused disk space in the backups, but it;s design intent is not data reduction but speed improvement.

With Release 10.3 this process was improved upon by creating database helper files that improve archive performance by as much as 20% especially on heavily burdened or busy systems. These are created at time of archive and are called SISOC files which will appear in your backup device, and are removed after cold copy is completed. These small files are created when the master.ini uses the default value [Archive] archiveFormat=3.

Performance Implications of 10.3+ use of SISOC: Typical performance increase on Unitrends Physical appliances is 5-20%. However, on some systems, especially UBs or Older On-Metal implementations where the Unitrends Database already resides on high performance storage but the backup device used for bulk storage itself has substantially poorer IO response, for example when /_Stateless in a UB is mounted to a low end NAS device, this setting may actually reduce archive performance dramatically. To resolve, simply change the master.ini to archiveFormat=2 to revert to 10.1 native archiving.

Supported settings for archiveFormat in 10.3+ releases are:
1 = rehydrate deduplicated backups to cold media.
2 = use Archive Format 2 partially deduplicated archives with index. (default in 10.1/10.2)
3 = use SIS Offset Cache to improve Archive Format 2 speeds (default in 10.3+)

Note, Archive Format 2 is not used for backup types that are not already deduplicated, which is the case for RAE backups like Oracle and Sharepoint as well as NDMP, or where deduplication level is set in a UB to 2 or less. Only deduplicated backups use Format 2, other backups are exported using Format 1, and a single archive job can create a mix of format types. This is by design for optimal performance. Packed CTAR format backups created by default for most backup types on 10.3+ will also always use archiveFormat=2, even when set to format 1 as this is most efficient.

When using archiveFormat=2 on Tape media, be aware recovery of Format 2 backups will require up to 2X the original backup raw size on import to the unitrends appliance, and granular recovery of individual files from Format 2 backups on tape is not supported. Customers using Tape archive media are encouraged to understand their recovery needs, and if appliance storage space is constrained, or if granular recovery is common from tape cold copies, changing to archiveFormat=1 is recommended.

RESOLUTION

Archive Troubleshooting

Helper script: hdware_checker, config_checker, disk_monitoring.py, and cb_syscheck
These scripts can be used with the assistance of Unitrends support. and are hosted in ftp.unitrends.com/utilities. Functions of these scripts can help identify SAS or esata card configuration or driver issues and determine disk status where disks may not be properly seen by the system or reported in the UI.

Note, customer firewalls that prevent FTP connectivity to ftp.unitrends.com may prevent installation of support tools. If so, the scripts can be downloaded through other systems and copied to the appliance's /samba share, then moved to /root. WinSCP may also be used to place the scripts in /root. Scripts must be marked executable to be launched.

These tools are intended to be used at the discretion of and with the assistance of Unitrends Support. Support will install these personally where a support tunnel and OS root logon credential is provided to Unitrends Support. These tools are not pre-installed as they are routinely modified for support and development purposes outside of standard release cycles.

Supported configurations

Depending on the model of DPU or UB host configuration, archiving may be performed via one of several devices. Many archive issues are commonly found to be related to the use of unapproved or unsupported devices. Content in this section has been moved and expanded upon in the following KB: Disk Archive Compatibility. Review this information to ensure you have a supported device and configuration. Troubleshooting for purposes other than recovery of prior successful archives will be strictly limited when dealing with unsupported devices.

USB Troubleshooting:

USB2 is supported only on appliances that do not have eSATA or SAS ports, and only on VMWare, not Hyper-V or Xen. It must be connected to a black USB port.

USB3 is supported only with Unitrends Provided docks shipped after mid 2016 having clearly labeled USB3 blue ports and is for use only on Gen 6, 7, and 8 hardware. USB3 support on VMWare is limited to specific VMWare supported chipsets (contact VMWare support). This dock is a proprietary device specifically manufactured for Unitrends Cold Copy Needs. Other USB 3 devices will be rejected support. The device must be connected to a blue USB3 port on a Unitrends Appliance.

Basic Dock Troubleshooting

Docking Station:
1. Is the Docking station plugged in?
2. Did you push power button (located on the rear of the docking station) ON? If you did, the dock power indicator should be illuminated. You should also hear/feel the drive spin up if it is inserted.

eSATA Dock Cable “Connectivity”:
1. Ensure that one end of the eSATA cable is firmly and squarely connected to the back of the docking station. Ensure the other end is firmly connected to the Appliance. In October 2009, Unitrends released an improved eSATA Cable with longer connector and more pliable cable.
2. If the Drive has already been determined good through some other means, ensure they have the most current version of cable:

4. Is the dock connected with the correct power supply? In March, 2011 Unitrends switched from the Thermaltake to the Azio eSATA dock. The Thermaltake does not fit securely in the Azio. Verify the customer is using the correct power supply for the dock. The iStar docking enclosure has a unique power connector incompatible with other models.
Azio docks use MODEL # PA-227SA, power supply with straight lines on top.
Thermaltake docks use MODEL # CS-1202000, power supply with wavy lines on top.

If USB3.0 Cable Connectivity:
1. Ensure firm connections between the Docking station and Appliance and that the dock is connected to a BLUE USB 3.0 port. USB 2.0 docking is not supported on USB3 equipped appliances. Older docks that include USB2 also include eSATA and must be connected to your appliance using eSATA instead, or, the old dock must be replaced with a USB3 dock. Contact your sales rep to acquire the appropriate new dock for a new appliance.
2. I can take 10 to 15 seconds after insertion of the disk into the dock to spin up and/or be responsive to software commands.

Rotational Drive considerations:
1. Ensure Drive is correctly orientated before inserting into the docking station. Not doing so can easily damage the data and power connection inside the docking station. Use the button (front center) to release the drive from the docking station.
2. Unitrends QA and development will officially support drives that we internally test and sell. These models are listed here and full support for these models will be provided by Unitrends. These models may be purchased from 3rd parties and do not need to be acquired directly from Unitrends or it's partners, however note that drives purchased elsewhere and replaced under warranty by Unitrends mistakenly may result in full Unitrends retail charge assessments. Drives of other models are not QA certified and though they may function, will receive limited troubleshooting from Unitrends Support.
3. devices of differing model or size must not be mixed in the same disk set in RXDA units at the same time. Only matching sets of 2, 3, or 4 drives are supported.
4. when using RXDA on release 10.0 and higher, at least 2 drives must be used with RXDA. For single-disk use, acquire a USB3 docking station in addition to RXDA to avoid schedule management complexities as a single disk device will be seen differently by Linux than a soft-raid stripe used in multi-disk sets. Single disk and multi-disk operations are not compatible with a single archive job schedule and will require multiple schedules or the consistency of always using multi-disk sets, or annual modification of the job at each run if changing from multi-disk to single-disk.  This was a change in 2017 made by development in response to changes in hardware generation and a desire to use a single unified process in code across all platforms.

Power Supply

Archive Speed Issues

Archive speed is a factor of several things.

First to understand is individual archive disks will top out between 80MB/Sec and 140MB/Sec depending on drive model and method of connectivity. USB2 will top out around 20-30MB/sec. This is the max theoretical performance of a single disk, and realistic archive speeds will likely be below these numbers.
Archive sets prepared prior to release 8.2 will be LVM stripes and are limited to the performance of a single disk regardless of number of disks in the set. Archive sets prepared after 8.2 are RAID 0 stripes and will have higher aggregate performance. However, performance is still not expected to scale linearly with more drives.
Typical performance seen is 30MB/sec to 90MB/sec per disk with rare cases of faster performance. "Good" performance would typically be above 60MB/sec average for the duration of the archive. A RAID 0 stripe of 3 drives will not have 3X performance as there is some overhead, but will be substantially faster than a single drive. Unitrends cannot directly rate NAS archive speeds, customers questioning NAS performance should archive to disk or samba and compare vs their NAS and engage their NAS or network vendor if it is dramatically slower than our internal data processing rate.

The above is the maximum speeds expected. However, other factors may render the appliance unable to export data at those speeds.
- Hashed backups will be slightly slower to archive vs unhashed backups.   This will alone usually not impact archive speed except on under-configured UBs or where Database IO performance is a serious concern.
- fully deduplicated backups, including those from inline dedupe or post process dedupe will archive significantly slower than simple compressed backups. Lower end units and some UBs may have trouble exporting fully deduplicated backups at disk IO speed. Customers that must archive frequently during tight time windows may need to revert to post-process dedupe level 2 in order to achieve their window performance. Upgrading to release 10.1 and higher defaults to a new archive data format that will mostly eliminate this performance disadvantage, though some speed impact is still seen on lower end systems lacking "S" configurations.
- Active backup deduplication for running inline deduped backups, replication processes, database maintenance, or post-process dedupe (including unhashing/deletion operations) may cause temporary table locking causing archive threads to pause waiting on database access. Each of these processes uses a processor thread, some RAM, and internal IO. Options exist to disable some of these services during cold copy, however this should only be done under support advisement as there may be other factors to consider.
- Under-configured UB units with too little CPU or RAM, or systems with strained DB or backup device IO may bottleneck when too many concurrent processes are active. Locating the database to higher performance storage may be needed in many UB deployments and is a best practice when deduplication is enabled. Unitrends recommends 4CPU and 16GB RAM minimum when using deduplication, see Deployment Best Practices for Unitrends Backup to determine your CPU, RAM, and IO needs for a UB for optimal performance.
Appliance database health also of course plays a role. Please ensure appliances are on release 10.2 or higher for the most current database management processes. After upgrading, it may take several weeks for database performance to become optimal if upgrading from releases prior to 10.1. For UBs, ensure database IO is not constrained. On units with larger protected capacities, or where low performance disk is used for primary backup storage, migration of the Unitrends Database to dedicated high performance spindles is strongly encouraged. It is not necessary to provide high speed disk for the entire system, only the database, typically 300GB-1TB in size.
- Active synthesis of a client asset will temporarily prevent export of that client's backups in the same chain causing archive to pause and wait for synthesis to complete. Use of inline synthesis and inline deduplication can reduce this impact dramatically. The largest impacts to synthesis will be seen in UBs not using Dedupe level 3 as well as older appliances sold and distributed prior to release 8.1 as those systems do not support inline deduplication and advanced in-place synthesis. Ensuring synthesis runs on clients every 14 days and not more often can help with archive more streamlined operations. Options to disable synthesis during archive exist and can be discussed with support staff to resolve extreme cases where storage capacity is otherwise sufficient.
- If an encryption key is enabled on-appliance, the SIS is encrypted, which will add performance overhead to rehydration of backups. Hardware appliances in Gen 6 or higher will have little impact from this but older systems lacking hardware encryption processors and most UBs will have notable performance impact when using encryption.
- Compress with Encrypt master.ini setting added in 8.2 is disabled by default for performance. Enabling this will make archives 10-30% smaller when encryption is used, but at a great cost in time. This setting is ignored with Archive Format 2.
- When archiving hashed or deduplicated backups, database bloat health, index health, and database IO performance (especially in UBs) can negatively impact archive performance, as can the IO performance of the volume the Unitrends Database resides on. Ensure your appliance is on release 10.2 or higher for the best available database management and maintenance automation. If database storage is too slow, maintenance may be unable to complete normally. Support can diagnose this condition and render manual offline assistance to correct the DB and assist moving it to faster storage.
- Encrypted SIS data to an encrypted archive device requires decryption to read from SIS and re-encryption on hydration during export. This method introduces the highest overhead and produces the slowest archives, in many cases failing to achieve disk IO speed, especially in slow UBs or appliances without "S" enhancements. Customers on UB systems with limited CPU or limited IO performance may be asked to redeploy with deduplication disabled to improve archive performance.

Export speeds of individual jobs can be found in the uarchive log output. Logs are found in /usr/bp/logs.dir. Total time to complete an archive not only includes the individual archive times and speeds, but is also impacted as noted by times the archive engine is paused waiting on processes related to an asset's backup chain or hashes to complete. This will appear as gaps in time in log output where no operations appear to have been running. This total time can extend archive windows. If large gaps are frequently appearing, it may indicate changes are needed to a customer's scheduling, retention model, advanced archive settings to limit synthesis or purging during cold copy operations, or a need for increased system resources.

Archive Format Types

NOTES

The best way to get data reliably offsite in 24 hours is with Hot Copy replication. Cold copies are best targeted for long term compliance retention based on monthly rotations for large data sets, not for RPO/RTO operations on a daily level. When archiving very large sets, the time to export to reasonable media must be considered, and better technologies exist than legacy archives to move data offsite for DR purposes as op[posed to long term compliance needs.

Archive Troubleshooting

SUMMARY

ISSUE

Archive Troubleshooting

USB Troubleshooting:

Basic Dock Troubleshooting

Archive Speed Issues

Archive Format Types

RESOLUTION

Archive Troubleshooting

USB Troubleshooting:

Basic Dock Troubleshooting

Archive Speed Issues

Archive Format Types

NOTES

Browse this section