Error: No more space on device

SUMMARY

Parent article to consolidate information to understand and alleviate "Out of Space" conditions on a Unitrends appliance.

ISSUE

I deleted many backups but the disk space has not been reclaimed
Why don't I have more free space available

A schedule summary report or backup failure indicates:

NO MORE SPACE ON DEVICE

Backup fails with error message:

FATAL: NO MORE SPACE ON DEVICE (DESPITE RECOVERY ATTEMPTS).
No purgeable backups were found.

RESOLUTION

• Check your license for the D2D value.  If the value is not "INF" make sure the sum of your D2D devices takes advantage of all of this space.  If the backup system is also a vault then you will need to understand that part of this space cannot be used for D2D devices as some is needed for vaulting storage as well.  Instant Recovery space allocation further reduces this value.  A common issue after redeployment is that the D2D value will be reduced to 200GB and may require adjustment.
  • For more detailed information on setting up your D2D devices see How to reclaim storage space
• Check your Capacity Report to see if you have exceeded your total capacity.  If you have then you need to back up less data.  To do this, you should remove a client from the backup system.  Removing a single backup, such as a master, for a client will not help because the report will simply reference the next newest backup of that type, to reduce capacity it is necessary to remove all schedules for a client, then remove the client itself (or in the case of VMs or databases manually delete all backups for that instance).  Note, total capacity assumes 2:1 compressibility of data.  If your data reduction values from the Data Reduction report indicate your ration is less than 2:1, you may be out of space on your appliance before reaching this value.  
  • For more detailed information on how capacity and change rate affect storage space see Insufficient space for backups caused by capacity and change rate
• If you have assigned Instant Recovery allocation, ensure this value is appropriate.  You should assign 110% of the native master backup size for any client using on-host WinIR and 10% of the size of off-host WinIR.  Higher values are not typically required.  
• Make sure you are not performing a backup of the appliance to itself.  In the legacy UI, if the system client is active in the NAV pane ensure that there are no backups for the system to itself.  Remove them if they exist, delete associated schedules, and then disable the view in the system client using the NAV pane options.   It is not possible to see the system client in the Satori UI to avoid this potential conflict. 
• If you are using a UEB and have added VMDK or VHDX storage as "internal" extended disk, consider adding more disk.  Follow the instructions to expand storage in the Unitrends RS/UEB Admin Guide.  If you have attached NAS or other direct attached storage to your UEB then it is not expandable, but you may deploy additional UEBs under your SAR license key. UEB licensing is not limited to a single deployed backup unit, the license limitation is on total protected content across all deployed units.
  • For detailed steps to add an attached virtual disk for backup storage see Adding an Attached Disk to an Appliance
• If the capacity numbers appear normal, typically 50-70% max in use, check for legal hold backups using the legal hold report. If you have any backups on legal hold, consider reducing or removing the legal hold status from backups.  Keep in mind, legal hold protects all backups in a chain until the most recent backup in a chain is more days old than the hold time.  If you are doing an incremental forever backup chain, a master will typically synthesize every 14 days. If you have legal hold set to 7 days, you must be able to retain 21 days of data for that client before backups can purge (the youngest backup in the chain, which might be the 14th day before the new master is created, would have to age a further 7 days to be eligible to purge.  Since the master before it is a dependent of that incremental, it also cannot be purged), which will include more than 2 masters.  Generally, legal hold should not be used on an appliance unless you can routinely maintain at least 3 masters of all clients all times, and then legal hold should be set to a max of 2 backup rotations less than you can attain.  Legal Hold example
• After reviewing the above, check for backups that have very high change rates.   Review the Backup History report, sort by largest backups, and then look for incremental or differential backups that are more than 1-3% of the size of the client's associated master.  In normally operating units, incremental backups added together over 7 days should be 10-15% or less than the master size. If your data change rate on some clients (or on some days) is larger than 10%, you should review the nature of the data on that system to determine why the change rate is high.   Common issues are the use of database dump scripts or maintenance plans, recursive permission changes to folders, antivirus activity, large page files or heavy page use, databases that have not been excluded from file level backups.  There may be other reasons, and that change rate may be normal and expected for some applications. If you have high change rate that cannot be avoided a larger system appliance may be required to accommodate it.
  • For more detailed information on how capacity and change rate affect storage space see Insufficient space for backups caused by capacity and change rate
• Check for failed or canceled backups that have not been automatically removed yet. Some of these can hang around for up to 30 days. To do this you can go to the Recover screen and set the filter to the last 30 days or more and then deselect the 'Successful' check box. This will present all backups including the failures. Then you can manually delete all failed backups. Once deleted these should start clearing out within the next few hours.
• Release 9.2 and 10.0 customers: a defect in space calculation can cause issues for a small number of customers.  If the above has been reviewed, contact Unitrends support to see if a patch is required, or, upgrade to release 10.1 which should include this patch.  More info on this issue is noted in KB 5917.  
• If none of the steps above work then it may be time to pursue purchasing an additional backup system.  Please contact Unitrends Sales for details on how to have your system capacity reviewed and assess if a larger system is required.  If you believe the storage reported is in error, our would like technical review of your unit, contact Unitrends support instead, and support can also engage your sales representative if that is determined required to resolve a capacity issue. 

CAUSE

The logical D2D device configured for the job is at capacity and is having difficulty purging older data in real time (or at all).
The appliance may not be large enough to support the unique data set being protected.