system_storage file system is 100% full

SUMMARY

system_storage file system is 100% full

ISSUE

Purpose                                                                                

This article addresses discusses how to address an issue that results in the system_storage file system filling up causing is to become 100% full.

Description

Vaulting will use local storage to create “delta” files.  Delta file contain the change blocks between backups which are used to reduce the bandwidth requirements for offsite synchronization.  If there are more than 400G of changed blocks, the default file system may become full and cause vaulting failures.

Resolution

To resolve this issue, either the amount of space used by delta creation must be decreased or the space allocated to create deltas files must be increased.

Option 1. To decrease the amount of space delta files consume:

1.  In the UI, navigate to Settings > Vaulting > Vaulting Attributes > Connection Options and Vaulting Control.
2. Decrease the amount of space allocated in the Maximum Space (GB) field.

Option 2. To increase the amount of space available for delta file creation:

*** Caution:  For advanced users only. ***

*** May cause un-repairable damage! ***

1. Use putty.exe to open a bash console on the backup system or login at a console and hit “F2” for a bash console.
2. Verify that the system_storage file system if full to determine whether these steps are necessary.  In the example below, system_storage is contained on the software raid partition md4.  This may differ based on you backup system model.

[root@system ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/md0               14G  1.6G   12G  12% /

/dev/md4              429G  429G  11K   100% /var/opt/unitrends/system_storage

/dev/md3              370G  262M  351G   1% /var/opt/unitrends/database

/dev/sdc1             236M   19M  205M   9% /mnt/boot_sync

/dev/md2               47G  958M   43G   3% /usr/bp

/dev/sdb1             236M   19M  205M   9% /boot

tmpfs                  24G     0   24G   0% /dev/shm

/dev/sda               28T  3.0T   25T  11% /backups

 3.  Change your current working directory to /backups.

[root@system ~]# cd /backups/

cd /backups

 4.  View a listing of files/directories within the /backups directory.  If this applies to your system, you will see the tmp directory linked (->) to /var/opt/unitrends/system_storage/tmp.

[root@system backups]# ls -lah

total 20K

drwxr-xr-x 13 root root                     4.0K Oct 12 11:45 .

drwxr-xr-x 24 root root                     4.0K Sep 14 16:25 ..

drwxr-xr-x  2 root root                        6 Jul 30 14:43 D2Dbackups

-rw-r-----  1 root root                      439 Oct 12 11:45 .DPUusers

lrwxrwxrwx  1 root root                       39 Jul 30 14:44 dumps -> /var/opt/unitrends/system_storage/dumps

drwxr-xr-x  3 root root                       19 Jul 31 08:42 samba

-rw-------  1 root root                     8.0K Oct 15 09:59 SISinfo

lrwxrwxrwx  1 root root                       37 Jul 30 14:44 tmp -> /var/opt/unitrends/system_storage/tmp

drwxrwx---  2 root root                       22 Sep 25 09:48 trash

lrwxrwxrwx  1 root root                       45 Jul 30 14:44 UnitrendsDataBase -> /var/opt/unitrends/database/UnitrendsDatabase

5.  Remove the symbolic link which link tmp to /var/opt/unitrends/system_storage/tmp.  If your results do not match exactly, abort the process using CTRL-C and consult an expert.

[root@system backups]# rm tmp

rm: remove symbolic link `tmp'? y

6.  Create a tmp directory inside /backups.

       [root@system backups]# mkdir tmp

7.  Now, remove the contents of the /var/opt/unitrends/system_storage/tmp directory.

[root@system backups]# rm –rf /var/opt/unitrends/system_storage/tmp/*