ISSUE
Use an SSH client such as PuTTY to access the Unitrends system at the command line level. Note: Ensure you have the OS password to access the Unitrends system’s command line. The OS password may differ from the password used to access the User Interface.
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md3 15G 2.1G 12G 16% /
tmpfs 16G 96K 16G 1% /dev/shm
/dev/md0 243M 63M 169M 27% /boot
/dev/md2 99G 2.8G 91G 3% /usr/bp
/dev/md1 788G 28G 721G 4% /var/opt/unitrends/database
/dev/sda 17T 1.8T 15T 11% /backups
If the % value before /usr/bp indicates 100%, the /usr/bp/ partition is full and needs to be cleaned.
RESOLUTION
1) From the command line, change directories to /usr/bp/.
[root@Recovery-Series ~]# cd /usr/bp [root@Recovery-Series bp]#
2) Within /usr/bp/, run the command "du -sh *". This will list all files and directories and their sizes.
# du -xsh *
# du -xsh * | grep G (rough filter for files over 1GB)
3) Determine which logs are using the most amount of space.
# cd logs.dir/ # ls -alh
4) Before deleting check to find out why the logs are so large in size by inspecting the log level.
Warning: Skipping this step will result in the re-occurrence of this issue.
5) Determine any process using an elevated log level. For more information, including which logs should use an elevated log level by default that shold not be lowered, see Log Level Definitions.
# /usr/bp/bin/check_logging Checking /usr/bp/bpinit/master.ini ... bpserver=5 # WARNING: DEBUG LOG LEVEL updatedb=5 # WARNING: DEBUG LOG LEVEL autosynth=5 # WARNING: DEBUG LOG LEVEL ===ERROR: Log parameters are out of bounds for /usr/bp Number of Log sources=55 LogRetentionInstances=100 LogsPerInstance=200 MaxLogKB=10240 Max Log size = 11264000000 KB /usr/bp size = 68849048 KB ===ERROR===
6) Now, lower the log level from the UI.
Navigate to Configure > <<Unitrends appliance>> >Edit > Advanced > General Configuration
Search the "Section" column for "debugging".
7) Delete the offending log.
Warning: Once removed, this data will not be available for retrieval. If you are unsure of how to proceed, stop here and ask for assistance.
# cd /usr/bp/logs.dir/ # ls -alh # rm <offending>.log
8) Check to confirm file is not held open and pending deletion
# lsof | grep -i delete | /var/log/messages # rsyslogd 6347 root 6w REG 8,23 4714463232 131147 /var/log/messages (deleted) # in:imjour 6347 6353 root 6w REG 8,23 4714463232 131147 /var/log/messages (deleted) # rs:main 6347 6366 root 6w REG 8,23 4714463232 131147 /var/log/messages (deleted)
9) Kill running PID of process holding logs open or reboot appliance to allow linux to clear the file system and reclaim space.
# kill -9 6347 # lsof | grep -i delete | grep /var/log/messages #
CAUSE
This issue is commonly caused by logging levels at an elevated level.
NOTES
For more information, see Log Level Definitions