SUMMARY

Linux Change Journal

ISSUE

Backup warning or failure that a Differential has failed due to Journal wrapping. 

RESOLUTION

You will be editing values in the /usr/bp/bpinit/master.ini file of the computer you are protecting.

IMPORTANT: Once you complete this process, you must perform a 1-Time Full/Master using the new HTML5 Web Administrative Interface
 

Configuring the Journal

Below are some of the more critical settings that can be changed to ensure the Change Journal is as efficient as possible in tracking the changes that occur in the computer you want to protect.

The Journal feature is transparent. There are no visible configuration options in the User Interface. The agent will use the Journal during Incremental or Differential backups, and during backup operations, several conditions are checked:

• The timestamp of the first entry in the Journal needs to be earlier than the prior backup (prior Master for Differentials, prior backup for Incrementals).  Otherwise, the Journal is said to be out of range.  The backup fails with error code 80 with a message in the backup output indicating that the Journal is out of range and the user must run another Master backup to clear up the condition.
• Journal entries are checked to see if any inotify subsystem resource errors occurred.  For example, in certain environments with a high rate of change in the file system, inotify cannot handle the high volume of change events that occur, and the Journal logs an overflow condition.  Should this occur, the backup fails with exit code 81, with a message in the backup output indicating that a Journal overflow error has been seen.  The user must run another Master to clear the overflow.

Resolving the condition: Change Journal has Wrapped

A Journal that is in the out of range condition will cause subsequent backups to fail, and you will see the following message in the backup summary for the specific backup operation:

Exit Code......: 80
Exit Status....: Journal range exceeded.  Please run a new master.

When a new Master backup has been completed, the out-of-range condition is cleared and the next backup should succeed.  If you continue to see the range being exceeded error, you should consider increasing the size of the Journal.

Another error condition occurs when the number of inotify change events exceeds the kernel tunable limit.  A Journal overflow condition will cause subsequent backups to fail, and you will see the following message in the backup summary for the specific backup operation:

Exit Code......: 81
Exit Status....: Journal overflow detected.  Please run a new master.

When a new Master backup has been run, the overflow condition is cleared and the next backup should succeed unless the change rate continues to be so high that the event queue cannot effectively track all file system changes. The default inotify event queue size set the agent is 64K.  When repeated Journal overflows occur, you can increase the maximum size of the inotify event queue by adding an entry to the Client Agent's master.ini file as follows:

         [Incremental Forever]
          MaxQueuedEvents=128K

In this example, we have increased the event queue length from 64K to 128K bytes.

If you have a very large file system with a very large number of subdirectories, you may need to increase the kernel tunable parameter that specifies the maximum number of directories that may be “watched” by inotify.  The default maximum is 512K subdirectories, and this may be changed by adding an entry in the Client Agent's master.ini file as follows:.

         [Incremental Forever]
          MaxUserWatches=1M

In this example, the maximum of directories that may be watched has been increased from 512K to 1M directories.

Keep in mind that raising either of these configurable inotify limits will increase the resource (memory, CPU) usage of the server.

After any change to Client Agent's master.ini file related to the [Incremental Forever] section, you need to restart the mynotify process by typing at a terminal prompt on the server:

/usr/local/bin/mynotify stop
/usr/local/bin/mynotify start

The change journal must be used in order to support an Incremental Forever backup strategy, so in some cases, the user may not need this functionality.  Without the mynotify change journal, Incremental and Differential backups will use the traditional inotify, which uses file system traversal and compare the modification date of the files to the date of the last Master (for a Differential) and the last Backup (for an Incremental).   If the user does not want to use the mynotify change journal and wants to instead use the OS's inotify process, you can change the UseJournal value to False in the /usr/bp/bpinit/master.ini file.

Defining the maximum size of the Change Journal

The default maximum Journal size may not be enough to allow proper tracking of changes. This size is configurable by modifying the entry located in Client Agent's master.ini /usr/bp/bpinit/master.ini file.  The size may be indicated by “M” for MB.

            [Incremental Forever]
             MaxJournalSize=50M

In Redhat, this was default set to 512K and should have been 512M.
     

In the example above, we could change the value from 50M to 1024M to allow for a larger change journal. The max value is 4096M. The downside of a larger change journal file is that mynotify service uses more resources on the server to manage the larger change journal, but there is less of a chance that the change journal gets out of range (and starts to remove old references). Your system must have enough free memory during the backup allow the change journal to load into RAM up to the value listed (1024M means 1GB of RAM).

When the Journal exceeds the maximum size, it is resized by dropping the oldest change events until it reaches a size that is less than or equal to a ratio of its maximum size, which is, by default, 80%.  The oldest events are no longer needed if a successful backup occurred after the date of these dropped events.  If you lower the Truncation Ratio, more change events in the Journal will be purged when the Journal is pruned and if raised, fewer items are purged.  The more items are purged, the more of a chance that the Journal enters the out-of-range condition, so take care when decreasing the ratio.  To change the ratio, modify the entry in the file.

Disclosing paths and path/filenames to be excluded from journal tracking

By default, the Unitrends Linux Client Agent’s mynotify process will watch for changes in all system files and directories.  When a server creates and deletes a large number of temporary files, this can cause mynotify to needlessly consume server computational resources.  To handle this situation, mynotify is configurable and allows the user to exclude directories that contain temporary files from the list of “watched” directories.   To exclude these directories, add the full path or path/filename to the proper exclude file, one item per line.

• Newer OSes: /usr/bp/incremental_forever/journal.excludes
• Older OSes: /usr/bp/bpinit/watches.exc

For example, if you wish to add the subdirectory /var/opt/myplatform/tmp, /home/user1/tmp and the file /root/insltall.txt from the exclude from watch list, add these entries one item per line to the proper watch exclude file:

/var/opt/myplatform/tmp/
/home/user1/tmp/
/root/install.txt

Please note that anything added to the watch exclusion file will not be monitored by mynotify and therefore changes will not be recorded in the Journal.  This means that changes to files in these directories or their subdirectories will not be included in the incremental or differential backups.  Please ensure that only directories that you do not need protected, e.g., temporary files, are included in the watches exclude file.

To stop using mynotify and revert to using the OS's inotify process

Recent development efforts have classified a rule of thumb of 100 changes per second as being the threshold to safely track mynotify changes . If after setting the MaxJournalSize and entering all items you want to ignore in the proper Excludes for for your OS you still encounter nonstop high CPU usage or Change Journal Wrap conditions continue, you can stop using mynotify and use the OSes native inotify process. To top using the mynotify process, follow the instructions in KB 2477 titled "How do I disable mynotify from running".

To return to using the Unitrends mynotify process, set the value to UseJournal=True

Remote Mounts
Inotify will only detect changes on file systems that are directly connected to the server on which the Linux agent is installed.  If file systems are remotely mounted on the Linux server, the Journal will not contain changes to these remote file systems.  To protect these file systems you must either install an agent on the server that hosts these file systems locally, or you can configure NAS storage on the DPU and back up the files using backups on the appliance itself.  NAS backups do not use the Journal but instead use the directory traversal/file modification time scanning technique in order to determine which files should be included in a backup.

CAUSE

Overview of the Linux Agent Change Journal

Today, customers with large numbers of files (typically in the millions) can experience very long Incremental and Differential backup times. The primary reason for this is that the Linux agent traditionally has scanned all files/directories on the specified volumes to determine modification times. Files with modification times more recent than the last successful backup are included in the Incremental backup. This means that a large percentage of the time is spent performing the backup.

Use of a journal that keeps track of changed files can eliminate much of the overhead contributed by these factors. The agent uses a Linux kernel subsystem, inotify, which can detect changes to files that are in “watched” directories.  When the agent is installed, a process known as mynotify is started on the server.  This utility initializes the inotify module, builds a Journal of all directories and files on the system, then logs all file changes along with the nature and time of the change to the Journal.

During an Incremental or Differential backup, the Unitrends agent will query the Journal to discover the changes made to files on the server.  It will analyze the data logged for each change to determine if the time of the change qualifies the data for backup. This is done by comparing the modified data to the time of the last backup (the last Master backup for a Differential, the last Backup for an Incremental). Since Journal entries are kept only for changes to files or directories, determining which files to back up on the system requires just a fraction of the time needed for the traditional file system scan.  As the number of files on a file system increases, the benefit of using the Journal also increases.

NOTES

Note: For Oracle on Linux. If you are running an incremental forever schedule, you must also exclude
Oracle database directories from journal tracking. See KB 3358 titled "Oracle Database: Incremental Forever Schedules on Linux Platforms" for details.