Oracle Database Backups Fail Due to "cp: cannot stat `/usr/bp/mnt/<user_directory>': No such file or directory

SUMMARY

Oracle backups fail due to local mounted directory (/usr/bp/mnt/

ISSUE

Oracle backups fail due to local mounted directory (/usr/bp/mnt/<oracle_db_name>/<backup_type>.task) cannot be found.  For Instance, customer has an Oracle database called 'test', and we are running a full backup on that database.  Error in rae logs would indicate (note: n=the backup number that failed):

[LOG4] pghelper.c:1861: Executing query with text: SELECT status FROM bp.jobs WHERE job_no = n
[LOG4] pghelper.c:1247: row 0, col 0(status): '512'
[LOG4] pghelper.c:1861: Executing query with text: SELECT status FROM bp.jobs WHERE job_no = n
[LOG4] pghelper.c:1247: row 0, col 0(status): '512'
[LOG4] [methods.c:134] Reading complete. Read 83 bytes
[LOG0] [methods.c:597] Command "(no description)" completed with exit code 1
[LOG0] [methods.c:604] Details:
Method: BP-Proto
Execute: cp -f /usr/bp/mnt/test/oracle_full.n.task /tmp/oracle_full.n.task
Output:
cp: cannot stat `/usr/bp/mnt/test/oracle_full.n.task': No such file or directory
 

RESOLUTION

Perform the following steps to add fsid=0 to the exportfs command in the /usr/bp/bpinit/rae.conf.d/utilities.conf file:

1. Save current file:

cp /usr/bp/bpinit/rae.conf.d/utilities.conf  /usr/bp/bpinit/rae.conf.d/utilities.conf.orig

2. Open /usr/bp/bpinit/rae.conf.d/utilities.conf for editing:

vi /usr/bp/bpinit/rae.conf.d/utilities.conf

3. Search for 'exportfs:

/exportfs

4. Line before change:

<Execute>exportfs -o rw,all_squash,anonuid=0,anongid=0 ${ClientName}:${SharePath}</Execute>

5. Line after change:

<Execute>exportfs -o rw,all_squash,anonuid=0,anongid=0,fsid=0 ${ClientName}:${SharePath}</Execute>

6. Save the file:

:wq!

7. Restart nfs service on Oracle if stale 
8. Run a full backup of the Oracle Database

CAUSE

The issue occurs because we are not cleanly unmounting the NFS share, and causing a stale mount.  The fix will properly mount and unmount the NFS share used for Oracle database backups.