Stale Task Found in Job Manager

SUMMARY

Backup catalog delete stuck for hours and not moving

ISSUE

Jobs in the job manager tab are stuck or not making progress.

RESOLUTION

Make sure the task is not running on the engine, login to the specific engine and run:

sudo phdvb task list

sudo phdvb task destroy [list of uuids]

psql service=client

update client_job
set job_status = 3,
job_state = 7,
timestamp = extract(epoch from now()),
start_date = case when start_date is null then now() else start_date end,
finish_date = now()
where job_state < 6;


update client_jobinventory
set job_status = 3,
job_state = 7,
timestamp = extract(epoch from now()),
start_date = case when start_date is null then now() else start_date end,
finish_date = now()
where job_state < 6;

This will clear ALL jobs and mark them as completed and cancelled. You may also want to restart the management and presentation appliances.

CAUSE

The most likely cause is communication related with RabbitMQ. If RabbitMQ stops responding to messages then this type of thing will happen. Not only system jobs but also backups and replication.