I have a python code main.py that runs bash script, the bash script inturn submits a job job.bash and obtains its JOBID using echo $JOBID | awk {'print $4'}. If I run python in the terminal, the bash script works and I am able to obtain and echo the JOBID as follows:

#!/bin/bash 
JOBID=`sbatch ~/job.bash  | tee  output.log`
JOBID=`echo $JOBID | awk {'print $4'}`
echo $JOBID

Running above as part of python works in terminal python main.py, but doing nohup python main.py &, the echo does not print or store JOBID.

Any reason for this?

I am submitting a slurm job hence the JOBID is the pid from slurm

(Update Jul 17) Looks like the issue is with the command sbatch ~/job.bash | tee output.log, it doesnt get submitted using nohup and hence JOBID never gets stored and echo'd.

(Update Jul 18) As per the comments from @pynexj adding set -x in the script results:

nohup: ignoring input and redirecting stderr to stdout
  date
Mon Jul 18 21:46:35  03 2022
   sbatch ~/job.bash
   tee output.log
  JOBID=
   echo
   awk '{print $4}'
  JOBID=
  echo

The issue still persists. It appears that nohup is incompatible with sbatch.

Question: Why should nohup prevent submission of slurm job? Its objective is merely to capture terminate signal?

CodePudding user response：

If this problem only happens with nohup present, you can get the benefits of nohup without actually using it with:

yourscript </dev/null >file.log 2>&1 & disown -h "$!"

This does the following:

• Redirects stdin from /dev/null with </dev/null
• Redirects stdout and stderr to a log file with >file.log 2>&1
• Tells the shell not to forward HUP signals to the background process with disown -h "$!"

...which is everything nohup does.