Bash code runs when directly called, but not when called and fed vars via a Makefile. But other bash-CodePudding

I had a makefile I used to create jobs in one step then submit them to a cluster in another, then finally merge them. This worked well and I had no issues with it, I was using a python script to break csv files down into smaller ones in the create-jobs step however, and it took longer than I'd like so I decided to add a step to do this with bash code instead, thinking it would be faster.

I laid the new step out in the Makefile in the exact same way as the others, and created a small bash script that would accept the name of the csv, how many parts to split it into and eventually I was going to add a variable for the location to place the output csves but I tried testing it with it just sticking them in the same directory as itself first. I first defined the variables that would later be passed on in the bash file so that I could check it worked by itself, it does. Those variables are commented out now but this is the bash file in question:

#!/bin/bash

#INPUT_DATA="/scratch/cb27g11/Looping_Bash/missing.csv"
#nJobs=2

tail -n  2 ${INPUT_DATA} | split -l ${nJobs} - split_
for file in split_*
do
    head -n 1 ${INPUT_DATA} > tmp_file
    cat $file >> tmp_file
    mv -f tmp_file ../$file
done

Nothing super complicated. The Makefile looks like this:

########################
### --- Split jobs - ###
########################

SPLIT_JOB_NAME        = "Name_for_job"
SPLIT_JOB_INPUT_DATA  = "/scratch/cb27g11/Looping_Bash/missing.csv"
SPLIT_JOB_HEADER      = "header/default.header"
SPLIT_JOB_nJobs       = 2
#######################

DECLARE_ROOT_DIR = ROOT_DIR="${THDM_T3PS_SCANNER_DIR}/job_submission/MadGraph/"

VAR_CREATE_JOB    = $(shell echo '$(.VARIABLES)' |  awk -v RS=' ' '/CREATE_JOB_/' | sed 's/CREATE_JOB_//g' )
EXPORT_CREATE_JOB = $(foreach v,$(VAR_CREATE_JOB),$(v)="$(CREATE_JOB_$(v))") $(DECLARE_ROOT_DIR)

VAR_SUBMIT_JOB    = $(shell echo '$(.VARIABLES)' |  awk -v RS=' ' '/SUBMIT_JOB_/' | sed 's/SUBMIT_JOB_//g' )
EXPORT_SUBMIT_JOB = $(foreach v,$(VAR_SUBMIT_JOB),$(v)="$(SUBMIT_JOB_$(v))") $(DECLARE_ROOT_DIR)

VAR_MERGE_JOB    = $(shell echo '$(.VARIABLES)' |  awk -v RS=' ' '/MERGE_JOB_/' | sed 's/MERGE_JOB_//g' )
EXPORT_MERGE_JOB = $(foreach v,$(VAR_MERGE_JOB),$(v)="$(MERGE_JOB_$(v))") $(DECLARE_ROOT_DIR)

VAR_SPLIT_JOB    = $(shell echo '$(.VARIABLES)' |  awk -v RS=' ' '/SPLIT_JOB_/' | sed 's/SPLIT_JOB_//g' )
EXPORT_SPLIT_JOB = $(foreach v,$(VAR_SPLIT_JOB),$(v)="$(SPLIT_JOB_$(v))") $(DECLARE_ROOT_DIR)

#########################################################

create-jobs:
         @$(EXPORT_CREATE_JOB) ./utils/new_create-jobs.sh

submit-jobs:
         @$(EXPORT_SUBMIT_JOB) ./utils/new_submit-jobs.sh

merge-jobs:
         @$(EXPORT_MERGE_JOB) ./utils/merge-jobs.sh

split-jobs:
         @$(EXPORT_SPLIT_JOB) ./utils/split-jobs.sh

I'm not currently using all of the variables set-up for split-jobs here but surely that doesn't matter? The other parts, create-jobs, submit-jobs and merge-jobs work just fine and I can't see that I've made a mistake in how I've set up split-jobs compared to them. So I guess there's something I'm missing about the bash file itself?

When I try to run make split-jobs I get this:

(THDM) [cb27g11@cyan53 MadGraph]$ make split-jobs
/bin/sh: ./utils/split-jobs.sh: Permission denied
make: *** [split-jobs] Error 126

Since this is on a shared cluster I don't have sudo rights or anything like that, could this be related to that somehow? Am I calling something wrong? Any help or suggestions would be much appreciated!

CodePudding user response：

I can't really understand what all the shell commands are trying to do, but I'm completely confident that whatever they are doing there's a much simpler and more straightforward way to do it.

However, that doesn't have anything to do with your error, which is:

/bin/sh: ./utils/split-jobs.sh: Permission denied

Did you remember to give your shell script execute permissions:

chmod  x ./utils/split-jobs.sh

? Also, please remove the @ from your makefile lines until AFTER your makefile is completely working, and show us the command line that make invoked (maybe once you see it, you will be able to figure out the problem for yourself). Using @ is like trying to debug your program while sending all the output to /dev/null.