Run multiple Google Apps Script clasp commands in parallel using a Bash script-CodePudding

I have several hundred Google Apps Script projects and have a variety of Bash scripts for managing the projects using the clasp tool (a Node.js app). Many of the scripts require using clasp pull to first pull the projects locally before taking some actions on the local files, so I have a script which loops through local clasp project folders and runs clasp pull on each. The loop iterates through directories sequentially so if it takes 3-4 seconds to pull a project, it ends up taking 5-6 minutes to run it per 100 projects.

My goal is to be able to run the clasp pull commands in parallel so that they all start at the same time, and to be able to know which projects were successfully pulled vs which projects failed to be pulled.

Given a directory structure like this:

├── project-1
│   ├── .clasp.json
│   ├── .claspignore
│   ├── _main.js
│   └── appsscript.json
├── project-2
│   ├── .clasp.json
│   ├── .claspignore
│   ├── _main.js
│   └── appsscript.json
├── project-3
│   ├── .clasp.json
│   ├── .claspignore
│   ├── _main.js
│   └── appsscript.json
└── pull_all.sh

And this pull_all.sh Bash script:

#!/bin/bash

# use Node 14.17.5 to prevent "Error: Looks like you are offline." errors
# (see https://github.com/google/clasp/issues/872)
[ -s "/usr/local/opt/nvm/nvm.sh" ] && . "/usr/local/opt/nvm/nvm.sh"
nvm install 14.17.5
nvm use 14.17.5

find . -name '.clasp.json' | 
while read file; do
    (
        cd "$(dirname "$file")"
        project_dir_name="$(basename "$(pwd)")"
        echo "Pulling project ($project_dir_name)"
        clasp pull
    ) &
done

When running this script it outputs the line for "Pulling project" for each directory, then gives a shell prompt, implying that the script has finished executing. But then without the user doing anything, 3-4 seconds later it shows the output of all the clasp pull commands (apparently running in parallel because some of the output of the commands are out of order/overlapping), then hangs, and does not give a new shell prompt. At this point I have to press ctrl c to terminate the script.

The complete output ends up looking like this:

$ ./pull_all.sh
v14.17.5 is already installed.
Now using node v14.17.5 (npm v6.14.14)
Now using node v14.17.5 (npm v6.14.14)
Pulling project (project-3)
Pulling project (project-2)
Pulling project (project-1)
$
Cloned 2 files.
⠙ Pulling files…└─ appsscript.json
└─ _main.js
Cloned 2 files.
└─ _main.js
└─ appsscript.json
Cloned 2 files.
└─ _main.js

To force one of the scripts to fail, I can change the scriptId to an invalid script ID in any of the .clasp.json files. In this case I do see the expected output of:

Could not find script.
Did you provide the correct scriptId?
Are you logged in to the correct account with the script?

... but it's still mixed in with the rest of the output and it's not clear which project that came from.

How can I make it so that:

The script does not cause a new shell prompt to appear during the execution of the script.
The script outputs a line indicating the success or failure of each clasp pull operation, referenced by the directory name of the project (where the .clasp.json file was found).
Bonus: suppress the output of clasp pull so the script only shows the success or failure result of each project (referenced by the directory name).

Note: I've mentioned clasp pull as an example command, but a valid solution would allow me to run any clasp command as a background process in a bash while loop, including, but not limited to clasp push, clasp deploy, etc.

CodePudding user response：

The script does not cause a new shell prompt to appear during the execution of the script.

The new shell prompt is occurring because you are creating a new subshell in the while loop (for further guidance on how subshells work in bash, reference this page from tldp.org: link). To prevent this from occurring, call the command directly without placing them within parentheses.

The script outputs a line indicating the success or failure of each clasp pull operation, referenced by the directory name of the project (where the .clasp.json file was found).

You can generally catch if a command fails by adding an || after the command (e.g. grep "foobar" file.txt || echo "Error: 'foobar' not found in file.txt"). You could also put the command in an if/else and echo the appropriate status message for each.

Bonus: suppress the output of clasp pull so the script only shows the success or failure result of each project (referenced by the directory name).

Note: This response uses the aforementioned solution from the second question. You could create 2 arrays—1 for success and 1 for failures, and then inside of the if/else statements, add the current iteration element to the correct array.

Feel free to ask for clarification if any part of the above was not clear!

CodePudding user response：

If I understood your question right, you want to know which projects weren't pulled. Clasp doesn't offer that option as you can see on the docs, but you can efficiently work around that. You only need to update that bash script to compare the project iteration file with the final directory tree. Every project found in your original file but not in the directory tree represents a faulty id.