Download multiple files from URL to a particular path in linux using curl command (Shell Script)-CodePudding

URL return list of files separated by comma example

URL : http://sometimesdownload.com/list_of_files

To get list of files

curl -0 http://sometimesdownload.com/list_of_files

Output of above curl command/List returned :

[" file1.csv","file2.csv","file3.csv"...."filen.csv"]

How to download all files in a single run to a specified path in /tmp/downloaded_files directory in linux and delete the file after downloading. File name of files should be same as original files.

CURL GET to get the files, Download the files and then delete the files after downloading using CURL DELETE.

CodePudding user response：

You can do this in Java without curl, but if you want to use curl, you're going to want to use either Runnable.exec or ProcessBuilder. See pass multiple parameters to ProcessBuilder with a space for an example

CodePudding user response：

This could be done as something like

for i in $(curl http://sometimesdownload.com/list_of_files | jq -c '.[]'); then
    # GET file
    curl -O --output-dir /tmp/downloaded_files http://sometimesdownload.com/"$i"

    # DELETE file
    # However this is mostly not working straightforward and need tuning 
    # as no enough information here about how the files are hosted and 
    # configured by the remote server 
    curl -X DELETE http://sometimesdownload.com/"$i"
done

-c is the compact output option in jq to get the plan text output from a json array.

$ echo '["file1.csv","file2.csv","file3.csv"]' | jq -c '.[]'
"file1.csv"
"file2.csv"
"file3.csv"