Add files to S3 Bucket using Shell Script-CodePudding

Goal: to push files in gri/ to S3 bucket using SendToS3.sh shell script.

I am following this Tutorial.

SendToS3.sh is in cwd. It needs to fetch all files, that are not in sub-folders, in cwd's gri/.

Terminal:

me@PF2DCSXD:/mnt/c/Users/me/Documents/GitHub/workers-python/workers/data_simulator/data$ ./SendToS3.sh
./SendToS3.sh: line 17: logInfo: command not found
curl: Can't open '/gri/*'!
curl: try 'curl --help' or 'curl --manual' for more information
curl: (26) Failed to open/read local data from file/application
./SendToS3.sh: line 27: logInfo: command not found

SendToS3.sh:

bucket=simulation
files_location=/gri/  # !
now_time=$(date  "%H%M%S")
contentType="application/x-compressed-tar"
dateValue=`date -R`
# your key goes here..
s3Key=  # CENSORED
# your secrets goes here..
s3Secret=  # CENSORED

function pushToS3()
{
  files_path=$1
  for file in $files_path*
  do
    fname=$(basename $file)
    logInfo "Start sending $fname to S3"
    resource="/${bucket}/${now_date}/${fname}_${now_time}"
    stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}"
    signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
    curl -X PUT -T "${file}" \
     -H "Host: ${bucket}.s3.amazonaws.com" \
     -H "Date: ${dateValue}" \
     -H "Content-Type: ${contentType}" \
     -H "Authorization: AWS ${s3Key}:${signature}" \
      https://${bucket}.s3.amazonaws.com/${now_date}/${fname}_${now_time}
     logInfo "$fname has been sent to S3 successfully."
  done
}
pushToS3 $files_location

Please let me know if there is anything else I can add to post.

CodePudding user response：

Your system doesn't have loginfo, so maybe switch that command to echo. For your curl error it could be a file permission errors, try running: chmod -rwx gri.

Alternatively, you could use the aws cli instead, which is much easier to use imo.

CodePudding user response：

The error is at this following line. The folder /gri/ is empty or the user launching the script cannot have access to it.

curl: Can't open '/gri/*'!

Moreover, it seems that your server does not have the executable LogInfo installed, or it is not accessible from your script SendToS3.sh. Verify the installation and add the binary to the PATH env variable.

./SendToS3.sh: line 17: logInfo: command not found

Bonus: instead of using curl, you can use aws-cli which is optimized to interract with aws components. Please find the documentation for s3 here: https://docs.aws.amazon.com/cli/latest/reference/s3/

For example, you can copy a file to w bucket with this command:

aws s3 cp <path_to_file> s3://<bucket_name>/<path>/