Convert log files to base64 and upload it using Curl to Github-CodePudding

I am trying to upload a file to Github using its API.

Following code works, but only with smaller size content which is approx less than 1MB.

tar -czvf logs.tar.gz a.log b.log
base64_logs=$(base64 logs.tar.gz | tr -d \\n)

content_response=$(curl \
  -X PUT \
  -u :"$GIT_TOKEN" \
  -H "Accept: application/vnd.github.v3 json" \
  "$content_url" \
  -d '{"message": "Log files", "content": "'"$base64_logs"'"}')

For content that is a bit large, I get the following error:

/usr/bin/curl: Argument list too long

Now, there is already a question on SO about this error message, and it says that to upload a file directly. See here: curl: argument list too long

When I try this, I get a problem parsing JSON error message.

tar -czvf logs.tar.gz a.log b.log
base64_logs=$( base64 logs.tar.gz | tr -d \\ ) > base64_logs.txt

content_response=$(curl \
  -X PUT \
  -u :"$GIT_TOKEN" \
  -H "Accept: application/vnd.github.v3 json" \
  "$content_url" \
  -d '{"message": "Log files", "content": @base64_logs.txt}')

Can anyone point me out where I am making mistake here? Thanks!

CodePudding user response：

Use the base64 command rather than the @base64 filter from jq, because the later can only encode textual data and not binary data as from a .gz archive.

Pipe the base64 stream to jq to format it into a JSON data stream.

Curl will read the JSON data stream and send it.

# use base64 to encode binary data and -w0 all in one-line stream
base64 --wrap=0 logs.tar.gz |

# JSON format from raw input
jq \
  --raw-input \
  --compact-output \
  '{"message": "Log files", "content": . }' |

# Pipe JSON to curl
curl \
  --request PUT \
  --user ":$GIT_TOKEN" \
  --header "Accept: application/vnd.github.v3 json" \
  --header 'Content-Type: application/json' \
  --data-binary @- \
  --url "$content_url"