Get/fetch a file with a bash script using /dev/tcp over https without using curl, wget, etc-CodePudding

I try to read/fetch this file:

https://blockchain-office.com/file.txt with a bash script over dev/tcp without using curl,wget, etc..

I found this example:

exec 3<>/dev/tcp/www.google.com/80
echo -e "GET / HTTP/1.1\r\nhost: http://www.google.com\r\nConnection: close\r\n\r\n" >&3
cat <&3

I change this to my needs like:

exec 3<>/dev/tcp/www.blockchain-office.com/80
echo -e "GET / HTTP/1.1\r\nhost: http://www.blockchain-office.com\r\nConnection: close\r\n\r\n" >&3
cat <&3

When i try to run i receive:

400 Bad Request
Your browser sent a request that this server could not understand

I think this is because strict ssl/only https connections is on.

So i change it to :

exec 3<>/dev/tcp/www.blockchain-office.com/443
echo -e "GET / HTTP/1.1\r\nhost: https://www.blockchain-office.com\r\nConnection: close\r\n\r\n" >&3
cat <&3

When i try to run i receive:

400 Bad Request
Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.

So i even can't get a normal connection without get the file!

All this post's does not fit, looks like ssl/tls is the problem only http/80 works, if i don't use curl, wget, lynx, openssl, etc...:

how to download a file using just bash and nothing else (no curl, wget, perl, etc.)

Using /dev/tcp instead of wget

How to get a response from any URL?

Read file over HTTP in Shell

I need a solution to get/read/fetch a normal txt file from a domain over https only with /dev/tcp no other tools like curl, and output in my terminal or save in a variable without wget, etc.., is it possible and how, or is it there an other solution over the terminal with the standard terminal utilities?

CodePudding user response：

You can use openssl s_client to perform the equivalent operation but delegate the SSL part:

#!/bin/sh

host='blockchain-office.com'
port=443
path='/file.txt'

crlf="$(printf '\r\n_')"
crlf="${crlf%?}"

{
  printf '%s\r\n' \
    "GET ${path} HTTP/1.1" \
    "host: ${host}" \
    'Connection: close' \
    ''
} |
  openssl s_client -quiet -connect "${host}:${port}" 2 >/dev/null | {

  # Skip headers by reading up until encountering a blank line
  while IFS="${crlf}" read -r line && [ -n "$line" ]; do :; done

  # Output the raw body content
  cat
}

Instead of cat to output the raw body, you may want to check some headers like Content-Type, Content-Transfer-Encoding and even maybe navigate and handle recursive MIME chunks, then decode the raw content to something.

CodePudding user response：

After all the comments and research, the answer is no, we can't get/fetch files using only the standard tools with the shell like /dev/tcp because we can't handle ssl/tls without handle the complete handshake.

It is only possbile with the http/80.

i dont think bash's /dev/tcp supports ssl/tls

If you use /dev/tcp for a http/https connection you have to manage the complete handshake including ssl/tls, http headers, chunks and more. Or you use curl/wget that manage it for you.

then shell is the wrong tool because it is not capable of performing any of the SSL handshake without using external resources/commands. Now relieve and use what you want and can from what I show you here as the cleanest and most portable POSIX-shell grammar implementation of a minimal HTTP session through SSL. And then maybe it is time to consider alternative options (not using HTTPS, using languages with built-in or standard library SSL support).

We will use curl, wget and openssl on seperate docker containers now.

I think there are still some requirements in the future to see if we keep only one of them or all of them.

We will use the script from @Léa Gris in a docker container too.