Download file from curl command into python script don't work-CodePudding

I would like to send a complex curl command into python (3.9.7) script to download an output file .tif I use a TimeNum variable because i want to download different files and clip the .tif in a personal area of intrest (I want to use a for loop).

The curl command is something like this:

curl --data '{"productType":"VMI","productDate":"1648710600000"}' -H "Content-Type: application/json" -X POST https://example.it/wide/product/downloadProduct --output hrd2.tif

I try different solutions:

import shlex
import subprocess

TimeNum=1648710600000

cmd ='''curl --data \'{"productType":"VMI","productDate":"%s"}\' -H "Content-Type: application/json" -X POST https://example.it/wide/product/downloadProduct --output %s.tif''' % (TimeNum,TimeNum)
args = shlex.split(cmd)
process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = process.communicate()

The download work but the output takes up 1KB instead of about 11MB, and an error occur when i try to open it in Windows.

I write a txt file with a list of curl command line by line and then:

file = open('CURL_LIST.txt', 'r')
lines = file.readlines()
for line in enumerate(lines):
    os.system(line.strip())

but don't work fine and i have the same output of case 1 above. Now i try to use urllib.request but i'm not able to use it very well.

Someone have a suggestion? Thanks in advance for your help

CodePudding user response：

Important info
There is a self-signed certificate on that server so you get a warning (and this is also the reason why you get small files).
In the example below I disabled certificate checking but this is DANGEROUS and you should use it only if you understood the risks and it is ok for you anyway (e.g. you are the owner of example.it).
Given the nature of example.it I'd assume that you are just using it only for learning purpose but please be careful and read more about the risks of self-signed certificates anyway.
The proper solution from a risk / security standpoint for a similar problem is NOT connecting to such a server.

Once this is clear, just for the sake of testing / learning I would suggest using Python's requests library (please note the verify=False to disable certificate check):

import requests

time_num = 1648710600000

headers = {
    # Already added when you pass json=
    # 'Content-Type': 'application/json',
}

json_data = {
    'productType': 'VMI',
    'productDate': time_num,
}

response = requests.post('https://example.it/wide/product/downloadProduct', headers=headers, json=json_data, verify=False)

with open(time_num   '.tif', 'wb') as f:
    f.write(response.content)

If you prefer the approach you posted it is possible to disable cert check also in curl (-k option):

import shlex
import subprocess

TimeNum=1648710600000

cmd ='''curl -k --data \'{"productType":"VMI","productDate":"%s"}\' -H "Content-Type: application/json" -X POST https://example.it/wide/product/downloadProduct --output %s.tif''' % (TimeNum,TimeNum)
args = shlex.split(cmd)
process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = process.communicate()