Different response content when on docker-CodePudding

I am making a request to get a download link through the following request:

headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0',
'Accept': '*/*',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'X-Requested-With': 'XMLHttpRequest',
'Origin': 'https://x2download.com',
'DNT': '1',
'Connection': 'keep-alive',
'Referer': 'https://x2download.com/fr54',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'TE': 'trailers',
}

video= 'https://www.youtube.com/watch?v=kpz8lpoLvrA'

data = f'q={video}&vt=home'

response = requests.post('https://x2download.com/api/ajaxSearch', headers=headers, data= data)
print(response.content)

From my windows laptop and my ubuntu server I am getting the following content:

b'{"vid":"kpz8lpoLvrA","title":"Interstellar Main Theme - Hans Zimmer","fn":"X2Download.com-Interstellar Main Theme - Hans Zimmer","a":"Aura Music","t":244,"links":{"ogg":{"1":{"f":"ogg","k":"128","q":"128kbps","size":"4.02 MB","key":"128kbps","selected":null}},"mp3":{"2":{"f":"mp3","k":"128","q":"128kbps","size":"4.02 MB","key":"128kbps","selected":null}},"mp4":{"3":{"f":"mp4","k":"1080p","q":"1080p","size":"16.87 MB","key":"1080","selected":""},"4":{"f":"mp4","k":"720p","q":"720p","size":"12.48 MB","key":"720","selected":"selected"},"5":{"f":"mp4","k":"480p","q":"480p","size":"4.21 MB","key":"480","selected":""},"6":{"f":"mp4","k":"360p","q":"360p","size":"7.39 MB","key":"360","selected":""},"7":{"f":"mp4","k":"240p","q":"240p","size":"7.19 MB","key":"240","selected":""},"8":{"f":"mp4","k":"144p","q":"144p","size":"817.20 KB","key":"144","selected":""}},"3gp":{"9":{"f":"3gp","k":"144p","q":"144p","size":"817.20 KB","key":"144","selected":null}}},"token":"1cc3a03822a2582bcb47b70da2012cdf43fc66d899e6f0a5d14064c7dcec1154","timeExpires":"1660554472","status":"ok","p":"convert","mess":""}'

But when I try on a heroku app, AWS lambda or even a docker container, I am getting this:

b'\x83%\x02\x00\xc4/\x9d\xf9U\xcb\xbcZf\x14\x96\xb4\x9d\xfdC\xee~\xeet\xb0\x17%Av\xe4\x7fo\xf9\xb6\xd2Y\xc6\x17\x0eh\xe4\xff\x00\x0c,\xe2\xcbf\xd1I\xf1\xfd\xbc\x17 \xa9E\x10q\xc6i\xbbL\x13\xc9ob\xae\xce\x9b\xe3\x15\xdb\xa5\x03\xe36\xbc\xd4a\xe8\xbfo\xd3=\x14\x96\xcb\x12\x04\x8c/i\x91i^$\x04?e?\xfc%e?\xcf\x12%\xbb\xcb>\xfb4g\xff/1\xca\x04\x85c\x02\xe3\xaf}\xdf?\xa6\xd0\xfb.o\xfbx\xf7\xe5\xfe\xfd\x1e\x8c\xfbu\xf2\xd9\x8fu\xbe\xb4PX\xc0\x96H!\\\xd2m\x06\xbf\xa2?\x9d\xc0\xaf0\xe0W\x1c\xc1{\xc7\n70\x8c\xad\xa10\xaa\'\xdc\x0e\xc3\x0c\x85\xf9\xf2"P\xaem\xf6\xe3\x01\n7y>t]\x82\xb4\x8bt\xe0\xb4\x86\xb0\xef\nqp\xe0W\xd8G>\x15\x07W\x04\x85\x9cO\x99\xae\xf5\xd0Q\xe3\x1a\x9b2\xaf\xabV\xd7\xb5\x1e\x13\x80]\x81\x8c\xd3\xca\x12%M\xcb\xe6T\xbb*K\xae\x03\xbbB\xe1\x9e\xa3\x88\x8b\xc4\x0e\xe5\xd6\xb0\x92\x98PjW\xa2DS\xab\xca]creB\xa5-!\x18Qc\xb2\x94P\x07\x86\x08\x81\xd8\xadM\x95[\x9d}K\x92\x93(\xce\xb3]\xc1\x9d\x06\xf0 \x9a\5\xecN\x83=\xfc\xae\x9f\xd9\x15\x96\xfe&\t\x8cFW\xe2\xa5\xb2\xae\xf6dj\xd3\xd9\xb6\xf2\x9a\xa8p\xb5o\\xa3\xa5\xf4E\xeb\x9d\xf1\x9d\x18{(\x8co}\xeb;[jr\xf5\xd1\xda\xee\x00\x85\xe5\x12\xe5\xc3\xd3p\x99d\x06\xc3\x94\xa5n\xc8\x14M\t\x85y\xf1\xcb:\x83\xd1\xdf\xa00\x80\xd1\xf6i\x93i\x81B\x94y\x06\x03\xbb\x01\x03'

I tried:

Modifying the parameters of the platform (LANG, C-LANG, etc)
Decoding in any way I found
Putting the same python libraries to the same versions
with urllib
all other UnicodeDecodeError related solutions

The result stays the same. Any idea on how to change the received result or to decrypt it is welcome

CodePudding user response：

Your headers have 'Accept-Encoding': 'gzip, deflate, br' which means you are telling the server that your request accepts compressed responses.

The reason this is working on your local but not on docker is that your local environment has brotli library installed but your docker container might not.

In this case, the server is sending a broti compressed response which your docker container cannot decode.

You can simply install brotli in your docker image by including this in your Dockerfile

RUN pip3 install brotli

or adding brotli in your requirements.txt. The same goes for your lambda.

Alternatively, you can request the server to not send brotli compressed responses by using 'Accept-Encoding': 'gzip, deflate'