Python requests code worked yesterday but now returns TooManyRedirects: Exceeded 30 redirects-CodePudding

I am trying to get the data from a site using requests using this simple code (running on Google Colab):

import requests, json  
def GetAllStocks():
    url = 'https://iboard.ssi.com.vn/dchart/api/1.1/defaultAllStocks'        
    res = requests.get(url)
    return json.loads(res.text)

This worked well until this morning and I could not figure out why it is returning "TooManyRedirects: Exceeded 30 redirects." error now.

I can still get the data just by browsing the url directly from Google Chrome in Incognito mode so I donot think this is because of the Cookies. I tried passing the whole headers but still it does not work. I tried passing 'allow_redirects=False' and the returned status_code is 302. I am not sure if there is anything I could try as this is so strange to me.

Any guidance is much appreciated. Thank you very much!

CodePudding user response：

You need to send user-agent header to mimic a regular browser behaviour.

import requests, json, random


def GetAllStocks():
    user_agents = [
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36",
        "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0",
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:77.0) Gecko/20190101 Firefox/77.0",
        "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:77.0) Gecko/20100101 Firefox/77.0",
    ]
    headers = {
        "User-Agent": random.choice(user_agents),
        "Accept": "application/json",
    }
    url = "https://iboard.ssi.com.vn/dchart/api/1.1/defaultAllStocks"
    res = requests.get(url, headers=headers)
    return json.loads(res.text)


data = GetAllStocks()
print(data)