Error: Time out while performing API call in Python-CodePudding

I have a list (lst) which is a list of list. There are 19 elements in this list and each element has ~2500 strings.

lst

[['A', 'B', 'C',...]['E', 'F', 'G',....][.....]]

I am using these strings (A,B....) to call an API endpoint ('q':element). However after ~1800 strings, I am getting a time out.

I am running following lines of code.

def get_val(element):
    url = 'https://www.xxxx/yyy/api/search'
    headers = {'Content-Type': 'application/json'}
    param = {'q': element, 'page' : 500}
    
    try:
        response = requests.get(url, headers = headers, params = param, timeout=(3.05, 27))
        docs = response.json()['response']['docs']
        
        for result in docs:
            file.write("%s\t%s\n" % (element,result['short_form']))
    
    except Timeout:
        print('Timeout has been raised.')

#loop through elements of list
for i in lst:
    for element in i:
        get_val(element)

How can I modify my code to avoid this time out?

CodePudding user response：

One reason for this timeout could be a protection against mass requests, that means, that there are too many requests in a short time.

To overcome this problem a short pause could be added after for example every 100 requests. However this is a try and error approach but it could work. Worst case would be to add a delay after every request.

import time

time.sleep(0.5)

The parameter is added in seconds so 0.5 sec for example.