Loop through a list and call API in python-CodePudding

I have list of author names (more than 1k) and my google api quote limit is 20k I want to pass the author name into the API to get book informations. When I tested my code I got "429 Client Error: Too Many Requests for url..." error, How can I slow down my running time without stopping the application. (I'm using Python in google colab )

author_List = ["J. K. Rowling", "mark twain","Emily Dickinson"] #there are more around 1k - 2k
author_List = author_List.to_dict()

newList = []
for key in author_List:
  author = author_List[key]
  newList.append(searchData(author))


print(newList)



def searchData(authorName):
     try:

        key= "**********************************"
        api = f'https://www.googleapis.com/books/v1/volumes?q={author}&key={key}'
        response = requests.get(api)
        response.raise_for_status()
        print(response)
    except requests.RequestException as e: 
        raise sys.exit(e)

CodePudding user response：

you can import time and then add:

time.sleep(1)

at the end of your for loop, to pause for a second between each iteration.

CodePudding user response：

You could slow down your for loop like this: first, you need to import time

delay = 2
for key in author_List:
   author = author_List[key]
   newList.append(searchData(author))
   time.sleep(delay)

The delay you can set a number how many seconds the loop gets delayed (here it would be 2 seconds)

CodePudding user response：

You probably don't want unconditional delays which will slow down your processing unnecessarily. On the other hand, if you start getting HTTP 429 you can't be sure when or even if the server is going to allow you to continue. So you need a strategy that only introduces delays when/if required but also doesn't get into an infinite loop. Consider this:

import requests
import time

listofauthors = ['Mark Twain', 'Dan Brown', 'William Shakespeare']

with requests.Session() as session:
    for author in listofauthors:
        params = {'q': author}
        delays = 60 # approximately 1 minute total delay time for any given author
        while True:
            try:
                (r := session.get('https://www.googleapis.com/books/v1/volumes', params=params)).raise_for_status()
                print(r.text)
                break # all good :-)
            except Exception as e:
                if r.status_code == 429:
                    if delays <= 0:
                        raise(e) # we've spent too long delaying
                    time.sleep(1)
                    delays -= 1
                else:
                    raise(e) # some other status code