KeyError: 'users' when using tweepy to scrape the twitter data, how can I bypass such erro-CodePudding

I am going to scrape the information from twitter using the academic account through the tweepy package, it works fine for some months but in particular months it showed "KeyError: 'users'". How can I bypass the error?

Here is the code that I execute;

tweets = []

for i,j in zip(start_time,end_time):
    print(i)
    print(j)
    for response in tweepy.Paginator(client.search_all_tweets, 
                                     query = 'ไทรอยด์ -is:retweet lang:th',
                                     user_fields = ['username', 'public_metrics', 'description', 'location'],
                                     tweet_fields = ['created_at', 'geo', 'public_metrics', 'text'],
                                     expansions = ['author_id', 'geo.place_id'],
                                     start_time = i,
                                     end_time = j):
                                     
        time.sleep(1)
        tweets.append(response)
    result = []
    user_dict = {}
    
    for response in tweets:
        
        for user in response.includes['users']:
            user_dict[user.id] = {'username': user.username, 
                                  'followers': user.public_metrics['followers_count'],
                                  'tweets': user.public_metrics['tweet_count'],
                                  'description': user.description,
                                  'location': user.location
                                 }

Results:

2020-08-01T00:00:00Z
2020-08-31T23:59:59Z
Rate limit exceeded. Sleeping for 30 seconds.
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_13308/265973239.py in <module>
     23     for response in tweets:
     24         # Take all of the users, and put them into a dictionary of dictionaries with the info we want to keep
---> 25         for user in response.includes['users']:
     26             user_dict[user.id] = {'username': user.username, 
     27                                   'followers': user.public_metrics['followers_count'],

KeyError: 'users'

CodePudding user response：

You can make your code use an empty default value so that it doesn't fail if the users field doesn't exist:

for user in response.includes.get('users', ''):
     # do something with user

This fixes only the symptom, however. To find out the reason why the field doesn't exist, you could try to check for the keys before using them (e.g. if 'users' in response.includes) and print out the full response data to get more information. One hint is the rate limiting message you have received.