I have a python code which is requesting a Rest API. The API has more than 5000 pages so i tried to request it but always i am getting error at 2000th request.

The error is: "df = pd.json_normalize(json_data["items"]) KeyError: 'items'" How can i solve this problem ? P.S. In locally, the code is working clearly.

CodePudding user response：

I found a solution for this problem. Like @elad said, this is not a airflow error. Due to airflow web service and scheduler the system working a little bit slower. So, my token expired while airflow tasks running. I reorganized my code, I generated token in loop with specific conditions such as try-except. Thanks for everything !