i need help with api(json) and store as dataframe. From json, i want only data in 'projects'. I have this function
def get():
url_address="https://xx.com/projects.json?limit=100&offset=" str('0')
r = requests.request("GET",'https://xx.com/projects.json?limit=1',verify=False, auth=('xy','xy')).json()
total_record=r['total_count']
print("Total record: " str(total_record))
all_items = pd.DataFrame()
for offset in range(0, total_record,100):
url = "https://xx.com/projects.json?limit=100&offset=" str(offset)
response = requests.get(url=url,verify=False, auth=('xy','xy'))
data=response.json()
df = pd.DataFrame(data['projects'])
all_items.append(df)
all_items.to_csv("text.csv", encoding='utf-8', index=False)
print(get())
But I have a problem with the append command because the resulting Dataframe is always empty. When I try it manually for one iteration without append, I get results. But this append probably doesn't do anything to me. Can anyone advise? Do I need a better way to do this? BTW: I have to do this like a cycle, because the API returns a maximum of 100 results.
CodePudding user response:
Use a list instead of an empty dataframe then concatenate all items into a dataframe:
all_items = [] # <- HERE
for offset in range(0, total_record,100):
url = "https://xx.com/projects.json?limit=100&offset=" str(offset)
response = requests.get(url=url,verify=False, auth=('xy','xy'))
data=response.json()
df = pd.DataFrame(data['projects'])
all_items.append(df)
all_items = pd.concat(all_items) # <- HERE
all_items.to_csv("text.csv", encoding='utf-8', index=False)