JSON Response is being truncated-CodePudding

Hi

I'm trying to scrap some data from a website the data is displayed in a chart ( the data is currency prices over years)

I was able to get the XHR Request and the API link for the JSON data but when I open the response in the ( network tab or in a new tab ) the data is not completely displayed but in the chart the data is represented.

the api link

searched about the problem and I found this post which says that the dev-tools truncates long network response I tried the solution but the same problem is still happening .
tried to use wget to download them but it didn't help same issue appeared.

I'm opening the link in a separate tab on Brave browser (also tried Firefox) I don't know what's the problem Can you please help me ?!

CodePudding user response：

You can scrape that API endpoint (with Python) like below:

import requests
import pandas as pd

pd.set_option('display.max_columns', None, 'display.max_colwidth', None)

headers = {
    'content-type': 'application/json',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'
}

r = requests.get('https://sy-exchange-rates-iwi3arxhhq-uc.a.run.app/api/rates?from=Thu, 31 Mar 2011 21:00:00 GMT&to=Mon, 23 Jan 2023 02:03:23 GMT&name=USD&source=liranews.info,sp-today.com,dei-sy.com&city=damascus', headers=headers)

df = pd.json_normalize(r.json())
print(df)

Result in terminal:

    timestamp   source  city    name    buy sell
0   2021-01-03T07:00:00Z    sp-today.com    damascus    USD 2855    2880
1   2021-01-03T07:00:00Z    liranews.info   damascus    USD 2855    2880
2   2021-01-03T07:00:00Z    dei-sy.com  damascus    USD 2845    2855
3   2021-01-03T08:00:00Z    sp-today.com    damascus    USD 2855    2880
4   2021-01-03T08:00:00Z    liranews.info   damascus    USD 2855    2880
... ... ... ... ... ... ...
50772   2023-01-22T22:00:00Z    liranews.info   damascus    USD 6625    6685
50773   2023-01-22T23:00:00Z    sp-today.com    damascus    USD 6625    6685
50774   2023-01-22T23:00:00Z    liranews.info   damascus    USD 6625    6685
50775   2023-01-23T01:00:00Z    sp-today.com    damascus    USD 6625    6685
50776   2023-01-23T01:00:00Z    liranews.info   damascus    USD 6625    6685
50777 rows × 6 columns