I scrape daily courses from the website tatrabanka.sk. They recently updated the website and my script returns the following error. How can I handle that? Do I need to use bs4 and classically download the entire website? Thanks for any advice.
import pandas as pd
import numpy as np
from datetime import datetime
tmp_url = "https://www.tatrabanka.sk/rest/tatra/exchange/list/20.11.2022-00:00"
pd.read_json(tmp_url)
Output:
...
ValueError: All arrays must be of the same length
Full script:
dr = pd.date_range(start = datetime.today().strftime('%m/%d/%Y'),
end = datetime.today().strftime('%m/%d/%Y'),
freq = '1440min')
df_date = pd.to_datetime(dr, format = '%Y-%m-%d').strftime('%d.%m.%Y')
df_date = df_date '-00:00'
url_list = 'https://www.tatrabanka.sk/rest/tatra/exchange/list/' df_date
smbl = ["USD", "PLN", "HUF", "CZK", "HRK", "RON"]
data = []
tmp_url = "https://www.tatrabanka.sk/rest/tatra/exchange/list/20.11.2022-00:00"
pd.read_json(tmp_url)
for urls in url_list:
print(urls)
dft = pd.read_json(urls)
dft['DateReal'] = urls[51:61]
data.append(dft.loc[dft["feCycd"].isin(smbl)])
out_df = pd.concat(data)
CodePudding user response：
To load the Json into a dataframe try:
import requests
import pandas as pd
tmp_url = "https://www.tatrabanka.sk/rest/tatra/exchange/list/20.11.2022-00:00"
data = requests.get(tmp_url).json()
df = pd.DataFrame(data["rates"])
print(df)
Prints:
id feDate feCntr feCycd feAmnt feDnrt feDprt feDsrt feVnrt feVprt feVsrt feLccy rateListId status formattedDate translatedCountry translatedCode flag
0 198 1668985200000 Austrália AUD 1 1.5868 1.5094 1.5481 0.0000 0.0000 0.0000 EUR 10212 1 21.11.2022 00:00 Austrália AUD /templates/tatrabanka/assets/img/flags/au.svg
1 195 1668985200000 Česká republika CZK 1 24.9790 23.7610 24.3700 25.2230 23.5170 24.3700 EUR 10212 1 21.11.2022 00:00 Česká republika CZK /templates/tatrabanka/assets/img/flags/cz.svg
2 192 1668985200000 Chorvátsko HRK 1 7.7152 7.3388 7.5270 7.9034 7.1506 7.5270 EUR 10212 0 21.11.2022 00:00 Chorvátsko HRK /templates/tatrabanka/assets/img/flags/hr.svg
3 194 1668985200000 Dánsko DKK 1 7.6243 7.2523 7.4383 0.0000 0.0000 0.0000 EUR 10212 1 21.11.2022 00:00 Dánsko DKK /templates/tatrabanka/assets/img/flags/dk.svg
4 190 1668985200000 Japonsko JPY 1 148.2400 141.0000 144.6200 0.0000 0.0000 0.0000 EUR 10212 0 21.11.2022 00:00 Japonsko JPY /templates/tatrabanka/assets/img/flags/jp.svg
5 181 1668985200000 Južná Afrika ZAR 1 18.3326 17.4384 17.8855 0.0000 0.0000 0.0000 EUR 10212 0 21.11.2022 00:00 Južná Afrika ZAR /templates/tatrabanka/assets/img/flags/za.svg
6 197 1668985200000 Kanada CAD 1 1.4193 1.3501 1.3847 0.0000 0.0000 0.0000 EUR 10212 1 21.11.2022 00:00 Kanada CAD /templates/tatrabanka/assets/img/flags/ca.svg
7 191 1668985200000 Maďarsko HUF 1 417.0600 396.7200 406.8900 427.2300 386.5500 406.8900 EUR 10212 0 21.11.2022 00:00 Maďarsko HUF /templates/tatrabanka/assets/img/flags/hu.svg
8 189 1668985200000 Nórsko NOK 1 10.7917 10.2653 10.5285 0.0000 0.0000 0.0000 EUR 10212 1 21.11.2022 00:00 Nórsko NOK /templates/tatrabanka/assets/img/flags/no.svg
9 188 1668985200000 Poľsko PLN 1 4.8218 4.5866 4.7042 4.9394 4.4690 4.7042 EUR 10212 1 21.11.2022 00:00 Poľsko PLN /templates/tatrabanka/assets/img/flags/pl.svg
10 187 1668985200000 Rumunsko RON 1 5.0632 4.8162 4.9397 0.0000 0.0000 0.0000 EUR 10212 1 21.11.2022 00:00 Rumunsko RON /templates/tatrabanka/assets/img/flags/ro.svg
11 196 1668985200000 Švajčiarsko CHF 1 1.0108 0.9614 0.9861 1.0206 0.9516 0.9861 EUR 10212 1 21.11.2022 00:00 Švajčiarsko CHF /templates/tatrabanka/assets/img/flags/ch.svg
12 185 1668985200000 Švédsko SEK 1 11.2559 10.7069 10.9814 0.0000 0.0000 0.0000 EUR 10212 1 21.11.2022 00:00 Švédsko SEK /templates/tatrabanka/assets/img/flags/se.svg
13 184 1668985200000 Turecko TRY 1 20.2097 18.2849 19.2473 0.0000 0.0000 0.0000 EUR 10212 0 21.11.2022 00:00 Turecko TRY /templates/tatrabanka/assets/img/flags/tr.svg
14 182 1668985200000 USA USD 1 1.0593 1.0025 1.0335 1.0697 0.9973 1.0335 EUR 10212 0 21.11.2022 00:00 USA USD /templates/tatrabanka/assets/img/flags/us.svg
15 193 1668985200000 V. Británia GBP 1 0.8910 0.8476 0.8693 0.9084 0.8302 0.8693 EUR 10212 0 21.11.2022 00:00 Veľká Británia GBP /templates/tatrabanka/assets/img/flags/gb.svg
To filter the dataframe:
smbl = ["USD", "PLN", "HUF", "CZK", "HRK", "RON"]
df = df[df["feCycd"].isin(smbl)]
print(df)
Prints:
id feDate feCntr feCycd feAmnt feDnrt feDprt feDsrt feVnrt feVprt feVsrt feLccy rateListId status formattedDate translatedCountry translatedCode flag
1 195 1668985200000 Česká republika CZK 1 24.9790 23.7610 24.3700 25.2230 23.5170 24.3700 EUR 10212 1 21.11.2022 00:00 Česká republika CZK /templates/tatrabanka/assets/img/flags/cz.svg
2 192 1668985200000 Chorvátsko HRK 1 7.7152 7.3388 7.5270 7.9034 7.1506 7.5270 EUR 10212 0 21.11.2022 00:00 Chorvátsko HRK /templates/tatrabanka/assets/img/flags/hr.svg
7 191 1668985200000 Maďarsko HUF 1 417.0600 396.7200 406.8900 427.2300 386.5500 406.8900 EUR 10212 0 21.11.2022 00:00 Maďarsko HUF /templates/tatrabanka/assets/img/flags/hu.svg
9 188 1668985200000 Poľsko PLN 1 4.8218 4.5866 4.7042 4.9394 4.4690 4.7042 EUR 10212 1 21.11.2022 00:00 Poľsko PLN /templates/tatrabanka/assets/img/flags/pl.svg
10 187 1668985200000 Rumunsko RON 1 5.0632 4.8162 4.9397 0.0000 0.0000 0.0000 EUR 10212 1 21.11.2022 00:00 Rumunsko RON /templates/tatrabanka/assets/img/flags/ro.svg
14 182 1668985200000 USA USD 1 1.0593 1.0025 1.0335 1.0697 0.9973 1.0335 EUR 10212 0 21.11.2022 00:00 USA USD /templates/tatrabanka/assets/img/flags/us.svg