Python Pandas - How to iterate every N rows until end of dataframe-CodePudding

How can I iterate 100 rows at a time in a Pandas dataframe? The forex_python package seems to error out when applying the lambda function all at once.

Here is what I've tried that doesn't seem to be working:

import pandas as pd
from forex_python.converter import CurrencyRates
c = CurrencyRates()
row_processed = 0

for row in df.iterrows():
    try:
        df['USD_rate'] = df.apply(lambda x: c.get_rate(x['Currency'], x['Date']), axis = 1)
        row_processed  = 1
        if row_processed == 100:
            row_processed = 0
            time.sleep(5)
    except:
        continue

CodePudding user response：

In case I understood you correctly, you want to operate over 100 rows at a time then sleep for 5 sec and continue again. Here is how you can do it:

from forex_python.converter import CurrencyRates
c = CurrencyRates()
for g_name, df_group in df.groupby(df.index//100):
    try:
        condition = df.index.isin(df_group.index)
        df['USD_rate'] = np.where(condition, df_group.apply(lambda x: c.get_rate(x['Currency'], x['Date']), axis = 1), np.nan)
        time.sleep(5)
    except:
        continue

CodePudding user response：

It's a pretty basic api with no rate limits. Another alternative is to just iterate over your date/currency combinations using this framework:

for i in range(2):
    url = 'https://theforexapi.com/api/2021-06-01/?base=USD'
    print(requests.get(url).text)

{"date":"2021-06-01","base":"USD","rates":{"EUR":0.8179959100204499,"JPY":109.65235173824132,"BGN":1.599836400817996,"CZK":20.827811860940695,"DKK":6.083108384458078,"GBP":0.7058077709611452,"HUF":283.4028629856851,"PLN":3.6532515337423317,"RON":4.023803680981596,"SEK":8.258077709611452,"CHF":0.8986503067484664,"ISK":120.65439672801637,"NOK":8.271002044989775,"HRK":6.141922290388548,"RUB":73.54707566462167,"TRY":8.52040899795501,"AUD":1.2918609406952966,"BRL":5.202126789366054,"CAD":1.2031083844580779,"CNY":6.383885480572597,"HKD":7.758936605316975,"IDR":14254.887525562372,"INR":72.8961145194274,"KRW":1108.7034764826178,"MXN":19.900122699386504,"MYR":4.126462167689162,"NZD":1.3772597137014315,"PHP":47.7758691206544,"SGD":1.3226175869120655,"THB":31.169734151329244,"ZAR":13.76564417177914}}
{"date":"2021-06-01","base":"USD","rates":{"EUR":0.8179959100204499,"JPY":109.65235173824132,"BGN":1.599836400817996,"CZK":20.827811860940695,"DKK":6.083108384458078,"GBP":0.7058077709611452,"HUF":283.4028629856851,"PLN":3.6532515337423317,"RON":4.023803680981596,"SEK":8.258077709611452,"CHF":0.8986503067484664,"ISK":120.65439672801637,"NOK":8.271002044989775,"HRK":6.141922290388548,"RUB":73.54707566462167,"TRY":8.52040899795501,"AUD":1.2918609406952966,"BRL":5.202126789366054,"CAD":1.2031083844580779,"CNY":6.383885480572597,"HKD":7.758936605316975,"IDR":14254.887525562372,"INR":72.8961145194274,"KRW":1108.7034764826178,"MXN":19.900122699386504,"MYR":4.126462167689162,"NZD":1.3772597137014315,"PHP":47.7758691206544,"SGD":1.3226175869120655,"THB":31.169734151329244,"ZAR":13.76564417177914}}

Then collect the data and create a dataframe after the fact. I ran range(200) with no issues.