My code are scraping data from public site and storing in a df and saving as csv, but my code are not working well.

    for ano in lista_ano:
     for distribuidora in lista_distribuidores:
      for mes in lista_mes:
        scraping = pd.read_html('https://www2.aneel.gov.br/aplicacoes/indicadores_de_qualidade/decFecSegMensal.cfm?mes={}&ano={}&regiao=SE&distribuidora={}&tipo=d'.format(mes,ano,distribuidora))
        dfs= pd.DataFrame(scraping[0])
        dfs.drop(dfs.tail(3).index,inplace=True)
        dfs.drop(dfs.head(2).index,inplace=True)
        dfs = dfs.assign(MES = '{}'.format(mes))
        dfs = dfs.assign(ANO = '{}'.format(ano))
        dfs = dfs.assign(DISTRIBUIDORA = '{}'.format(distribuidora))
        all_dfs = pd.DataFrame(dfs)
        all_dfs.to_csv('final_data.csv', encoding= 'utf-8')

My problem here is my all_dfs.to_csv are creating a new csv for each looping and not storing data in the same local.

CodePudding user response：

You are overwriting the existing csv on each iteration.

To fix it, simply indicate that you want to append instead of write by

all_dfs.to_csv('final_data.csv', encoding='utf-8', mode='a')