How to write one array value at a time (dataframe to csv)?-CodePudding

This is working great, but I have thousands of rows to write to csv. It takes hours to finish and sometimes my connection will drop and prevent the query from finishing.

import pandas as pd
from yahooquery import Ticker

symbols = ['AAPL','GOOG','MSFT'] 
faang = Ticker(symbols)
faang.summary_detail
df = pd.DataFrame(faang.summary_detail).T
df.to_csv('output.csv', mode='a', index=True, header=True)

Above is only three symbols: symbols = ['AAPL','GOOG','MSFT'], but imagine there are 50,000 symbols. What I am currently doing is breaking it down into 500 symbols at a time:

import pandas as pd
from yahooquery import Ticker

symbols = ['AAPL','GOOG','MSFT'] #imagine here are 500 symbols.
faang = Ticker(symbols)
faang.summary_detail
df = pd.DataFrame(faang.summary_detail).T
df.to_csv('summary_detailsample.csv', mode='a', index=True, header=True)

symbols = ['BABA','AMD','NVDA'] #imagine here are 500 symbols.
faang = Ticker(symbols)
faang.summary_detail
df = pd.DataFrame(faang.summary_detail).T
df.to_csv('output.csv', mode='a', index=True, header=True)

#Repeat the last five lines 100  times for 50,000 symbols (500 symbols x 100 blocks of code).

So the last five lines of code I copy 100 times to append/write all the symbols' data. It works great, but I would like to not have 500 lines of code. I would like it to append a record one symbol at a time and throw all the 50,000 symbols in there one time (not have to copy code over and over).

Perhaps most importantly I would like the first symbol's column headers to be followed by the rest of the symbols. Some of the symbols will have 20 columns and others will have 15 or so. The data ends up not matching. The rows won't match other rows, etc.

CodePudding user response：

Try the below looping through the list of tickers, appending the dataframes as you loop onto the CSV.

import pandas as pd
from yahooquery import Ticker


symbols = [#All Of Your Symbols Here]
for tick in symbols:
    faang = Ticker(tick)
    faang.summary_detail
    df = pd.DataFrame(faang.summary_detail).T
            
    df.to_csv('summary_detailsample.csv', mode='a', index=True, header=False)