Efficient method to append pandas rows into a list-CodePudding

I have a pandas DataFrame with ~5m rows. I am looking for an efficient method to append / store each rows into a list.

import pandas as pd

df = pd.DataFrame({ 
                    'id': [0, 1, 2, 3],
                    'val': ['w','x','y','z'],
                    'pos': ['p1','p2','p3','p4']
                 })

# Using List comprehensions 

df_lst = []
[df_lst.append(rows) for rows in df.iterrows()]

Given the size of the DataFrame; I am looking for other methods that are more efficient at storing rows to a list. Is there a vectorized solution to this?

CodePudding user response：

I'd recommend .tolist() as others also mentioned in the comments. So I'll give an example of it.

df_lst = df.values.tolist()

in terms of efficiency, which I see some have mentioned why you want to do that, that would depend on the use case. of course df is more efficient on performing various tasks on the data and it seems to be redundant to convert it, but note that a list is more memory-efficient than a dataframe. So converting that is not unrational if you don't need the features of the df.

CodePudding user response：

From here:

You can use df.to_dict('records') to convert the rows into a list of dicts. If this is useful depends on what you want to do with the list afterwards.