Turn one dataframe into several dfs and add them as CSVs to zip archive (without saving files locall-CodePudding

I have a data frame which I read in from a locally saved CSV file. I then want to loop over said file and create several CSV files based on a string in one column.

Lastly, I want to add all those files to a zip file, but without saving them locally. I just want one zip archive including all the different CSV files.

All my attempts using the io or zipfile modules only resulted in one zip file with one CSV file in it (pretty much with what I started with)

Any help would be much appreciated! Here is my code so far, which works but saves all CSV files just to my hard drive.

import pandas as pd
from zipfile import ZipFile

df = pd.read_csv("myCSV.csv")
channelsList = df["Turn one column to list"].values.tolist()
channelsList = list(set(channelsList)) #delete duplicates from list

for channel in channelsList:
    newDf = df.loc[df['Something to match'] == channel]
    
    newDf.to_csv(f"{channel}.csv") # saves csv files to disk

CodePudding user response：

DataFrame.to_csv() can write to any file-like object, and ZipFile.writestr() can accept a string (or bytes), so it is possible to avoid writing the CSV files to disk using io.StringIO. See the example code below.

Note: If the channel is simply stored in a single column of your input data, then the more idiomatic (and more efficient) way to iterate over the partitions of your data is to use groupby().

from io import StringIO
from zipfile import ZipFile

import numpy as np
import pandas as pd

# Example data
df = pd.DataFrame(np.random.random((100,3)), columns=[*'xyz'])
df['channel'] = np.random.randint(5, size=len(df))

with ZipFile('/tmp/output.zip', 'w') as zf:
    for channel, channel_df in df.groupby('channel'):
        s = StringIO()
        channel_df.to_csv(s, index=False, header=True)
        zf.writestr(f"{channel}.csv", s.getvalue())