How to iterate chosen operation through different csv files and output the result to new csv files-CodePudding

I'm attempting to use the pandas and os to take CSV files from a chosen directory and delete specified rows iteratively, as they are produced in a form that I don't like. Following the iteration, I'd like the CSVs to be output with the same name a suffix to indicate that the iteration has been performed.

I am coming back to Python after a long hiatus, but here is what I have so far:

DIR = 'Dummy_Folder/'

JV_suffix = "Current-Voltage Data.csv"
# For each file name in the chosen directory (DIR)
for filename in os.listdir(DIR):
    # If file name ends with chosen suffix perform...
    if filename.endswith(JV_suffix):
        # Read the original csv file InputName
        data = pd.read_csv(filename,header=None)
        # Defining the rows that we want to delete from the file
        trim = [0,1,2,len(data)-1,len(data)-2,len(data)-3]
        # Deleting these rows and defining new csv
        trim_data = data.drop(trim,axis=0) 
        # Making data frame with trimmed csv
        df = pd.DataFrame(trim_data)
        # Outputting df to CSV with suffix of '_number'
        df.to_csv('DIR/filename_{}'.format(filename),index=False)
    else:
        continue

When I run this on dummy data it throws error FileNotFoundError: [Errno 2] No such file or directory: 'Dummy_Current-Voltage Data.csv'

Thank you for taking the time to read this!

CodePudding user response：

It is probably an error concerning the relative path of your files.

I presume your folder Dummy/ is on the python projects path. If the folder is found correctly, the correct relative path for any file inside of the folder should be Dummy/filename.

You can use os.path.join(foldername,filename) to join the folder name and filename.

CodePudding user response：

Thank you qouify and Sam for the suggestion, this solved the problem, along with changing the output CSV line to df.to_csv('trimmed_{}'.format(filename),index=False). I think the inclusion of DIR in the filename was causing problems. I attach the whole script below as a demonstration for others who have similar queries

import pandas as pd
import os, os.path

DIR = 'Dummy_Folder/'
# For each file name in the chosen directory (DIR)
for filename in os.listdir(DIR):
    # If file name ends with chosen suffix perform...
    if filename.endswith("Current-Voltage Data.csv"):
        # Read the original csv file
        data = pd.read_csv(os.path.join(DIR, filename),header=None)
        # Defining the rows that we want to delete from the file
        trim = [0,1,2,len(data)-1,len(data)-2,len(data)-3]
        # Deleting these rows and defining new csv
        trim_data = data.drop(trim,axis=0) 
        # Making data frame with trimmed csv
        df = pd.DataFrame(trim_data)
        df.to_csv('trimmed_{}'.format(filename),index=False)
    else:
        continue