Loop in R to read Files in multiple folders with no name pattern-CodePudding

I have folders for each day of each month, each containing 5-40 CSV files. The files have paths like

rootfolder/2/20/CSV/date-field-num.csv

and I want to read them all into R at once instead of individually. Is there a way to use a loop to do this if the files are in separate folders? The folder names have an obvious pattern. The actual CSV file names are of the form date-field-num.csv, where field could be anything and num could be any number.

CodePudding user response：

If you want all of the CSV files under rootfolder, then something like this should work:

## Get character vector listing paths to CSV files
fn <- list.files("path/to/rootfolder", pattern = "[.]csv$", full.names = TRUE, recursive = TRUE)

## Construct list of data frames
l <- lapply(fn, read.csv, ...)

with ... listing optional arguments to read.csv. Otherwise, you need to filter fn before the lapply call. Maybe:

fn_full <- fn
fn <- grep("^rootfolder/\\d /\\d /CSV/date-[^-] -\\d [.]csv$", fn_full, value = TRUE)

I'd be somewhat worried about memory usage with ~1000 CSV files. You might need to process fn in batches.