R: efficient way to loop rbind for multiple files-CodePudding

I have 1000 permutations saved as RData objects. I want to combine the info object from all of these permutations into a single table (info_perm). This is very fast to start with, but ends up becoming much slower as the file size increases. Any suggestions about how to solve this problem more efficiently rather than rewriting info_perm each time? Thanks!

info_perm <- NULL; for (perm in 1:1000) {
  load(paste0("./output/", perm, ".RData"))
  info$perm <- perm
  rbind(info_perm, info) -> info_perm
  if (perm/100 == round(perm/100)) {print(perm)}
}; rm(perm)

note: I'm aware of options like do.call("rbind",l) which would work if I had all of my files in the environment at the same time, but is there a faster way to do it in this framework rather than saving intermediary version of info after each is read in?

CodePudding user response：

What about futures?

library(future)
library(future.apply)

plan(multisession)

perm = 1:1000 
info_perm <- do.call(rbind, future_lapply(perm, function(perm){
  load(paste0("./output/", perm, ".RData"))
  info$perm <- perm
  if (perm/100 == round(perm/100)) {print(perm)}
  info
})