I am working with the R programming language.
Normally, I print iterations from loops - this helps me keep track of things. For example:
my_list = list()
for (i in 1:1000)
{
{tryCatch({
frame_i = rnorm(1,1,1)
my_list[[i]] = frame_i
ifelse(i%% 10 == 0 , cat("\f"), print(i))
}, error = function(e){})
}}
# desired output
[1] 1.0000000 -0.9298258
Now, I am trying to do the same thing - but using the "doParallel" libraries in R:
library(doParallel)
registerDoParallel(cl <- makeCluster(3))
test = foreach(i = 1:1000, .combine = "rbind") %dopar% {
{tryCatch({
frame_i = rnorm(1,1,1)
my_list[[i]] = frame_i
ifelse(i%% 10 == 0 , cat("\f"), print(i))
}, error = function(e){})
}}
This loop seems to have run, but nothing seems to be printing.
I tried consulting the following posts:
- Print outputs in foreach loop in R (e.g. sink("Report.txt") does not appear to create a file on my computer)
- No standard output received inside foreach loop
- R foreach and print behavior
- Is it possible to create a text log within a foreach() %dopar% call in R?
But I still can't seem to figure out how to print the (intermittent) results from the "foreach loop" - can someone please help me figure out how to do this?
Thanks!
- Note: I am interested in printing out the actual iterations, and not creating a progress bar
CodePudding user response:
(Disclaimer: I'm the author)
The Futureverse combined with progressr will give you near-live progress updates also when running in parallel. An example adopted from https://progressr.futureverse.org/#foreach-with-dofuture is:
library(doRNG) ## %dorng%
library(doFuture)
plan(multisession, workers = 3)
registerDoFuture()
library(progressr)
handlers(global = TRUE)
test <- local({
p <- progressor(1000/10)
## %dorng% instead of %dopar% because of rnorm()
res <- foreach(i = 1:1000, .combine = "rbind") %dorng% {
frame_i <- rnorm(1,1,1)
if (i %% 10 == 0) p(sprintf("i = %d", i))
frame_i
}
## Drop %dorng% records (not needed with %dopar%)
attr(res, "rng") <- NULL
attr(res, "doRNG_version") <- NULL
res
})
CodePudding user response:
As suggested here, package {pbapply} offers a progress bar which can also be applied for calls to parallel processing. Example:
library(pbapply)
library(doParallel)
registerDoParallel(cl <- makeCluster(3))
my_fun <- function(repetitions = 10){
foreach( i = 1:repetitions, .combine = 'rbind' ) %dopar% {
Sys.sleep(.5)
}
}
tmp <- pbreplicate(3, my_fun(10))
## > tmp <- pbreplicate(3, my_fun(10))
## | | 100%
However, the progress bar increases only when a packet is finished, it does not increase with progress inside a packet.