How to parallelize for loops in R using multiple cores?-CodePudding

I am a beginner in R. I have a task in which I need to do a task similar to a for loop, but for a large number of observations, in this case, 500k times.

for (number in 1:500000){
sqrt(number)}

The problem is that I need to do this for loop over a lot of subjects. So the whole script would look something like this:

for (subject in 1:400){
   for (number in 1:500000){
        sqrt(number)}
}

I would want to parallelize this process over my multiple cores (16). I have found out that in the most tutorials they do something like this:

library(foreach)
library(doParallel)
registerDoParallel(16)

And then they proceed to simply change the syntax using:

foreach (subject=1:400) %dopar% {
   for (number in 1:500000){
     sqrt(number)}
}

To me, using all this together would mean that I will do each for loop that ranges from 1:500000 on one of my processors. So this would mean, if I had for instance a time complexity of one hour for one iteration in the 1:500000 loop, that I would do 16 iterations in one hour, as each one of my processors would do one iteration in parallel.

In practice, I checked by tracking the time of the elapsed time and it seems that my parallel process is actually much slower than the sequential one. Anything I am doing wrong? Is there an easy way (beginner-friendly) to just tell my machine to do each 1:500000 iteration on one core of my machine in parallel?

CodePudding user response：

You're not doing anything wrong, just that the operations you're running don't take enough time to make the parallel execution economical. Here's a snippet from the foreach vignette:

Running many tiny tasks in parallel will usually take more time to execute than running them sequentially, and if it already runs fast, there’s no motivation to make it run faster anyway. But if the operation that we’re executing in parallel takes a minute or longer, there starts to be some motivation.

You can see the benefits of parallel execution if we run sqrt not 500,000 times, but 50,000,000 times.

library(tictoc)
library(foreach)
library(doParallel)
registerDoParallel(16)

tic("no_parallel")

for (subject in 1:400){
  for (number in 1:50000000){
    sqrt(number)}
}

toc()
#> no_parallel: 271.312 sec elapsed

tic("parallel")

foreach (subject=1:400) %dopar% {
  for (number in 1:50000000){
    sqrt(number)}
}

toc()
#> parallel: 65.654 sec elapsed