I want to run multiple time-consuming shell commands from Ruby in a non-blocking (asynchronous) way.

I want to pass options to commands, receive output in Ruby, and (ideally) handle errors.

The script below will naturally take 15 seconds to execute:

test.rb

3.times do |i|
  puts `sleep 5; echo #{i} | tail -n 1` # some time-consuming complex command
end

$ /usr/bin/time ruby test.rb
0
1
2
       15.29 real         0.13 user         0.09 sys

With Thread, it can apparently be executed in parallel, and it takes only 5 seconds, as expected:

threads = []

3.times do |i|
  threads << Thread.new {
    puts `sleep 5; echo #{i} | tail -n 1`
  }
end

threads.each {|t| t.join() }

$ /usr/bin/time ruby test.rb
2
0
1
        5.17 real         0.12 user         0.06 sys

But is this the best approach? Is there any other way?

I have also written using Open3.popen2, but this seems to take 15 seconds to execute as in the first example (unless wrapped in a Thread):

require 'open3'

3.times do |i|
  Open3.popen2("sleep 5; echo #{i} | tail -n 1") do |stdin, stdout|
    puts stdout.read()
  end
end

The documentation describes "block form" and "non-block form", but this "block" refers to anonymous functions, and has nothing to do with concurrency, correct?

Is the Open3 class alone only capable of blocking execution?

CodePudding user response：

The problem with your code is that stdout.read is a blocking call.

You could defer the reading until the command is finished.

At first, create the commands:

commands = Array.new(3) { |i| Open3.popen2("sleep 5; echo hello from #{i}") }

Then, wait for each command to finish:

commands.each { |stdin, stdout, wait_thr| wait_thr.join }

Finally, gather the output and close the IO streams:

commands.each do |stdin, stdout, wait_thr|
  puts stdout.read
  stdin.close
  stdout.close
end

Output: (after 5 seconds)

hello from 0
hello from 1
hello from 2