Rob Seaman

Ruby provides several simple ways to launch subprocesses. Ruby's popen method comes into play in situations where cross platform handling of subprocess IO is required. Ruby also provides mechanisms to trap and send signals, such as the SYSINT (Control-C) signal. Putting popen and signal handling together to launch and manage a subprocess is straight forward but there's a limitation; popen lacks support for capturing standard error. By default standard error output is just displayed on the screen as if the subprocess was launched from the command line. There are a couple of workarounds available, in some cases it may be acceptable to discard standard error by redirecting it as part of the subprocess launch command, i.e. "command 2>/dev/null"; another, and typically better, option is to redirect standard error so that it's merged with standard output, i.e. "command 2>&1". Unfortunately the redirection impacts the Process ID (pid) for the launched subprocess, which breaks the simple standard signal handling that I mentioned earlier. In addition to the pid issue, getting the Control-C signalling support to work in Windows can be challenging. This post walks through an example that shows popen usage and how to send Control-C signals to a subprocess in Windows and Unix when standard error is redirected. It concludes with a brief discussion of popen3 and popen4 as alternatives to popen.

I'll start with a simple Ruby program called sub.rb that will stand in for a program that we want to execute:

1
2
3
4
5
6
7
8

trap("INT") do 
    STDERR.puts "sub pid #{$$} Control-C"
    exit 2
end

puts "#{$$}"
STDOUT.flush
sleep 10

The program prints its pid ($$) and exits normally after 10 seconds if it doesn't receive a Control-C. If Control-C is pressed or sent from another process the trap handler is executed, it prints an error message and exits with an error code. Ruby allows traps to be specified by number or as a long or abbreviated string. For example, 2, 'SYSINT' or just 'INT' can be used to specify the Control-C trap. For a complete list of available traps open irb and type Signal.list. If you intend to use signals in a application targeted to multiple platforms, keep in mind that the list is OS dependent with Windows only supporting a small number of traps. Back to sub.rb, it's easy to check out the program by running it from the command line and pressing Control-C:

[mac]$	 ruby sub.rb
2994
^Csub pid 2994 Control-C

Unix without Standard Error

Next up is program called popen_sub.rb that will execute sub.rb, capture its output and send it a Control-C:

1
2
3
4
5
6
7
8
9
10
11

IO.popen("ruby sub.rb") do |pipe|
  puts "parent pid: #{$$}, popen return (child) pid: #{pipe.pid}"
  line = pipe.gets      # pid from child
  puts "child says it's pid is: "+line
  Process.kill 'INT', pipe.pid
  childs_last_word = pipe.gets
  if childs_last_word
    puts "child's last word: " + childs_last_word
  end
end
puts "child's exit code: #{$?.exitstatus}"

This first pass shows a basic way to use popen and capture the subprocess output. Process.kill sends a SYSINT (Control-C) signal to the subprocess and concludes by printing $?.exitstatus, which contains the exit code. So what happens when it's run:

[mac]$	ruby popen_sub.rb
parent pid: 3046, popen return (child) pid: 3047
child says it's pid is: 3047
sub pid 3047 Control-C
child's exit code: 2

It works, popen captured standard output from sub.rb, sent it a Control-C and captured its exit code.

Adding Support for Standard Error

The code worked but popen_sub.rb didn't capture sub.rb's standard error output as the childs_last_word. The standard error output just went directly to the display. Since we're already capturing standard output a simple way to capture standard error is to redirect the error stream when sub.rb is launched:

IO.popen("ruby sub.rb 2>&1") do |pipe|

That's easy and an Internet search will make it clear that it's a common practice. so what happens when the updated popen_sub.rb is run:

[mac]$	ruby popen_sub.rb
parent pid: 3060, popen return (child) pid: 3061
child says it's pid is: 3062
# 10 second delay
child's exit code: 0

That not what we wanted, obviously the SYSINT signal wasn't received and you can see why by looking at the pid. The redirection caused the pipe.pid to return 3061 but the actual pid for sub.rb was 3062. Now what? We can't just increment the pid returned by pipe.pid; assuming pids are assigned in certain ways by all OS's and that the pipe.pid from IO.popen will never handle redirection correctly is weak. But we could send the SYSINT to all processes in the current process group. There are a couple of ways to do that:

Process.kill 'INT', -Process.getpgrp

or

Process.kill 'INT', 0

Passing a negative pid indicates that it's a group ID. Using Process.getpgrp (or it's close relative -Process.getpgid($$)) to retrieve the process group ID is direct and obvious. So how does using zero work? Unfortunately Ruby's rdoc for Process.kill is incorrect regarding using a pid of zero. It indicates the signal is sent to the current process but that's not true. Looking at the rb_f_kill() function in Ruby's signal.c module there's some handling to support Posix and Berkley style kill() calls but other than that pids are just passed to the Unix OS call. Some Unix documentation states that using zero for the pid will send the signal to all processes, except system processes but I've only seen behavior consistent with this description:

"If pid is 0, sig shall be sent to all processes (excluding an unspecified set of system processes) whose process group ID is equal to the process group ID of the sender, and for which the process has permission to send a signal."

With either group ID solution the parent process will also get the SYSINT signal so it will need to have a trap handler in place when the signal is sent. The handler doesn't have to do anything so it can be as simple as this:

trap("INT") {}

If the parent doesn't setup a trap handler then Ruby's default SYSINT exception handling will take place, printing an error and terminating the program. That leads to the biggest caveat with this approach, since the signal goes to processes in the group, not just the current process and child processes, all parent processes need to set-up a handler and ignore the signal. This isn't an issue when you run the application directly from Ruby since it's the top-level parent. It can even be made to work as a child if you write the code for the ancestor processes but it is an issue when the Ruby application is spawned by a process. Unless you go through the trouble to patch the top-level parent so it handles signals the way you want then when the Ruby app decides to send a SYSINT, the parent will get the signal too and in most cases terminate. For apps that are intended to be standalone this isn't a show stopper but it's something to beware of. One common situation where it shows up is when running tests. It's not an issue with directly running unit or coverage tests (i.e. ruby test/test_xxx.rb or rcov test/test_xxx.rb) but it is a problem when running those tests with rake or rake coverage. In my test code I put code in to check for Process.getpgrp == Process.pid, if it's false I print a warning and skip tests that result in SYSINTs. Another workaround would be to place similar code in the application itself to avoid sending a SYSINT when there's a parent.

Making it work in Windows

There's been Process.kill support for Windows since Ruby 1.7.x but Ruby's win32.c kill() function has this check at the top:

3179
3180
3181
3182

if (pid <= 0) {
  errno = EINVAL;
  return -1;
}

That makes it pretty obvious that only individual process IDs are supported in Windows. Actually, I'm not sure if it would matter if there was group pid support since I wasn't able to get Ruby's Process.kill to work with standard pids in Vista or XP using the original non-redirect popen_sub.rb from above. That seems hard to believe so may be I did something wrong. I was focussed on group pids by the time I looked at Windows, so, I didn't dig into it but it definitely fell short of my expectation that the same code would just work in both Windows and Unix.

Win32 Utils provides an alternative with its win32-process gem. The gem is installed with the One-Click Ruby Installer for Windows gem pack so it's already part of most Windows installations. At first glance it doesn't appear to be much better than Ruby's built in support. Again, there's no group PID support so Process.getpgrp and -Process.getpgid($$) won't work. It does support using zero for the pid but unfortunately it implements the support as Ruby's rdoc says it should instead of the way the Ruby code implements it. Specifically, it intercepts the zero and translates it to the pid of the current process. And, just like with the standard Ruby implementation, Process.kill didn't work for me with normal pids on Vista or XP.

Ah, but there's a loophole that we can take advantage of in the win32-process gem. If we use nil for the pid it will get translated to NULL (0L in c/c++) before being passed as the dwProcessGroupId parameter to the GenerateConsoleCtrlEvent() Windows function. A zero for that parameter means that "the signal is generated in all processes that share the console of the calling process". Yeah baby! OK, it's a hack and depends on the implementation of win32-process but we don't have a lot of choices.

That leads to the final version of popen_sub.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

if RUBY_PLATFORM =~ /win32/
  require "win32/process"
  
  # GenerateConsoleCtrlEvent, dwProcessGroupId = 0L < NULL < nil
  def sysint_gpid; nil; end
else
  def sysint_gpid; -Process.getpgrp; end
end

IO.popen("ruby sub.rb 2>&1") do |pipe|
  trap("INT") do
    puts "parent pid: #{$$} Control-C"
  end

  puts "parent pid: #{$$}, popen return (child) pid: #{pipe.pid}"
  line = pipe.gets      # pid from child
  puts "child says it's pid is: "+line
  Process.kill 'INT', sysint_gpid
  childs_last_word = pipe.gets
  if childs_last_word
    puts "child's last word: " + childs_last_word
  end
end
puts "child's exit code: #{$?.exitstatus}"

The trap handler could have been placed in the top-level scope but I put it inside the popen block to make it clear that scoping rules apply when defining a trap handler. Also, in a real application that wanted to ignore SYSINTs the handler would be defined as 'trap("INT") {}', like I showed above, but for the test application I wanted to see when the parent received the signal. Here's a run of the program from Windows:

Z:\work\ptest>ruby popen_sub.rb
parent pid: 200, popen return (child) pid: 1468
child says it's pid is: 376
parent pid: 200 Control-C
child's last word: sub pid 376 Control-C
child's exit code: 2

That was a Windows run but it works fine in Unix too. That's it, both standard output and standard error went through the parent process and the parent successfully sent a Control-C (SYSINT) signal to the subprocess. BTW: If you don't care about standard error then the same code could also be used without the redirection. That would be helpful if you need cross platform support and you have the same problems I had getting Process.kill to work with pipe.pid in Windows.

Other Options

If redirecting standard error doesn't meet your needs then popen3 and popen4 are alternatives to popen that provide separate access to the standard error stream.

Unfortunately standard library support for popen3 is limited to Unix and it doesn't provide an way to get to the subprocess pid so I don't see a way to send signals without opening up the method. There is a Win32 Utils win32-open3 library that provides popen3 support. I gave it a quick try but I couldn't get it to work. May be I installed the wrong version or since I had issues with all Windows Process.kill implementations using standard pids, may be I ran into some related user error but I didn't bother to look into it very much because the project I'm working on requires signal support, and I was concerned about the issues I saw reported when googling for 'popen3 ruby windows' and finally, because it's not included in the Win32 Utils gems that are distributed with the One-Click Ruby Installer for Windows, and I wasn't crazy about making it a separate prerequisite.

popen4's advantage over popen3 is that it provides access to the subprocess pid along with the three IO streams. I didn't use it instead of popen because not part of the Unix standard library and I thought the Windows installation requirements were too complex. So, I didn't want it as a dependancy for any application that I might end up distributing.