I like Unicorn because it's Unix

Eric Wong’s mostly pure-Ruby HTTP backend, Unicorn, is an inspiration. I’ve studied this file for a couple of days now and it’s undoubtedly one of the best, most densely packed examples of Unix programming in Ruby I’ve come across.

Unicorn is basically Mongrel (including the fast Ragel/C HTTP parser), minus the threads, and with the Unix turned up to 11. That means processes. And all the tricks and idioms required to use them reliably.

We’re going to get into how Unicorn uses the OS kernel to balance connections between backend processes using a shared socket, fork(2), and accept(2) – the basic Unix prefork model in 100% pure Ruby.

But first …

A note about Unix programming in Ruby in general

We should be doing more of this. A lot more of this. I’m talking about fork(2), execve(2), pipe(2), socketpair(2), select(2), kill(2), sigaction(2), and so on and so forth. These are our friends. They want so badly just to help us.

NOTE The links above are to Linux section 2 manpages; in many cases, BSD manpages are vastly superior.

Ruby, Python, and Perl all have fairly complete interfaces to common Unix system calls as part of their standard libraries. In most cases, the method names and signatures match the POSIX definitions exactly. Yet, of the groups, only the Perl people seem to regularly (and happily) apply common Unix idioms to a wide range of problem areas.

Unix is not one of the “perlisms” Ruby should be trying to distance itself from. Perl got that part right. And with immediately recognizable Unix system calls spewed all over the core library, Ruby feels like it was built for Unix hacking. It’s surprising to see how infrequently this stuff is used as intended or even talked about.

man 2 intro

Documentation is likely part of the problem. Here’s a small sample of Ruby core docs on an assortment of Unix system calls – the kind we don’t use enough:

That Ruby provides no useful documentation isn’t actually a problem if you happen to have experience programming Unix. Then you would of course just happen to know that there’s a secret manual section with extensive reference information on each of these commands.

Here’s a snippet of the (BSD) manpage for pipe(2) as shipped with MacOS X (also: Linux version):

$ man 2 pipe
PIPE(2)                     BSD System Calls Manual                    PIPE(2)

NAME
     pipe -- create descriptor pair for interprocess communication

SYNOPSIS
     #include <unistd.h>

     int
     pipe(int fildes[2]);

DESCRIPTION
     The pipe() function creates a pipe (an object that allows unidirectional
     data flow) and allocates a pair of file descriptors.  The first descriptor
     connects to the read end of the pipe; the second connects to the write end.

     Data written to fildes[1] appears on (i.e., can be read from) fildes[0].
     This allows the output of one program to be sent to another program: the
     source's standard output is set up to be the write end of the pipe; the
     sink's standard input is set up to be the read end of the pipe.  The pipe
<snip>

Most Ruby developers probably don’t have a background in Unix C programming. How are they supposed to know that all those undocumented parts of the standard library are undocumented because – DUH! – you just have to go enter man 2 THING into the console. Obviously.

Threads are out

There’s another problem with Unix programming in Ruby that I’ll just touch on briefly: Java people and Windows people. They’re going to tell you that fork(2) is bad because they don’t have it on their platform, or it sucks on their platform, or whatever, but it’s cool, you know, because they have native threads, and threads are like, way better anyways.

Fuck that.

Don’t ever let anyone tell you that fork(2) is bad. Thirty years from now, there will still be a fork(2) and a pipe(2) and a exec(2) and smart people will still be using them to solve hard problems reliably and predictably, just like they were thirty years ago.

MRI Ruby people need to accept, like Python (you have seen multiprocessing, yes?), that Unix processes are one of two techniques for achieving reliable concurrency and parallelism in server applications. Threads are out. You can use processes, or async/events, or both processes and async/events, but definitely not threads. Threads are out.

Anyway, Unicorn.

Unicorn, and preforking servers in general, create a listening socket in a parent process and then fork off one or more child processes, each of which calls accept(2) on the same shared listening socket. The kernel manages the task of distributing connections between accepting processes.

Let’s start with a simplified example. A simple echo server that balances connections between three child processes:

# simple preforking echo server in Ruby
require 'socket'

# Create a socket, bind it to localhost:4242, and start listening.
# Runs once in the parent; all forked children inherit the socket's
# file descriptor.
acceptor = Socket.new(Socket::AF_INET, Socket::SOCK_STREAM, 0)
address = Socket.pack_sockaddr_in(4242, 'localhost')
acceptor.bind(address)
acceptor.listen(10)

# Close the socket when we exit the parent or any child process. This
# only closes the file descriptor in the calling process, it does not
# take the socket out of the listening state (until the last fd is
# closed).
#
# The trap is guaranteed to happen, and guaranteed to happen only
# once, right before the process exits for any reason (unless
# it's terminated with a SIGKILL).
trap('EXIT') { acceptor.close }

# Fork you some child processes. In the parent, the call to fork
# returns immediately with the pid of the child process; fork never
# returns in the child because we exit at the end of the block.
3.times do
  fork do
    # now we're in the child process; trap (Ctrl-C) interrupts and
    # exit immediately instead of dumping stack to stderr.
    trap('INT') { exit }

    puts "child #$$ accepting on shared socket (localhost:4242)"
    loop {
      # This is where the magic happens. accept(2) blocks until a
      # new connection is ready to be dequeued.
      socket, addr = acceptor.accept
      socket.write "child #$$ echo> "
      socket.flush
      message = socket.gets
      socket.write message
      socket.close
      puts "child #$$ echo'd: '#{message.strip}'"
    }
    exit
  end
end

# Trap (Ctrl-C) interrupts, write a note, and exit immediately
# in parent. This trap is not inherited by the forks because it
# runs after forking has commenced.
trap('INT') { puts "\nbailing" ; exit }

# Sit back and wait for all child processes to exit.
Process.waitall

Run that with ruby echo.rb and then connect a few times with netcat:

$ echo "hello world" | nc localhost 4242
child 86900 echo> hello world

$ echo "hello world" | nc localhost 4242
child 86901 echo> hello world

$ echo "hello world" | nc localhost 4242
child 86902 echo> hello world

...

Use telnet if you don’t have netcat:

$ telnet localhost 4242
telnet localhost 4242
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
child 86902 echo> hello world
hello world

$ telnet localhost 4242
...

This isn’t exactly what Unicorn does but it nicely illustrates one of the basic concepts underlying the design. This technique can be used for simple network servers as above, or it could be used for general IPC-based balanced multiprocessing. For example, you could connect directly to the shared socket in the parent process to distribute work between a set of child processes.

Unicorns do it with select(2)

Instead of a blocking accept(2), Unicorn uses a blocking select(2) with an error pipe and a timeout so that it can bust out and do some other basic housekeeping, like reopening logs, processing signals, and maintaining a heartbeat with the master process.

You can watch this play out in Unicorn::HttpServer#worker_loop, the heart of each Unicorn child/worker process. Most notable is the following call to select(2):

ret = IO.select(LISTENERS, nil, SELF_PIPE, timeout) or redo

This blocks until one of three things happen:

  1. A connection is pending on the shared listening socket (passed in the LISTENERS array) and is ready to be dequeued by accept(2), in which case the ready socket is returned.

  2. Some notable error state occurs on the file descriptor in SELF_PIPE (like when it’s closed), in which case the child’s side of the pipe is returned as an IO object. This really deserves its own essay, but I’ll take a quick shot: the IO object in SELF_PIPE is created in the parent process with pipe(2) (IO.pipe) before the children are forked off. The children then write on the pipe to achieve basic one-way IPC between child and master. It’s used here in the call to select(2) to detect the master going down unexpectedly - parent death causes the pipe to close. Unicorn children go down fast when their master dies.

  3. The timeout elapses.

If select(2) returns due the first condition, the child process calls Socket#accept_nonblock on the shared listening socket, which either returns a newly established connection or fails with Errno::EAGAIN, signaling that some other child process beat us to the accept(2). In either case, accept returns immediately and does not block.

This is just one of many beautiful Unix idioms you’ll find in Unicorn. The signal handling, hot process replacement/reloading with rollback, the SELF_PIPE IPC technique, and the fchmod(2) based heartbeat implementation are at least as interesting. Check it out.