Eric Wong’s mostly pure-Ruby HTTP backend, Unicorn, is an inspiration. I’ve studied this file for a couple of days now and it’s undoubtedly one of the best, most densely packed examples of Unix programming in Ruby I’ve come across.
Unicorn is basically Mongrel (including the fast Ragel/C HTTP parser), minus the threads, and with teh Unix turned up to 11. That means processes. And all the tricks and idioms required to use them reliably.
We’re going to get into how Unicorn uses the OS kernel to balance
connections between backend processes using a shared socket,
fork(2), and accept(2) — the basic Unix prefork model in
100% pure Ruby.
But first …
It’s as though every other version control system I’ve ever used was created by people who were really into version control and Git was created by people who were really into hacking.
So you’ve decided to start a weblog and have a really clever idea for titling it based on a snippet of code you find particularly novel. Rad!
I can’t think of single piece (package?) of software I use, admire, and depend on more than GNU Coreutils. Maybe Firefox. Maybe OpenSSH. Some days rsync(1).
I didn’t know it was possible to build such nice closed-source programs.
Cheap branches make for new uses.
The Dilbert cartoon referenced in Neil Stephenson’s “In The Beginning was The Command Line”
MacOS X: How to turn textmode tools into first class applications. Mutt.app, Vim.app, Irssi.app, Top.app, etc.
How to get command line apps to respect the OS X network location. A neat little hack exploiting symlinks and $0.
I can’t think of anything I like better than the intersection of writing and shell hacking.
A gentle introduction to the world of UNIX IPC. Covers fork, signals, pipes, FIFOs, file locking, POSIX message queues, semaphores, shared memory segments, memory mapped files, UNIX sockets. Not a ton of depth, but that’s okay – you can read all of it in about 15 minutes and have a good feel for the pros and cons of all the different types of IPC.
Check out Beej’s Guide to Network Programming and Beej’s Quick Guide to GDB too.
Interesting concept. Layer the routing guts found in modern web frameworks over Linux’s FUSE userland filesystem stuff and you get a nice model for developing custom filesystems.
The small example (~30 LOC) shows how to build a simple GitHub filesystem, which gives you this:
opus:~ broder$ ./githubfs /mnt/githubfs
opus:~ broder$ ls /mnt/githubfs
opus:~ broder$ ls /mnt/githubfs/ebroder
anygit githubfs pyhesiodfs python-simplestar
auto-aklog ibtsocs python-github2 python-zephyr
bluechips libhesiod python-hesiod
debmarshal ponyexpress python-moira
debothena pyafs python-routefs
opus:~ broder$ ls /mnt/githubfs
ebroder
Pretty awesome.
I’m pleased to announce the first public release of a small project I’ve been working on: bcat is a command line utility that streams text or HTML input to a web browser. Input is unbuffered and displayed progressively as it’s read from standard input, so bcat works great with programs that generate output over longish periods of time like build tools, tail(1), etc. It’s also useful for previewing HTML output when working on Markdown, Textile, AsciiDoc, Ronn, DocBook, etc. source files.
The plan is to bring as many of TextMate’s excellent HTML output capabilities as is feasible to the shell and to editors like vim or Emacs.
I love stuff like this:
The unix-jun72 project has scanned in a printout of 1st Edition UNIX from June 1972, and restored it to an incomplete but running system. Userland binaries and a C compiler have been recovered from other surviving DECtapes.
There’s also a mirror on GitHub.
This is probably the nicest manpage site I’ve come across:

I haven’t heard of it. They imported 98,660 manpages from all available Debian packages plus some from other sources. The type is clean. URLs are short and sweet. Manual sections are presented in a nice TOC on the left. They have some other novel features like comments on each manpage.
I planned to do something very similar. I even registered mancutter.org. A great number of manpages are distributed under a liberal license. I wanted to throw up a nice and simple site and then ship a tool anyone could run to bomb roff up to the server for all manpages on a machine. You should be able to gather all Linux and BSD manpages fairly quickly with such a system. Or, you could push up a specific set of manpages so project maintainers could publish directly to the site. I might still but man.cx is a huge chunk of what I was looking for.
Chris Wanstrath makes the case for UNIX man pages and tours through a bunch of tools for creating, finding, and reading them.
Perfect. This was a huge piece missing from Ron and I had no clue how to address it. Chris’s gem extension adds a gem man command that brings up the man page for any gem and works with any gem that includes normal roff man pages.
Dustin Sallings lays out a nice list of simple, non-pid-polling process supervision techniques available on various Unix and Linux environments. Great reference. I’m pretty sure the /etc/inittab respawn directive is one of the most underrated utilities in Linux server environments.
The technique in a nutshell:
The basic idea of what’s going to happen is that we will create a pair of pipes and then
fork(). The child process will hold the pipe that does the writing and the parent the one that does the reading. Now, the parent willexec. This is a bit odd. Normally when you fork, then exec, it’s the child process which does the exec. However, here we really want the new version of the program to have access to all of the old file descriptors. Luckily,execlpreserves these. As an added benefit, the program gets the exact same process ID.
Boom. Nice.
Nice list of Ruby one-liners when working at the shell. e.g., emulating nl(1):
# number each line of a file (left justified).
$ ruby -ne 'printf("%-6s%s", $., $_)' < file.txt
# number each line of a file (right justified).
$ ruby -ne 'printf("%6s%s", $., $_)' < file.txt
# number each line of a file, only print non-blank lines
$ ruby -e 'while gets; end; puts $.' < file.txt
Unlike some other things, the -p and -e switches are something I’ve always been glad ruby adopted from perl.
ASCII punctuation characters and their various pronunciations. e.g., the entry for Exclamation point (!) lists:
exclamation (mark), (ex)clam, excl, wow, hey, boing, bang, shout, yell, shriek, pling, factorial, ball-bat, smash, cuss, store, potion (NetHack), not (UNIX) ©, dammit (UNIX)
That last one has a footnote: “as in ‘quit, dammit!’ while exiting vi and hoping one hasn’t clobbered a file too badly.”
Insanely useful when you’re trying to avoid thread and process synchronization primitives — mutexes, flock, etc. — in concurrent code, which should basically be always. Rack::Cache’s file stores use some of these techniques to allow multiple backends to work against the same filesystem without file locks or a separate central writing process.
Tips for Ruby project maintainers on increasing the changes of getting your stuffs packaged for Debian. Most are just good sense. Use setup.rb, don’t explicitly require rubygems in your libraries and tests, use the most portable shebang (#!/usr/bin/env ruby), and provide a man page. Ron can help with that last one.
defunkt’s hub is a command line utility that adds GitHub knowledge to git. Sweet. It expands GitHub repository references so you can do stuff like: git clone defunkt/gist, git remote add bmizerany, etc.
I’ve released a tool for authoring UNIX manual pages using a markdown-ish source format:
Ron is a humane text format and toolchain for creating UNIX man pages, and things that appear as man pages from a distance. Use it to build and install standard UNIX roff man pages or to generate nicely formatted HTML manual pages for the web.
It still needs some work but can produce useful output for both roff and HTML. The sources are on GitHub.
Tobi’s log tail and grep app is precisely what I’ve wanted on every single syslog machine I’ve ever had to deal with. And the code has some great examples of using EventMachine features to do real async HTTP stuff.
I recently started a repository for my dotfiles, shell environment, vim config, and utility scripts. As of right now, I’m about 25% through all of the stuff in my $HOME — it should all fill in shortly.
I’ve dumped MacPorts for pkgsrc. This quick tutorial helped me get going and this package browser is awesome.
Not sure how I never heard of this program before:
dtachis a tiny program that emulates the detach feature of screen, allowing you to run a program in an environment that is protected from the controlling terminal and attach to it later.dtachdoes not keep track of the contents of the screen, and thus works best with programs that know how to redraw themselves.dtachdoes not, however, have the other features of screen, such as its support of multiple terminals or its terminal emulation support. This makes dtach extremely tiny compared to screen, making it more easily audited for bugs and security holes, and also allows it to fit in environments where space is limited, such as on rescue disks.
GitHub has rake tasks that use dtach to manage redis and maybe some other things.
Nice little self-contained perl script that shows a basic memcached top display for a list of servers.
$ curl http://memcache-top.googlecode.com/files/memcache-top-v0.6 >
~/bin/memcache-top
$ chmod +x ~/bin/memcache-top
$ memcache-top --sleep 1 --instances memcache1,memcache2,memcache3
That gives you this:

Nifty.
Jeremy Zawodny takes a look at the * is Unix thing and throws in some additional goodness: more on fork(2), the benefits of copy-on-write, and atomic file operations.
@paulsmith’s simple preforking echo server in C.
Aristotle Pagaltzis comes through with the simple preforking echo server in Perl.
Unix Network Programming, volumes 1 and 2, and Advanced Programming in the UNIX Environment. Yes.
Jacob Kaplan-Moss does the prefork echo server example from my Unicorn is Unix piece in Python. Awesome. Let’s see some more of these. Where you at, Perl?
Chuck presents the results of a few ab runs against Thin and Unicorn. Surprising, even if the benchmarks lack rigor. I’d like to see good autobench runs for all of the backends on same hardware/network.
I didn’t know about this:
In Snow Leopard, Apple has introduced a C language extension called “blocks.” Blocks add closures and anonymous functions to C and the C-derived languages C++, Objective-C, and Objective C++.
They go on to list code samples in each language. The syntax is … not what I expected. Check out the section on LLVM and Clang also.
Amazing, the way these things happen:
… in August 1969, Ken Thompson’s wife took their new baby to see relatives on the West Coast. She was due to be gone for a month and Thompson decided to use his time constructively – by writing the core of what became Unix. He allocated one week each to the four core components of operating system, shell, editor and assembler.
Constraints are powerful things.
Nice. The xargs(1) switch -P N will run up to N separate processes in parallel. Combine with the -n M switch for a quick and dirty process pool.
It’s important to understand how fork(2), pipe(2), and exec(2) work. I don’t want to hear anymore of this “fork is a hack” shit from any of you :)
Big list of new features in bash 4.0.
proggit on shoddy reporting by NPR.
A lost art, indeed.
Best. Program. Ever.
“The syntax for many of the commands in bashdb mimics that of gdb, the GNU debugger. You can step into functions, use next to execute the next line without stepping into any functions, generate a backtrace with bt, exit bashdb with quit or Ctrl-D, and examine a variable with print $foo.”
Laurence Tratt: “I had implicitly bought into the idea that C programs segfault at random, eat data, and generally act like Vikings on a day trip to Lindisfarne; in contrast, programs written in "higher level” languages supposedly fail in nice, predictable patterns. Gradually it occurred to me that virtually all of the software that I use on a daily a basis – that to which I entrust my most important data – is written in C. And I can’t remember the last time there was a major problem with any of this software – it’s reliable in the sense that it doesn’t crash, and also reliable in the sense that it handles minor failures gracefully."
RDiscount, a fast Markdown library for Ruby, is now included with the FreeBSD ports collection thanks to Daniel Roethlisberger.
There’s so many great workflow hacks in here.
“Varnish implementes a subset of the ESI Language 1.0 defined by W3C, this document lays out some of the thoughts and rationale for choices made and advice for usage of these features.”
This lets you perform includes at the cache layer so that each included resource can have its own caching policy. Akamai edge proxies have supported this for some time, apparently.
Looks like a really solid improvement on 1.0. I haven’t had a chance to play with any of the betas but I’m anxious to see whether If-Modified-Since/If-None-Match validation made it in. There’s a note on “serving expired objects until we have a fresh one” but that sounds more like stale-while-revalidate.
Dan Kegel: “You can buy a 1000MHz machine with 2 gigabytes of RAM and an 1000Mbit/sec Ethernet card for $1200 or so. Let’s see – at 20000 clients, that’s 50KHz, 100Kbytes, and 50Kbits/sec per client. It shouldn’t take any more horsepower than that to take four kilobytes from the disk and send them to the network once a second for each of twenty thousand clients. (That works out to $0.08 per client, by the way. Those $100/client licensing fees some operating systems charge are starting to look a little heavy!) So hardware is no longer the bottleneck. ”
Looks like this is from 2003 but is still pretty accurate as far as I can tell.
Talk about a religious attachment…
Dump the stack trace of all threads in a running ruby process by signaling with -QUIT. Requires patching the ruby interpreter, which sucks because I need it for a process running right now.
Alexander Sandler’s get-up-and-running guide to the tcpdump packet sniffer.
Sometimes! Or, fork(2) is a very fast operation on legitimate operating systems. I didn’t realize it could be as fast as spawning a thread, though.
All manners of good stuff here.
Erik Engbrecht: “Java took cheap Unix processes and made them expensive. To compensate, it provided primitives for multithreading.”
Okay, I’ve read about five of these articles purporting to explain Git’s internal conceptual framework. This was the first that really made things click in any significant way.
Most of these are relevant to POSIX sh(1). This one gets me every time: echo <<EOF :)
More praise for GitHub from a small team of Django hackers that built a site in three hours on one night with a little help from git…
Make Firefox like Vim. No, like, insanely like Vim. Not just h,j,k,l mappings but everything. Looks like it’s been around for awhile. I’m not sure how I missed it.
I thought I had a few more months. Dammit. This is going to be a huge time-sink.
Uggghhh. I just spent 30 minutes hunting some arcane tcsh bug caused by coreutils dircolors. This is my revenge. I don’t even know I had any csh code running on this machine. It turns out that MacOS X’s /usr/bin/which is implemented in csh. Dumb.
This is why I love Unix.
“… adds a chroot(2) facility to sshd, controlled by a new sshd_config(5) option ‘ChrootDirectory’. This can be used to ‘jail’ users into a limited view of the filesystem, such as their home directory …”
Seriously interesting web based git browser and collaboration tool from the folks at Engine Yard. If anyone has a spare invite laying around, hook me up: rtomayko@gmail.com. I have a bunch of stuff sitting in bzr repos that I’d like to flip over to git.
I have a strange fetish for init systems (sysv, rc, launchd, etc). This is the first quick introduction to Ubuntu’s new init system (Upstart) I’ve seen. Nice examples of using the initctl command and writing job files.
Finally: “this manual is designed to be readable by someone with basic UNIX command-line skills, but no previous knowledge of git.”
There’s some good questions here. I’ve been running into a few of the same issues while experimenting with moving some of my bzr projects to git. Can one of the git pros out there have a look?
Runs on Linux and FreeBSD (with linproc mounted on /compat/linux/proc). I’ve always wondered why top(1) just kind of stopped being developed 10 years ago.
What PrinceXML is coded in, apparently. It’s like Prolog for large systems: declarative, strongly typed and type inferencing, module system, closures, currying, lambdas, and with a strong determinism system. Compiles down to C (as a portable assembler).
pv can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, and an estimate of how long it will be until completion.
Oops: “At $1 billion … Sun paid a multiple of 10 times sales for MySQL today. Optimistically assuming a 20% profit margin, they are looking at a multiple of 50 times earnings for a return on investment of around 2% per year. Optimistically.”
Dennis Ritchie: “There was a facility that would execute a bunch of commands stored in a file; it was called runcom for ‘run commands’, and the file began to be called ‘a runcom’. rc in Unix is a fossil from that usage.”
Steve does the Sun/MySQL aquisition Q&A and speculates on some interesting effects of the deal: “… YouTube sold for $1.6 billion, and consumed virtually no software. If that acquisition was to take place today, they would have been buying from Sun.”
Payware GUI shell thingy for MacOS. This is not a QuickSilver/Launchbar clone. It’s more like a magical bash interpreter that knows things about what’s happening in various Mac GUI applications (like Finder, Safari, etc).
“I am going to introduce you to bash’s vi editing mode and give out a detailed cheat sheet with the default keyboard mappings for this mode.”
“Other than the fact our child will be bright, text-based and sarcastic, we will otherwise be a normal family.”
I can never remember nmap args for some reason…
“Like with unix, cells are not ‘spawned’ – they are forked. All cells started out from your ovum which has forked itself many times since. Both halves of the fork() are identical to begin with, but they may from then on decide to do different things.”
“Installation is left as an exercise for the reader.”
“The goal of the GNU PDF project is to develop and provide a free, high-quality, complete and portable set of libraries and programs to manage the PDF file format, and associated technologies. ”
Manual page URL handler for Safari (e.g., “man:bash”, “man:sort” in URL box). References to other man pages are hyperlinked very nicely and the pages themselves are formatted quite nicely.
Brilliant!
Schwartz: “… we will be going after sizable monetary damages. And I am committing that Sun will donate half of those proceeds to the leading institutions promoting free software and patent reform, and to the legal defense of free software innovators.”
“v1.8.0 represents nearly a year of development and testing to bring you a host of new features and improvements”
Looks like they’re bringing the basic capabilities of readline up to the GUI level. Definitely interesting.
Vim add-in for viewing manpages, perldoc (both system and embedded), help, info, and php files. Maybe I’ll finally be able to read all that GNU info doc I keep hearing about in the GNU coreutils man pages.
“To average users, the suggestion that they use the command line – or the shell, or the terminal, or whatever else you want to call it is only slightly less welcome than the suggestion that they go out and deliberately contract AIDS.” That’s a damn sham
Starting with absolutely no configuration file. This is why I’ve prefered lighttpd, because I can put together a separate config in about five minutes. httpd’s sprawling default config has always scared the crap out of me.
Some detail on rsync’s “rolling checksum” algorithm invented by Andrew Tridgell.
Very nice look at different methods (good and bad) for handling the command line in sh scripts.
I’m gonna give this a try for managing home directories now that I’ve convinced myself that version control is the wrong solution. I moved my homes from CVS to SVN a couple years ago and just tried going with bzr but VCS just isn’t right here.
“… cron’s pathological behavior has be petrified into the Unix standards. So if it isn’t broken, it isn’t cron.”
Pretty much what you thought but with great detail :)
The emacs(1) manpage from Bell Labs’s Plan 9.
Java becomes 100% more viable. So simple — why didn’t someone do this in the very beginning?
You had me at “SSH”.
Nifty combo.
About half of these will be well-known to the UNIX hacker but there’s a couple I’ve not seen elsewhere: lsbom, softwareupdate, screencapture, and lipo.
PARC’s Van Jacobson (traceroute(8), tcpdump(1)) on, well, everything that matters. Hands down best talk I’ve seen in years. I’m going to watch it again tomorrow.
Slashdot has become a horrible discussion forum for most topics. Disk theory and UNIX sysadmin type stuff is an exception, though. This story on ZFS might have the most informational comments I’ve seen in years.
What’s next? which(1)?
Michael Dell runs Ubuntu 7.04 on his personal laptop :)
Nice look at techniques for writing portable sh.
I haven’t had a chance to play yet but you can consider me on the ZFS bandwagon. It’s coming to FreeBSD 7.0, too. Oodalolly!
Seth is on the market. Hire him.
This is pretty darn close to my configuration but I used the mutt-devel port… Oh, and my ~/.procmailrc is pretty insane also :)
Recent presentation by Mr. Bram Moolenaar on how to be a bad-ass with Vim.
“It looks like you are trying to do a regular expression. Do you need some help with that?”
exec 3<> /dev/tcp/$HOST/80 What?! How cool is that.
Best idea ever. EVER!
“et see” :)
A complete look at the little used utilities for processing arguments in scripts.
Lots of stuff from Sun (ZFS, dtrace), Linuxulator translates Linux syscalls to BSD syscalls with not performance penalty, lots of performance enhancements to the network stack from the card up, and a new malloc.
Love it! This is less of an article and more of a minute by minute account of hacker seeing something he doesn’t understand and following the trail (man, code, calculus) to understanding.
Anyone who doesn’t know every single one of these probably hasn’t been developing for the web very long. Probably a useful crash course for newbies making their way over from FrontPage or ASP.net though.
“Essentially, this add Vi command functionality (albeit a small subset) to any (and all) text editors that use the Cocoa text system; e.g., Safari, TeXShop, XCode, etc.”
“The demoroniser keeps you from looking dumber than a bag of dirt when your Web page is viewed by a user on a non-Microsoft platform.”
I’ve been using a fetchmail, procmail, and mutt setup on my Mac for a few months now in an attempt to get control over five different mailboxes and it’s working pretty well. If you’ve got some free time and lots of mail, consider playing around with one o
We moved from Windows / MS SQL Server to FreeBSD / PostgreSQL about 5 months ago and I’ve been nothing but completely happy with the transition. 8.2 is a pretty nice upgrade if you’re doing data warehousing style stuff.
“The archetypal perversely awful algorithm”
Best UNIX productivity article I’ve read in a long while.
All you have to do is add a few lines to ~/ssh/config.
How to get rdiff-backup to not do that.
… and not just the usual suspects either.
Nice and compact…
How to not be fucked with…
Stallman on the EU software patent mess.
Worse is better.
Alright, it looks like I’m going to have to break down and learn how to package ports since none of this crap is working on Tiger.
Need to move away from history | grep -i
Introduction to being a complete bad-ass.
All on one page :)
yeah whatever… I’ve been trying to learn emacs for years.
Neal Stephenson on UNIX.
“KDE and GNOME have both gotten much better, but let’s get real. They’re not even in the same ballpark.” — Ouch. True though…
What do Moore’s law and boiling oceans have in common? Sun’s Jeff Bonwick explains in three easy paragraphs. Really brilliant stuff.
Some really good info on various methods of dealing with synchronization between processes on *NIX based systems.
ctags/etags, strace, fuser, ps, time, nm, strings, od/xxd, file, objdump
“A guide to writing shell scripts for C/C++/Java and unix programmers”
A remote file editing package for Emacs. Uses ssh/scp.
Super useful tips on diving into Emacs.
A bunch of extremely useful notes on hacking emacs. (Ftrain.com)
Old and still very valid. What’s the best mix of Simplicity, Correctness, Consistency, and Completeness in software design? Describes MIT and “NewJersey” approaches.
Single page printable version available.
After using Emacs for three years, I think I finally need to learn how to use it. This has some good pointers.
This much sed will eat your brains!