Previously

This is the sixth in a series of “Nifty and Minimally Invasive qmail Tricks”, following

  1. qmail + SMTP AUTH + SSL + TLS - patches
  2. qmail + badrcptto - patches
  3. qmail + NetBSD nightly maintenance
  4. qmail + IMAP-before-SMTP
  5. qmail + spam filtering

Losing services (and eventually restoring them)

When my Mac mini’s hard drive died in the Great Crash of Fall 2008, taking this website and my email offline with it, I was already going through a rough time, and my mental bandwidth was extremely limited. I expended some of it explaining to friends what they could do about their hosted domains until such time as my brain became available again (as I assumed and/or hoped it eventually would). I expended a bit more asking a friend to do a small thing to keep my email flowing somewhere I could get it. And then I was spent.

The years where I used Gmail and had no website felt like years in the wilderness. That feeling could mostly have been about how I missed the habit of reflecting about my life now and again, writing about it, and sharing. But when the website returned four years ago (in order to remember Aaron Swartz), the feeling didn’t go away. All I got was a small sense of relief that my writings and recordings were available and that I could safely revive my old habit. After a year and half of reflecting, writing, and sharing, the feels-needle hadn’t rebounded much further.

It was only after painstakingly restoring all my old email (from Mail.app’s cache, using emlx2maildir), moving it up to my IMAP server, carefully merging six years’ worth of Gmail into that, accepting SMTP deliveries for @schmonz.com, and not needing Gmail at all for several weeks that I noticed my long, strange sojourn had ended.

Hypothetically speaking

If it so happened that I’d instead fixed email first, I’d also have felt a tiny bit weird till my website was back. But only a tiny bit. When my web server’s down, you might not hear from me; when my mail server’s down, I can’t hear from you — or, as happened in 2008, from my professors during finals week. So while web hosting can be interesting, mail hosting keeps me attached to what it feels like to be responsible for a production service.

Keeping it real

I value this firsthand understanding very, very highly. I started as a sysadmin, I’m often still a developer, and that’s part of why I’m sometimes helpful to others. But since I’m always in danger of forgetting lessons I learned by doing it, I’m always in danger of being harmful when I try to help others do it.

As a coach, one of my meta-jobs is to remind myself what it takes to know the risks, decide to ship it, live with the consequences, tighten the shipping-it loop until it’s tight enough, and notice when that stops being true.

And that’s why I run my own mail server.

What’s new this week

My 2014 mail server was configured just about identically with my 2008 one, for which it was handy to consult the earlier articles in this series.

Then, recently, my weekly build broke on the software I’ve been using to send mail. It was a trivial breakage, easy to fix, but it reminded me about a non-trivial future risk that I didn’t want hanging over my head anymore. (For more details, see my previous post.)

Now I’m sending mail another way. Clients are unchanged, the server no longer needs TMDA or its dependencies, and I no longer have a specific expectation for how this aspect of my mail service will certainly break in the future. (Just some vague guesses, like a newly discovered compromise in the TLS protocol or OpenSSL’s implementation thereof, or STARTTLS or Stunnel’s implementation thereof.)

A couple iterations

First, I tried the smallest change that might work:

  1. Replacing tmda-ofmipd with the original ofmipd from mess822 (by the author of qmail, the software around which my mail service is built),
  2. Wrapped in SMTP AUTH by spamdyke (new use of an existing tool),
  3. Wrapped in STARTTLS by stunnel (as before).

It worked! TMDA no longer needed. I committed an update to my qmail-run package with a new shell script to manage this ofmipd service, uninstalled TMDA, and removed its configuration files.

Next, I tried a change that might shorten the chain of executables:

  1. Committing an update to pkgsrc’s mess822 package adding a build-time option to apply John R. Levine’s SMTP AUTH patch,
  2. Doing a build with the new option enabled,
  3. Letting ofmipd handle SMTP AUTH on its own.

It worked! Second instance of spamdyke no longer needed. To start a mail submission service on localhost port 26, these are the lines I added to /etc/rc.conf:

qmailofmipd=YES
qmailofmipd_datalimit="160000000"
qmailofmipd_postofmipd="'' `cat /etc/qmail/control/me` /usr/pkg/bin/checkpassword true"

To make the service available on the network, this is the config from /etc/stunnel/stunnel.conf:

[submission]
accept = submission
connect = localhost:26
protocol = smtp

(It already had this stanza, but with 8025 where tmda-ofmipd was listening. I simply changed the port number and restarted stunnel.)

I’m still relying on spamdyke for other purposes, but I’m comfortable with those. I’m still relying on stunnel for STARTTLS, but I’m relatively comfortable keeping OpenSSL contained in its own address space and user account.

Refactoring for mail hosting

The present configuration is a refactoring: no externally visible change to email clients, yes internally visible change to email administrator (moi). I believe this refactoring was one of the best kind, able to be performed safely and reducing the risk I was worried about. The current configuration is much more likely to meet my future need to not have a production outage that interrupts my work for arbitrary duration while I scramble to understand and fix it. I don’t have any more cheap ideas for lowering my risk, and it feels low enough anyway. So I’m comfortable that this is the right place to stop.

Conclusion

Want to learn to see the consequences of your choices and/or help other people do the same? Consider productionizing something important to you.