September 2010 Archives

VIM Plugin: Makesd/Makecsd

| No TrackBacks

I wanted to share a little VIM plugin I just got done writing, makesd.vim. This plugin is pretty straightforward, and is adapted from a couple of Perl command-line scripts I tend to haul around called makesd and makecsd.

In short, they produce clean looking separators for use inside source code:

:Makesd "Public Interface"
# ========================================================================== #
# ============================ Public Interface ============================ #
# ========================================================================== #

:Makecsd "Public Interface"
/* ======================================================================== */
/* =========================== Public Interface =========================== */
/* ======================================================================== */

It's my first VIM script. VIM scripting is pretty easy -- give it a try!

Brain Damage

| No TrackBacks

Another one for the record books. Once again, the kind folks at Redmond have proven how truly incompetent they are, and why at the end of the day, a power user can only survive on an open source platform.

Microsoft has stripped the ability to save raw e-mail messages from Outlook 2007. Supposedly this capability exists in Outlook Express and/or Windows Mail. But there are mountains of bad advice suggesting that export to those programs, followed by an import, or an export to .txt, is an acceptable alternative. (It isn't, in all cases, the transport headers aren't included. Who knows how else Microsoft is molesting the message).

Commercial solutions exist - one for only $14 (are you kidding me?) and one for over $60 (are you kidding me???)

I've put out a new transocks_ev patch transocks_ev-performance-reliability-dns-logging.patch. transocks_ev is a neat little program by Bernd Holzmueller at tiggersWelt.net that uses the Linux netfilter/iptables stack to intercept outgoing TCP connections and transparently convert them into SOCKS5 proxy connections. It's based on transocks which does the same thing. transocks uses a forking model, while transocks_ev uses libevent to multiplex connections in a single process's event loop.

I'm planning on using these changes to transparently intercept outgoing Postfix SMTP connections on some backend mail servers and to use DNS-based load balancing to fan those connections out across multiple proxy servers/Internet connections.

In addition, I've improved the logging support of transocks_ev, giving it three levels of verbosity with basic statistics collection for the connections. All blocking operations have been converted to non-block (connect(), write(), the new DNS). The code is using libevent bufferevent to manage low level socket access.

libsoup Ignores DNS TTLs

| No TrackBacks

I've been using libsoup to run a small SOAP engine for one of the back-office programs I maintain. We've recently upgraded to a new load-balanced architecture, and we are using DNS-based load balancing to fan these SOAP requests out across our servers.

It only took a few days in production to realize that libsoup was doing something nasty. Prior to any HTTP request, you need to create a SoupSession object. This object manages things like connection pools / keepalive. It contains a GHashTable called hosts, which it uses as a cache of connections to a given hostname.

/* Requires host_lock to be locked */
static SoupSessionHost *
get_host_for_uri (SoupSession *session, SoupURI *uri)
{
    SoupSessionPrivate *priv = SOUP_SESSION_GET_PRIVATE (session);
    SoupSessionHost *host;

    host = g_hash_table_lookup (priv->hosts, uri);
    if (host)
        return host;

    host = soup_session_host_new (session, uri);
    g_hash_table_insert (priv->hosts, host->uri, host);

    return host;
}

Unfortunately, entries in this hash table are never removed or expired unless the SoupSession object itself goes away. This sucks for a few reasons:

  1. DNS TTL values are ignored. Instead, the result of the DNS query is cached forever. Obviously this means that if the record is ever changed, libsoup clients need to be restarted to know about it.
  2. DNS load balancing is broken by libsoup, which will repeatedly connect to the same IP address regardless of whether multiple IPs are included in the response to an A query.
  3. You really wouldn't want to write a robot or some other long lived program that would make lots of connections to lots of different hosts using libsoup, as it stands. Aside from the obvious correctness issues listed above, the hosts hash table will experience unbounded growth. Thankfully all of our connections are to the same small set of URLs and hostnames.

I'm not sure how easy it would be to patch libsoup to behave correctly. As far as I can tell the GResolver that libsoup relies on doesn't even report TTLs.

Given the nature of this bug I can only see a few workarounds:

  1. Set the Host HTTP header yourself, do the DNS query yourself using GResolver, and supply the server's IP address to the SoupURI instead of a hostname. This breaks SSL certificate validation.
  2. Recycle/create the SoupSession per-request. This breaks keepalive/connection pooling and has obvious overhead issues.

Given the nature of how I'm using libsoup, I chose the latter option. YMMV.

I'm doing another small code release. This one is asterisk-func_dns, a dialplan function DNS() to do an explicit DNS query without requiring you to launch an external program. It's an alpha release currently intended for Asterisk 1.4 and only supporting IPv4 / DNS A record types.

I'm using this to implement DNS-based load-balancing for outgoing calls across a series of proxies and internet connections.

In my dialplan, I request the IP addresses of my proxy servers ahead of any attempt to Dial(). This module returns the list of IP addresses published in the record, separated by commas. This allows me to sequentially fork across the proxy servers, and since I don't need to rely on Dial()'s forking support, I can add additional processing in between attempts. Since I obtain the proxy set by looking up a single DNS name, my Asterisk dialer configurations do not have to change if I add more proxy servers to my network, also meaning that those Asterisk dialers will not waste time trying to contact outbound proxy servers that have gone offline for maintenance or due to a failure. Each Asterisk dialer will try every call amongst all the working proxy servers, up to one attempt each, in a random order.

This code could benefit from some obvious todos: forward port to modern Asterisk versions, implementation of the ability to grab other record types like SRV or AAAA, etc. I may address these eventually but at the moment this is "good enough for me". I release this code (under the same Asterisk licensing terms: GPLv2) with the hopes that someone finds it useful.

Okay, I'm pleading with developers. I'm very impressed at the number of options and switches that your program exposes via its configuration file(s) / directories / databases. Bonus points for those of you who have managed to extensively document each switch and its default setting with inline comments.

Actually, that strategy even works up to a point. But once your configuration file exceeds a few screens in length you're starting to go off the deep end. Your program's defaults should be minimal, sensible and secure, especially in the case of network daemons.

There are some hideous offenders out there like Asterisk, whose Christmas tree default configuration is often only lightly modified by novice administrators. A default RPM installation of Asterisk on my development virtual machine ships with 63 configuration files -- 7511 lines in total. But I run some perfectly good inbound SIP IVRs with 10 files and 251 lines.

When you throw a huge mess of a default configuration in my face, you leave me with the feeling that I can't even approach your software until I have had the time to digest the security implications of every one of the switches you are exposing.

There are other programs which do it well like OpenVPN. They ship sample configuration files for different configurations, from which you can copy and paste your own configuration files together. This approach is much saner than editing a huge file -- take what you need, leave what you don't.

I advise all system administrators faced with such configuration mountains to grit their teeth and write their own configs from scratch after carefully studying the stock configuration. Turn on and configure only the specific features you need, lightly document your intent with comments, and leave the other garbage out of the configuration files. The more scrolling past heaps of irrelevant comments and settings you must do to scan the configuration file, the less you will be able to focus on the big picture of how your system is set up.

Corosync::CPG

| No TrackBacks

I'm working on releasing a small piece of useful Perl XS module code to the world... Corosync::CPG. This allows you to use the corosync cluster stack's reliable, ordered multicast messaging bus from within Perl.

I have applied for a PAUSE id and plan on submitting this module to CPAN as well. For now, this is a super-early alpha. It works for me, but the POD documentation is incomplete and there is obviously no warranty.

The terms of this release are the same licensing terms as Perl itself (GPL/Artistic).

Ksplice Review

| No TrackBacks

We recently signed up for Ksplice, a service offering live Linux kernel updates. (Yes, live means while the kernel is running.)

Now, the kinds of updates we are talking about are patches to running code intended to apply critical security fixes. Ksplice won't let you upgrade from 2.6.34 to 2.6.35. Nevertheless, for mission critical servers (especially tough ones like database servers, routers and the like), the prospect of not having to reboot to install security updates is a huge win.

LWN looked at Ksplice, for those who are interested in knowing how it works. (Even if you're not a kernel programmer, you can learn so much from watching the kernel programmers at work.)

Ksplice is a young company but has racked up an impressive list of clients. I've found their solution easy to use. Billing and support is straightforward, and they're very friendly people. We've had no problems applying the updates. I would recommend Ksplice to anyone looking to keep their production GNU/Linux systems up to date.

daemontools patches

| No TrackBacks

I've been relying extensively on daemontools to manage services on my production servers for years. There are newer entries into the arena like runit and even upstart which replaces init. I'm not ready to replace daemontools on any of my servers just yet; its simple design is something I've come to depend on.

All that being said, I'm like many of the other users of djb code: small patches here and there make things better. So in that spirit, here are two of my contributions.

daemontools-0.76-readproctitle-clear-on-sigusr1.patch
Causes readproctitle to clear its buffer (reset it to periods) when it receives SIGUSR1. It's useful if you're setting up new services and you need to see if a certain error has really been fixed.

diff -Nru a/admin/daemontools-0.76/src/readproctitle.c b/admin/daemontools-0.76/src/readproctitle.c
--- a/admin/daemontools-0.76/src/readproctitle.c        2001-07-12 11:49:49.000000000 -0500
+++ b/admin/daemontools-0.76/src/readproctitle.c        2010-09-06 16:09:28.000000000 -0500
@@ -1,10 +1,21 @@
+#include <signal.h>
 #include <unistd.h>
 #include "error.h"

+static char *buf;
+static unsigned int len;
+
+static void
+clear_line(int sig)
+{
+  int i;
+  for (i = 0;i < len;i++) {
+    buf[i] = '.';
+  }
+}
+
 int main(int argc,char **argv)
 {
-  char *buf;
-  unsigned int len;
   int i;
   char ch;

@@ -14,6 +25,8 @@
   while (buf[len]) buf[len++] = '.';
   if (len < 5) _exit(100);

+  signal(SIGUSR1, clear_line);
+
   for (;;)
     switch(read(0,&ch,1)) {
       case 1:

daemontools-0.76-setuidgid-initgroups.patch
Causes setuidgid to initialize the supplementary groups for the user account it is changing to.

diff -Nru a/admin/daemontools-0.76/src/setuidgid.c b/admin/daemontools-0.76/src/setuidgid.c
--- a/admin/daemontools-0.76/src/setuidgid.c    2001-07-12 11:49:49.000000000 -0500
+++ b/admin/daemontools-0.76/src/setuidgid.c    2010-06-22 16:06:05.000000000 -0500
@@ -21,6 +21,8 @@

   if (prot_gid(pw->pw_gid) == -1)
     strerr_die2sys(111,FATAL,"unable to setgid: ");
+  if (initgroups(pw->pw_name, pw->pw_gid))
+    strerr_die2sys(111,FATAL,"unable to initgroup: ");
   if (prot_uid(pw->pw_uid) == -1)
     strerr_die2sys(111,FATAL,"unable to setuid: ");