ChaseVenters.org

Migrating from old krb5/NFSv4 to newer krb5/NFSv4

2011-11-11T06:17:08Z

I thought I'd post a quick tip for anyone upgrading a set of clients in a kerberized NFSv4 network. I'm in the process of pushing out CentOS 6 to a cluster currently supported by NFSv4 on CentOS 5 and my standard "setup krb5/nfsv4 client" script didn't leave me with a working client. Instead, I got this error on the NFS server every time I attempted the NFS mount:

gss_kerberos_mech: unsupported algorithm 6

gss_kerberos_mech: unsupported algorithm 23

Some advice pointed out that the keytab might need to be written out without the newer key types, but attempting to limit to des-cbc-crc did not fix the problem.

Instead, I found that the following settings in the [libdefaults] section of /etc/krb5.conf fixed my environment:

[libdefaults]
 # cventers: These overrides are TEMPORARY until we have abandoned CentOS 5
 default_tgs_enctypes = des-cbc-md5 des-cbc-crc arcfour-hmac-md5 arcfour-hmac-exp
 default_tkt_enctypes = des-cbc-md5 des-cbc-crc arcfour-hmac-md5 arcfour-hmac-exp
 permitted_enctypes = des-cbc-md5 des-cbc-crc arcfour-hmac-md5 arcfour-hmac-exp
 allow_weak_crypto = true

qpsmtpd plugins

2011-04-07T00:24:48Z

As part of deploying a new Postfix-and-qpsmtpd based mail architecture at work, I have developed some qpsmtpd plugins and extended its native queue/smtp-forward plugin.

filter/dkimsign: Signs e-mail using Mail::DKIM. There are a other dkimsign plugins out there but I wanted to take a stab at doing one myself.
filter/header_whitelist: Possibly controversial, could break many things if misused. I wanted a way to clean up all the extra garbage version headers, etc added by the multitude of scripts and platforms generating email in our environment. If the mere existence of this plugin doesn't violate RFC2822 or e-mail best practices, certain configurations certainly would. Use with care.
queue/smtp-forward: I have extended the stock plugin to support the Postfix XCLIENT verb. This allows a qpsmtpd to pass information about the client (their IP and HELO, in particular) which Postfix can then use for access control and/or logging. I'll try and submit this back upstream.

You can find the plugins at my GitHub page.

Southwest In-Flight Wifi Review

2010-11-19T02:14:51Z

I just took a Southwest Airlines flight that was wifi enabled. I couldn't resist the temptation to give the wifi a spin.

My review, in a nutshell? It costs $5, it appears to work with (at least) HTTP(S), Outlook, and ssh... but the performance leaves something to be desired.

The Wifi gateway also appears to mangle HTML passing through it in order to display a Southwest airlines banner over the top of the pages. This does provide some function -- it indicates your current altitude and ETA. But they could have included a way to disable the behavior.

The bottom line? It works, and that much is neat. But given the performance limitations (and keep in mind this is just one data point), I doubt its utility for anything but basic surfing. But on a long flight, I don't think books or magazines could make the time pass by with such ease.

VIM Plugin: Makesd/Makecsd

2010-09-24T12:32:50Z

I wanted to share a little VIM plugin I just got done writing, makesd.vim. This plugin is pretty straightforward, and is adapted from a couple of Perl command-line scripts I tend to haul around called makesd and makecsd.

In short, they produce clean looking separators for use inside source code:

:Makesd "Public Interface"
# ========================================================================== #
# ============================ Public Interface ============================ #
# ========================================================================== #

:Makecsd "Public Interface"
/* ======================================================================== */
/* =========================== Public Interface =========================== */
/* ======================================================================== */

It's my first VIM script. VIM scripting is pretty easy -- give it a try!

Brain Damage

2010-09-23T18:35:34Z

Another one for the record books. Once again, the kind folks at Redmond have proven how truly incompetent they are, and why at the end of the day, a power user can only survive on an open source platform.

Microsoft has stripped the ability to save raw e-mail messages from Outlook 2007. Supposedly this capability exists in Outlook Express and/or Windows Mail. But there are mountains of bad advice suggesting that export to those programs, followed by an import, or an export to .txt, is an acceptable alternative. (It isn't, in all cases, the transport headers aren't included. Who knows how else Microsoft is molesting the message).

Commercial solutions exist - one for only $14 (are you kidding me?) and one for over $60 (are you kidding me???)

transocks_ev Patch: DNS, Performance, Reliability, Logging

2010-09-20T23:39:00Z

I've put out a new transocks_ev patch transocks_ev-performance-reliability-dns-logging.patch. transocks_ev is a neat little program by Bernd Holzmueller at tiggersWelt.net that uses the Linux netfilter/iptables stack to intercept outgoing TCP connections and transparently convert them into SOCKS5 proxy connections. It's based on transocks which does the same thing. transocks uses a forking model, while transocks_ev uses libevent to multiplex connections in a single process's event loop.

I'm planning on using these changes to transparently intercept outgoing Postfix SMTP connections on some backend mail servers and to use DNS-based load balancing to fan those connections out across multiple proxy servers/Internet connections.

In addition, I've improved the logging support of transocks_ev, giving it three levels of verbosity with basic statistics collection for the connections. All blocking operations have been converted to non-block (connect(), write(), the new DNS). The code is using libevent bufferevent to manage low level socket access.

libsoup Ignores DNS TTLs

2010-09-19T01:41:03Z

I've been using libsoup to run a small SOAP engine for one of the back-office programs I maintain. We've recently upgraded to a new load-balanced architecture, and we are using DNS-based load balancing to fan these SOAP requests out across our servers.

It only took a few days in production to realize that libsoup was doing something nasty. Prior to any HTTP request, you need to create a SoupSession object. This object manages things like connection pools / keepalive. It contains a GHashTable called hosts, which it uses as a cache of connections to a given hostname.

/* Requires host_lock to be locked */
static SoupSessionHost *
get_host_for_uri (SoupSession *session, SoupURI *uri)
{
    SoupSessionPrivate *priv = SOUP_SESSION_GET_PRIVATE (session);
    SoupSessionHost *host;

    host = g_hash_table_lookup (priv->hosts, uri);
    if (host)
        return host;

    host = soup_session_host_new (session, uri);
    g_hash_table_insert (priv->hosts, host->uri, host);

    return host;
}

Unfortunately, entries in this hash table are never removed or expired unless the SoupSession object itself goes away. This sucks for a few reasons:

DNS TTL values are ignored. Instead, the result of the DNS query is cached forever. Obviously this means that if the record is ever changed, libsoup clients need to be restarted to know about it.
DNS load balancing is broken by libsoup, which will repeatedly connect to the same IP address regardless of whether multiple IPs are included in the response to an A query.
You really wouldn't want to write a robot or some other long lived program that would make lots of connections to lots of different hosts using libsoup, as it stands. Aside from the obvious correctness issues listed above, the hosts hash table will experience unbounded growth. Thankfully all of our connections are to the same small set of URLs and hostnames.

I'm not sure how easy it would be to patch libsoup to behave correctly. As far as I can tell the GResolver that libsoup relies on doesn't even report TTLs.

Given the nature of this bug I can only see a few workarounds:

Set the Host HTTP header yourself, do the DNS query yourself using GResolver, and supply the server's IP address to the SoupURI instead of a hostname. This breaks SSL certificate validation.
Recycle/create the SoupSession per-request. This breaks keepalive/connection pooling and has obvious overhead issues.

Given the nature of how I'm using libsoup, I chose the latter option. YMMV.

asterisk-func_dns: Asterisk 1.4 DNS() dialplan function

2010-09-13T08:07:52Z

I'm doing another small code release. This one is asterisk-func_dns, a dialplan function DNS() to do an explicit DNS query without requiring you to launch an external program. It's an alpha release currently intended for Asterisk 1.4 and only supporting IPv4 / DNS A record types.

I'm using this to implement DNS-based load-balancing for outgoing calls across a series of proxies and internet connections.

In my dialplan, I request the IP addresses of my proxy servers ahead of any attempt to Dial(). This module returns the list of IP addresses published in the record, separated by commas. This allows me to sequentially fork across the proxy servers, and since I don't need to rely on Dial()'s forking support, I can add additional processing in between attempts. Since I obtain the proxy set by looking up a single DNS name, my Asterisk dialer configurations do not have to change if I add more proxy servers to my network, also meaning that those Asterisk dialers will not waste time trying to contact outbound proxy servers that have gone offline for maintenance or due to a failure. Each Asterisk dialer will try every call amongst all the working proxy servers, up to one attempt each, in a random order.

This code could benefit from some obvious todos: forward port to modern Asterisk versions, implementation of the ability to grab other record types like SRV or AAAA, etc. I may address these eventually but at the moment this is "good enough for me". I release this code (under the same Asterisk licensing terms: GPLv2) with the hopes that someone finds it useful.

Christmas Tree Configuration Files

2010-09-09T11:30:00Z

Okay, I'm pleading with developers. I'm very impressed at the number of options and switches that your program exposes via its configuration file(s) / directories / databases. Bonus points for those of you who have managed to extensively document each switch and its default setting with inline comments.

Actually, that strategy even works up to a point. But once your configuration file exceeds a few screens in length you're starting to go off the deep end. Your program's defaults should be minimal, sensible and secure, especially in the case of network daemons.

There are some hideous offenders out there like Asterisk, whose Christmas tree default configuration is often only lightly modified by novice administrators. A default RPM installation of Asterisk on my development virtual machine ships with 63 configuration files -- 7511 lines in total. But I run some perfectly good inbound SIP IVRs with 10 files and 251 lines.

When you throw a huge mess of a default configuration in my face, you leave me with the feeling that I can't even approach your software until I have had the time to digest the security implications of every one of the switches you are exposing.

There are other programs which do it well like OpenVPN. They ship sample configuration files for different configurations, from which you can copy and paste your own configuration files together. This approach is much saner than editing a huge file -- take what you need, leave what you don't.

I advise all system administrators faced with such configuration mountains to grit their teeth and write their own configs from scratch after carefully studying the stock configuration. Turn on and configure only the specific features you need, lightly document your intent with comments, and leave the other garbage out of the configuration files. The more scrolling past heaps of irrelevant comments and settings you must do to scan the configuration file, the less you will be able to focus on the big picture of how your system is set up.

Corosync::CPG

2010-09-08T09:28:12Z

I'm working on releasing a small piece of useful Perl XS module code to the world... Corosync::CPG. This allows you to use the corosync cluster stack's reliable, ordered multicast messaging bus from within Perl.

I have applied for a PAUSE id and plan on submitting this module to CPAN as well. For now, this is a super-early alpha. It works for me, but the POD documentation is incomplete and there is obviously no warranty.

The terms of this release are the same licensing terms as Perl itself (GPL/Artistic).

Ksplice Review

2010-09-08T07:39:56Z

We recently signed up for Ksplice, a service offering live Linux kernel updates. (Yes, live means while the kernel is running.)

Now, the kinds of updates we are talking about are patches to running code intended to apply critical security fixes. Ksplice won't let you upgrade from 2.6.34 to 2.6.35. Nevertheless, for mission critical servers (especially tough ones like database servers, routers and the like), the prospect of not having to reboot to install security updates is a huge win.

LWN looked at Ksplice, for those who are interested in knowing how it works. (Even if you're not a kernel programmer, you can learn so much from watching the kernel programmers at work.)

Ksplice is a young company but has racked up an impressive list of clients. I've found their solution easy to use. Billing and support is straightforward, and they're very friendly people. We've had no problems applying the updates. I would recommend Ksplice to anyone looking to keep their production GNU/Linux systems up to date.

daemontools patches

2010-09-08T07:06:32Z

I've been relying extensively on daemontools to manage services on my production servers for years. There are newer entries into the arena like runit and even upstart which replaces init. I'm not ready to replace daemontools on any of my servers just yet; its simple design is something I've come to depend on.

All that being said, I'm like many of the other users of djb code: small patches here and there make things better. So in that spirit, here are two of my contributions.

daemontools-0.76-readproctitle-clear-on-sigusr1.patch
Causes readproctitle to clear its buffer (reset it to periods) when it receives SIGUSR1. It's useful if you're setting up new services and you need to see if a certain error has really been fixed.

diff -Nru a/admin/daemontools-0.76/src/readproctitle.c b/admin/daemontools-0.76/src/readproctitle.c
--- a/admin/daemontools-0.76/src/readproctitle.c        2001-07-12 11:49:49.000000000 -0500
+++ b/admin/daemontools-0.76/src/readproctitle.c        2010-09-06 16:09:28.000000000 -0500
@@ -1,10 +1,21 @@
+#include 
 #include 
 #include "error.h"

+static char *buf;
+static unsigned int len;
+
+static void
+clear_line(int sig)
+{
+  int i;
+  for (i = 0;i < len;i++) {
+    buf[i] = '.';
+  }
+}
+
 int main(int argc,char **argv)
 {
-  char *buf;
-  unsigned int len;
   int i;
   char ch;

@@ -14,6 +25,8 @@
   while (buf[len]) buf[len++] = '.';
   if (len < 5) _exit(100);

+  signal(SIGUSR1, clear_line);
+
   for (;;)
     switch(read(0,&ch,1)) {
       case 1:

daemontools-0.76-setuidgid-initgroups.patch
Causes setuidgid to initialize the supplementary groups for the user account it is changing to.

diff -Nru a/admin/daemontools-0.76/src/setuidgid.c b/admin/daemontools-0.76/src/setuidgid.c
--- a/admin/daemontools-0.76/src/setuidgid.c    2001-07-12 11:49:49.000000000 -0500
+++ b/admin/daemontools-0.76/src/setuidgid.c    2010-06-22 16:06:05.000000000 -0500
@@ -21,6 +21,8 @@

   if (prot_gid(pw->pw_gid) == -1)
     strerr_die2sys(111,FATAL,"unable to setgid: ");
+  if (initgroups(pw->pw_name, pw->pw_gid))
+    strerr_die2sys(111,FATAL,"unable to initgroup: ");
   if (prot_uid(pw->pw_uid) == -1)
     strerr_die2sys(111,FATAL,"unable to setuid: ");

NSIS Plugin Released: Pwgen

2009-10-09T18:32:41Z

I've been working on an installer for a Windows service using Nullsoft Scriptable Install System. This installer creates a special account to run the service under, but to do so, it has to come up with a random password. NSIS offers a few existing mechanisms for random numbers, but neither are anything but basic PRNGs.

In order to get *good* random passwords, we want the equivalent of /dev/urandom. Microsoft provides the CryptoAPI, which includes the CryptGenRandom() API. I developed a DLL plugin for NSIS called Pwgen that collects entropy from the OS and generates a random password, restricted to the 62-character set of alphanumerics.

You can download pwgen-001 directly from me. I also recommend the official Pwgen plug-in wiki page on the NSIS wiki and the discussion thread on the Winamp NSIS forum.

Nokia Further Open-Sources Qt

2009-01-17T16:33:19Z

Nokia has announced that they are going to add an additional license to the QPL/GPLv2/GPLv3/Commercial lineup: LGPL v2.1. This is excellent news for the toolkit as it will lead to wider adoption (and more improvements to the core toolkit).

I love Qt, because when I'm paid to write software to run on Microsoft Windows, I write that software on Linux, compile it on Linux, test it on Linux, and then the last step is to cross-compile it to Windows and test it inside a kvm virtual machine. I barely have to touch Windows with a 10 foot pole.

I'm not extremely fond of C++, but Qt makes C++ tolerable. The toolkit is quite honestly miles beyond Gtk+, a statement I make having written a Gtk+ app before, a number of glib apps, and being very fond of C. They're now shipping Webkit and an XQuery processor, and your application scan be styled (on the fly) with CSS.

Urge to Punch Rising...

2007-09-15T21:46:42Z

People often find technology to be incredibly frustrating. For many, it's a matter of the difficulty in using technology, or the ways in which it misbehaves... but for me, the frustration often bubbles up when I come across technology that could and indeed should have been better.

Presently, my primary frustration is with the garage door opener market. There have been many technologies and protocols used by wireless garage door openers over the last decades. One of the consequences of this is vast incompatibility.

This bit me in the ass once when I cluelessly purchased a Genie-brand remote to open a Chamberlain Security+ door. That was actually a while ago, but I've now got a useless Genie remote. I ended up with the proper Chamberlain Security+ remote. It's a bulky unit, which is frustrating when space in my Miata is at a premium.

I moved to a new apartment recently. The garage door opener in this apartment is a LiftMaster, also bearing the Chamberlain name. Pleased that I wouldn't need to buy a new controller, I attempted to link my remote to the door with no success. Apparently, the "billion code" technology in this unit is obsolete.

Frustrated, I tried Home Depot, hoping to find a decent controller that would open both the new garage door and the old one. My purchasing choices were limited between mini-remotes (in a form factor I would much prefer) that wouldn't open the apartment garage door, equally bulky units that would only open the apartment garage door and not the old one, or an absolutely large "Clicker" with two buttons that was allegedly compatible with both.

I had the Clicker in my hand, about to give in and fork over the $20, but I was too disgusted with the size of the unit to make the purchase. I reasoned that I must be able to find a better unit on the Internet.

Unfortunately, it looks like I'm not in luck -- not in the slightest. Just shopping for an opener is a challenge -- the brand names, the color-coded learning buttons, frequencies, and manufacturing years all create a muddled mess. And just in case the shopping experience doesn't make you want to vomit, you'll find that nearly all of the devices will.

At this point, I have to step back and catch my breath. It's a fucking garage door opener -- a very simple RF device! Why are they all so bulky? You could probably cram 2... maybe 4 embedded Linux systems in the Clicker. Actually, with some of the "Linux in an ethernet connector" technology, you might be able to get 8 or 10 in there.

It's also gross that the landscape is littered with incompatible devices. I'm particularly surprised that Chamberlain doesn't sell a reasonably-sized opener that will open both their new and old doors. Perhaps what's holding back the market for a sane universal remote is that companies like Chamberlain would rather file disgustingly frivilous lawsuits against competitors making compatible openers than do any kind of real innovation themselves.

At times like this, I wish the landscape could all be blamed on one incompetent engineer or clueless manager - someone I could walk up to, then proceed to punch squarely in the face. But alas; I'm dealing with companies, industry and government. You can't punch a patent law and you can't punch proprietary "intellectual property." All you can do is hope that some day, the industry will behave more like the software industry is beginning to behave, by implementing open standards that benefit customer and the market alike.