August 2012 Archives

How to Fuck Up a (Re)launch

| No TrackBacks

My credit union recently decided to redo its online banking portal. I want to like it, but the first taste in my mouth is bitter.

On Tuesday, the old portal was canned, and they posted an announcement that logins would be disabled as they worked on the upgrade. Indeed, I was unable to access my banking portal until mid-afternoon Wednesday.

Upon attempting to log in, I fed in the same credentials I had been using for years. The credentials were immediately rejected, so I tried again, assuming I must have fat-fingered my password.

Instead of getting in, the system locked me out of my account. I started digging through their FAQ which was helpfully linked all over the place. The FAQ implied something about a temporary password but did not specify what it might be.

When I returned to the home page, I noticed an "Alert" icon, and upon hovering over it, got advised that my temporary password was the last four digits of my social security number and my birth year.

As a developer who put time into smoothing over relaunches before, I was a bit miffed. It's perfectly possible to check old passwords (even when you properly implemented salting and hashing). It's also perfectly possible to force the user to change their password on first login into your new system, so that you can store it in whatever new database table you want, using whatever new encoding scheme you like.

Put aside the fact that their new system couldn't validate my old login credentials. They overwrote everyone's password with the last four digits of their social security number, combined with their birth year. As if either string is really a secret. Everyone on the planet asks you for the last four digits of your social security number, or in some cases, the full number. If a user is not actively participating in online banking, will these temporary passwords ever expire?

Of course, I couldn't access my account using the new-found information, because two password attempts was enough to lock me out of my account. And again with their helpful suggestion: either use the "Forgot Password" mechanism to unlock your account, or contact support.

When I tried the former, I was given the option to validate my identity by SMS or a phone call. In both cases the first 6 digits of the phone number were masked (eg, XXX-XXX-1234), and in both cases, I didn't recognize the trailing 4 digits. Now I was concerned about how my credit union had two bogus phone numbers linked to my account.

So I called support, and waited on hold for half an hour. The support person was friendly enough, especially given the fact he must be taking hundreds of similar calls. My account was unlocked, and again I was advised that a new temporary password had been issued.

I made sure to try and plug this password in before getting off the phone. If something didn't work out, I did not want to sit on hold again.

This time, the portal urged me "for my security" to close my browser and re-open it, because I had an "active banking session". Sounds like someone has a nasty cookie bug and decided to paper over it with some baloney about how secure they are being.

No matter, I elected to log in on the laptop. Seeing that I had finally made progress, I bid goodbye to the support person and moved on. Now I was being required to set a new password, which had sensible requirements for character classes but had to be between 8 and 12 characters. Why on earth would you limit my online banking password to 12 characters?

At the next screen, I was forced to change my user ID. It used to be a member number, but now I had to pick a user ID with no special characters, but with a mandatory number. "For my security."

The next screen had me entering phone numbers and other means to restore access to my account in the event I lose my password. They asked me to fill out some dreaded security questions. No one could ever Google my mother's maiden name. I'm feeling really secure!

After finally completing the new registration process, I was booted back to a login screen and asked to use my new credentials. Twice, I tried, and twice, the credentials I had just created failed to allow me in to the system. Again, my account was locked out.

Thankfully, on my second attempt at dealing with their garbage, the system decided to remember my password. And it even let me log in!

Of course, a credit union portal is only useful if my account balances, transaction history, or other banking services are available. The landing screen was full of promise -- "New mobile app!" "Track your spending!" But none of it worked. All of the informational views contained errors.

The portal is still fucked as I speak. I actually wanted to know my balance, so I decided to try the telephone access system, which I used frequently in prior years when the online banking portal wasn't close at hand. Turns out that has also been overhauled, and that doesn't work either.

I got my last laugh when I landed at a new login screen after my session timed out. Never mind the fact that they have so many different login screens (excellent for training your users to be cautious of phishing attempts). It turns out I should be using Internet Explorer, Firefox, or Safari "For my security". I suppose I'll just toss this Chrome garbage for IE. You know, "For my security".

Hey - maybe their new system is awesome. After spending an hour just trying to get in, it sure would be great to find out. In the mean time, I can't help but think "For your security" is the newest excuse of the mighty Bastard Operator From Hell.

Update 08/24/2012

I don't always specifically name the subject of any of my grievances, since I try to use my criticism constructively, if only as an example of what not to do. But Texans Credit Union has fucked up too badly for me to keep my mouth shut. It is now Friday and after 4 business days of being completely unavailable, the Internet banking portal is still offline, and I still can't access the account information hotline. To make matters worse, both the main customer service number, and the account information hotline, are currently offline. That is to say: they aren't taking calls. The phone company plays a prompt as if their telco equipment isn't even acknowledging the call attempts. No ringback.

I know, I know.

I'm preaching to the crowd. But I suspect I may not be preaching to the converted.

Everyone has heard the mantra since the beginning of their computing lives: back up your important data! But it's incredible to me how many computer literate people still get it wrong. Even when they give the right advice to their novice friends, they fail to implement the correct strategies in practice.

When I was a teenager, I accidentally ruined a friend's semester-long research project. I was helping him install a new hard drive. He didn't need my help, but back then it was still really exciting to get a new drive, to crack open the case and to install new gear. He decided to trust my butterfingers to the task of reconnecting power to his old drive. Hard drive molex power connectors are supposed to be impossible to insert backwards, but I couldn't see into his case to monitor what I was doing, and managed to connect it backwards anyway.

He had no backups, and after paying hundreds to a data recovery expert, he never recovered his project. Sorry David!

More recently, a friend encrypted his laptop hard drive and discovered that he could use Unicode characters in the password. The only problem was that while the UI accepted those characters in the screen where your password is set, it did not accept those characters on the screen where you decrypt and log in. Whoops.

This weekend, my RAID array finally decided to crap out. I'd been operating a RAID 10 made of 4 Seagate 1 TB drives. All of the drives had been through at least one RMA cycle. (They don't make these things as reliable as they used to. But it's definitely not just Seagate.) After dealing with rounds of RMAs, I got lazy and ran the array in degraded mode for months. When you lose drives in a RAID 10, it effectively becomes a RAID 0. A RAID 0 is more dangerous than a bare hard drive, since a failure of any single disk in the stripe will destroy the array.

I'm not totally irresponsible, mind you. I have nightly automatic backups of my most critical data, courtesy of and Duplicity. I even tested the backups after setting them up.

But what I did not test is that all of my expected files were present in the backup. I misconfigured Duplicity and caused it to ignore my most critical directories - those with the source code of everything I've been working on for many years.

Many copies of these projects exist at different places - work, GitHub, etc. But there were some critical projects that were either not up to date, or not mirrored anywhere at all.

You can imagine my swearing and my feelings of panic. Thankfully, after I calmed down and carefully reassembled the RAID array, I was able to recover everything of importance. Nowadays, Linux is pretty resilient when reading from failing media.

Always back up your important data. Do it regularly. Back it up in more than one way. Don't assume your RAID array will save your data. Don't assume your backups will work, or will contain everything you need. Don't run your RAID arrays in degraded mode, don't let your backup process fail and not fix it, and don't forget to test your backups!

Implementing backups is boring. Testing backups is really boring. But some day you might be really glad you did.