A Humbling Upgrade
Yesterday I upgraded one of the servers I maintain from FreeBSD 4.11, which had reached its end of life, into the latest production release 6.2. It was a humbling experience.
The machine I upgraded supports many services. It provides Unix accounts with CVS and imap mail access, serves static web pages and wikis, runs a bug management database, is an authoritative and slave DNS server, runs mailing lists, sends CVS commit reports, and collects and reports weather statistics. The recommended way to upgrade from 4.x to 5.x and beyond is to "backup everything, reformat, reinstall, and restore everything". This turned out to be a lot more difficult than I anticipated.
In order to minimize service downtime, I setup a virtual machine, where, over a period of two weeks, I installed 6.2RC1 and all the software I required. I tested all the services I could test, and tuned settings that required updating. This was needed, because during the upgrade process I decided to move to newer versions of software like the Apache web server. I was unable to test some services, like the authoritative DNS server and the serial console, due to differences between the virtual and the actual setup. I also performed a test migration from the 4.11 to the virtual 6.2 machine, and drafted this as a transfer procedure I could execute step-by-step during the actual migration.
Given my preparations, I planned for the upgrade to take half a day,
but in the end it took about 20 hours, with some services restored
after about 10 hours.
As you can see from my notes above, what started (and should have continued)
as a series of neat ticks along my checklist, ended up as a frantic
hole-plugging exercise.
The procedure involved
- bringing both machines down to single-user mode,
- backing up the actual machine to tape,
- copying user data to the virtual machine, and
- restoring the virtual machine back to the freshly formatted actual machine.
Things went wrong from the beginning. To save some time, I planned to issue the first commands remotely from a serial console before commuting on-site. Distressingly, because on the previous day we attached a keyboard to the server, I lost touch with the machine when I brought it down to single-user mode. Other problems I encountered during the upgrade were DNS lookups that didn't work in single-user mode, and a partition editor that refused to write my data to disk giving me a write error.
However, the worst problems happened after the upgrade, when I brought up the server under 6.2 and began testing the various services. These are the problems I encountered (and fixed):
- Some user binaries linked to missing 4.x libraries
- Named would react with "connection refused" to queries (I did not comment out the line
listen-on { 127.0.0.1; };) - Libraries were missing for some Perl scripts (URI, HTTP) were causing them to abort
- Stricter taint checks in Perl scripts
- Awk scripts failed when invoking GNU awk functions that are not supported in the 6.2 BSD awk (gensub, strftime)
- Many ports required upgrading, from the time I setup the vritual machine to the time I had reliable on-site access to do the upgrade
- The UPS notification mechanism required additional tuning
- The serial console device had changed
I also forgot to transfer:
- the root home directory
- user crontabs
- user mail files
- the network time protocol configuration file (
/etc/ntpd.conf) - the kernel configuration file
- the system controls (
/etc/sysctl.conf), and - the daily periodic tasks (
/etc/daily.local).
