404 fun..

Found on the web:

404 File Not Found

I ate your Web page.
Forgive me. It was juicy

And tart on my tongue.

Translation:

The URL you requested could not be found on this server.

Some people! Some people’s web pages…

Bad, bad mailserver!

I had a client call me to day with a catastrophic failure of their Cyrus IMAP server running on Panther. They were getting this error in their Cyrus startup log:

DBERROR db4: 
    fatal region error detected; run recovery
DBERROR: dbenv->open '/var/imap/db' failed:  
DB_RUNRECOVERY: 
    Fatal error, run database recovery
    Fatal error: can't initialize mboxlist 
    environment
DBERROR db4: 
    fatal region error detected; run recovery
DBERROR: dbenv->open '/var/imap/db' failed: 
DB_RUNRECOVERY: 
    Fatal error, run Fatal error: can't 
    initialize mboxlist environment

Trying to reconstruct resulted in this error:

reconstruct:
    can't initialize mboxlist environment

Some searching in the Apple discussion forums led to this thread: 10.3.2 Mail Server: error on RECONSTRUCT which contains a fix by Dr. Fishopolis.

Since the Apple discussion forums are so hopelessly pokey right now I’ve decided to document the fix here (so I can find it again if I need it 🙂

Before you start: These instructions assume that the Cyrus mail store is in its default location: /var/spool/imap, and hasn’t been moved somewhere else (like to the XRAID for instance). Check the location in: Server Admin -> Mail -> Settings -> Advanced -> Mail store and replace any mention of /var/spool/imap with your sepecific path.

  • Become root for a while:
    sudo -s
  • Backup the /var/imap/ and var/spool/imap directories:
    ditto /var/imap/ /var/imap.bak/
    ditto /var/spool/imap/ /var/imap.spool.bak/
  • Blow away the /var/imap/ directory:
    rm -r /var/imap
  • And recreate it:
    /usr/bin/cyrus/tools/mkimap
  • Correct the permissions:
    chown -R cyrus /var/imap
    chgrp -R mail /var/imap
  • If the cyrus user doesn’t have a shell specified, do it now:
    niutil -createprop . /users/cyrus shell /bin/bash
  • Become the cyrus user:
    su cyrus
  • Now run the reconstruct command:
    /usr/bin/cyrus/bin/reconstruct
  • You should see cyrus work through your mail accounts like this:
    user/user1
    user/user1/Folder1
    user/user1/Folder2
    user/user2
    user/user2/Folder1
    user/user2/Folder2
  • If any of your users are missing personal mail folders reconstruct them individually using the account reference above (ie: user/user1) – You may need to run this command multiple times if the user has nested personal folders. Just run it until it stops producing errors:
    /usr/bin/cyrus/bin/reconstruct -r -f user/username
  • Finally, stop being cyrus, then stop being root:
    exit
    exit

There, now wasn’t that fun.

Follow-up 1: There is associated problem pertaining to log files which is documented here.

Follow-up 2: Apparently there is an undocumented flag for the reconstruct command in the Panther version of Cyrus: "-i". It is supposed to rebuild nested user folders. I haven’t tested it (because I don’t have a broken email server right now :-)) but it is being recommended by Apple support.

Follow-up 3: Apple has posted a knowledge base article on the subject, read about it here.

Follow-up 4: Could it be we’ve discovered the ultimate fix for this? Read on.

Update day

(kinda like a snow day, only… not)

So, I’ve installed OS X 10.3.3, and WebObjects 5.2.3, and every thing is just working swimingly…

Happy, happy, happy, happy, I’m so happy

Whoo hoo! WebObjects 5.2.3 is out!

According to the technote this update offers a number of fixes:

This update address issues with CLOSE_WAIT states in deployment using JavaMonitor and wotaskd. It also addresses a number of issues related to EOF under high load.

But most importantly it is fully qualified for development and deployment on Panther with Java 1.4.2!

Now all I need is qualified version of Project WONDER..

UPDATE: As noted by Art Isbell on the WebObjects-dev list this release requires Java 1.4.2… and the installer fails with an erroneous error message: "This update can be installed only on Mac OS X 10.3.3 or later." if you are trying to install on Panther with Java 1.4.1.