404 fun..

Found on the web:

404 File Not Found

I ate your Web page.
Forgive me. It was juicy

And tart on my tongue.


The URL you requested could not be found on this server.

Some people! Some people’s web pages…

Bad, bad mailserver!

I had a client call me to day with a catastrophic failure of their Cyrus IMAP server running on Panther. They were getting this error in their Cyrus startup log:

    fatal region error detected; run recovery
DBERROR: dbenv->open '/var/imap/db' failed:  
    Fatal error, run database recovery
    Fatal error: can't initialize mboxlist 
    fatal region error detected; run recovery
DBERROR: dbenv->open '/var/imap/db' failed: 
    Fatal error, run Fatal error: can't 
    initialize mboxlist environment

Trying to reconstruct resulted in this error:

    can't initialize mboxlist environment

Some searching in the Apple discussion forums led to this thread: 10.3.2 Mail Server: error on RECONSTRUCT which contains a fix by Dr. Fishopolis.

Since the Apple discussion forums are so hopelessly pokey right now I’ve decided to document the fix here (so I can find it again if I need it 🙂

Before you start: These instructions assume that the Cyrus mail store is in its default location: /var/spool/imap, and hasn’t been moved somewhere else (like to the XRAID for instance). Check the location in: Server Admin -> Mail -> Settings -> Advanced -> Mail store and replace any mention of /var/spool/imap with your sepecific path.

  • Become root for a while:
    sudo -s
  • Backup the /var/imap/ and var/spool/imap directories:
    ditto /var/imap/ /var/imap.bak/
    ditto /var/spool/imap/ /var/imap.spool.bak/
  • Blow away the /var/imap/ directory:
    rm -r /var/imap
  • And recreate it:
  • Correct the permissions:
    chown -R cyrus /var/imap
    chgrp -R mail /var/imap
  • If the cyrus user doesn’t have a shell specified, do it now:
    niutil -createprop . /users/cyrus shell /bin/bash
  • Become the cyrus user:
    su cyrus
  • Now run the reconstruct command:
  • You should see cyrus work through your mail accounts like this:
  • If any of your users are missing personal mail folders reconstruct them individually using the account reference above (ie: user/user1) – You may need to run this command multiple times if the user has nested personal folders. Just run it until it stops producing errors:
    /usr/bin/cyrus/bin/reconstruct -r -f user/username
  • Finally, stop being cyrus, then stop being root:

There, now wasn’t that fun.

Follow-up 1: There is associated problem pertaining to log files which is documented here.

Follow-up 2: Apparently there is an undocumented flag for the reconstruct command in the Panther version of Cyrus: "-i". It is supposed to rebuild nested user folders. I haven’t tested it (because I don’t have a broken email server right now :-)) but it is being recommended by Apple support.

Follow-up 3: Apple has posted a knowledge base article on the subject, read about it here.

Follow-up 4: Could it be we’ve discovered the ultimate fix for this? Read on.

Update day

(kinda like a snow day, only… not)

So, I’ve installed OS X 10.3.3, and WebObjects 5.2.3, and every thing is just working swimingly…

Happy, happy, happy, happy, I’m so happy

Whoo hoo! WebObjects 5.2.3 is out!

According to the technote this update offers a number of fixes:

This update address issues with CLOSE_WAIT states in deployment using JavaMonitor and wotaskd. It also addresses a number of issues related to EOF under high load.

But most importantly it is fully qualified for development and deployment on Panther with Java 1.4.2!

Now all I need is qualified version of Project WONDER..

UPDATE: As noted by Art Isbell on the WebObjects-dev list this release requires Java 1.4.2… and the installer fails with an erroneous error message: "This update can be installed only on Mac OS X 10.3.3 or later." if you are trying to install on Panther with Java 1.4.1.