Bad, bad mailserver!

I had a client call me to day with a catastrophic failure of their Cyrus IMAP server running on Panther. They were getting this error in their Cyrus startup log:

DBERROR db4: 
    fatal region error detected; run recovery
DBERROR: dbenv->open '/var/imap/db' failed:  
DB_RUNRECOVERY: 
    Fatal error, run database recovery
    Fatal error: can't initialize mboxlist 
    environment
DBERROR db4: 
    fatal region error detected; run recovery
DBERROR: dbenv->open '/var/imap/db' failed: 
DB_RUNRECOVERY: 
    Fatal error, run Fatal error: can't 
    initialize mboxlist environment

Trying to reconstruct resulted in this error:

reconstruct:
    can't initialize mboxlist environment

Some searching in the Apple discussion forums led to this thread: 10.3.2 Mail Server: error on RECONSTRUCT which contains a fix by Dr. Fishopolis.

Since the Apple discussion forums are so hopelessly pokey right now I’ve decided to document the fix here (so I can find it again if I need it 🙂

Before you start: These instructions assume that the Cyrus mail store is in its default location: /var/spool/imap, and hasn’t been moved somewhere else (like to the XRAID for instance). Check the location in: Server Admin -> Mail -> Settings -> Advanced -> Mail store and replace any mention of /var/spool/imap with your sepecific path.

  • Become root for a while:
    sudo -s
  • Backup the /var/imap/ and var/spool/imap directories:
    ditto /var/imap/ /var/imap.bak/
    ditto /var/spool/imap/ /var/imap.spool.bak/
  • Blow away the /var/imap/ directory:
    rm -r /var/imap
  • And recreate it:
    /usr/bin/cyrus/tools/mkimap
  • Correct the permissions:
    chown -R cyrus /var/imap
    chgrp -R mail /var/imap
  • If the cyrus user doesn’t have a shell specified, do it now:
    niutil -createprop . /users/cyrus shell /bin/bash
  • Become the cyrus user:
    su cyrus
  • Now run the reconstruct command:
    /usr/bin/cyrus/bin/reconstruct
  • You should see cyrus work through your mail accounts like this:
    user/user1
    user/user1/Folder1
    user/user1/Folder2
    user/user2
    user/user2/Folder1
    user/user2/Folder2
  • If any of your users are missing personal mail folders reconstruct them individually using the account reference above (ie: user/user1) – You may need to run this command multiple times if the user has nested personal folders. Just run it until it stops producing errors:
    /usr/bin/cyrus/bin/reconstruct -r -f user/username
  • Finally, stop being cyrus, then stop being root:
    exit
    exit

There, now wasn’t that fun.

Follow-up 1: There is associated problem pertaining to log files which is documented here.

Follow-up 2: Apparently there is an undocumented flag for the reconstruct command in the Panther version of Cyrus: "-i". It is supposed to rebuild nested user folders. I haven’t tested it (because I don’t have a broken email server right now :-)) but it is being recommended by Apple support.

Follow-up 3: Apple has posted a knowledge base article on the subject, read about it here.

Follow-up 4: Could it be we’ve discovered the ultimate fix for this? Read on.

One thought on “Bad, bad mailserver!

Comments are closed.