DBERROR: skiplist recovery

[morgant]

I’m getting the following errors in mailaccess.log:
[code]Jun 19 10:03:03 postoffice imaps[755]: DBERROR: skiplist recovery: 4324 should be INORDER
Jun 19 10:03:03 postoffice imaps[755]: DBERROR: opening /var/imap/mailboxes.db: cyrusdb error[/code]
And this in system log:
[code]Jun 19 10:03:03 postoffice lmtpunix[756]: FATAL: can’t read mailboxes file
Jun 19 10:03:03 postoffice master[352]: service lmtpunix pid 756 in READY state: terminated abnormally[/code]
After some initial research (specifically: [url]http://www.irbs.net/internet/info-cyrus/0212/0245.html[/url]) it sounds like my mailboxes.db database got corrupted during a crash. I’m currently digging through logs to try to track down a crash.
Unfortunately, I [i]just[/i] spent about five hours backing up and upgrading this mail server from 10.4.2 to 10.4.6 on Friday night. I can restore the databases to Saturday night’s backups, but I’d like to try to recover this e-mail from Sunday and this morning first, if possible.
Could this be related to the upgrade at all? Has anyone experienced something like this? I don’t usually work with Berkely DB, so I have no idea how to go about recovering the database.
Obviously, this is [i]very bad[/i], so the sooner you can reply the better.

[morgant]

I’m currently trying to reconstruct the database using mailbfr (my mail server still takes a good 4 hours to back up its 12+GB of data, including about 30 minutes to run ‘du’ alone, so it’s been chugging away at this for a while now).
Having had some time to dig through my logs, specifically mail.log, mailaccess.log, and system.log, I’m still finding these (in system.log):
[code]Jun 19 08:05:24 postoffice lmtpunix[7621]: DBERROR: opening /var/imap/deliver.db: Cannot allocate memory
Jun 19 08:05:24 postoffice lmtpunix[7621]: DBERROR: opening /var/imap/deliver.db: cyrusdb error
Jun 19 08:05:24 postoffice lmtpunix[7621]: FATAL: lmtpd: unable to init duplicate delivery database
Jun 19 08:05:24 postoffice master[21962]: service lmtpunix pid 7621 in READY state: terminated abnormally
Jun 19 08:05:24 postoffice lmtpunix[7622]: DBERROR db4: Logging region out of memory; you may need to increase its size
Jun 19 08:05:24 postoffice lmtpunix[7622]: DBERROR: opening /var/imap/deliver.db: Cannot allocate memory
Jun 19 08:05:24 postoffice lmtpunix[7622]: DBERROR: opening /var/imap/deliver.db: cyrusdb error
Jun 19 08:05:24 postoffice lmtpunix[7622]: FATAL: lmtpd: unable to init duplicate delivery database
Jun 19 08:05:24 postoffice master[21962]: service lmtpunix pid 7622 in READY state: terminated abnormally
Jun 19 08:05:24 postoffice lmtpunix[7623]: DBERROR db4: Logging region out of memory; you may need to increase its size[/code]
Part of the reason I upgraded from 10.4.2 to 10.4.6 on Friday was in hopes of getting rid of this. For the last two weeks I’ve had the delivery queue balloon to 700-1000+ messages with my logs filled with these. On one occasion I discovered that one user alone had somehow spawned 100 IMAP processes (I keep a limit of 128).
Also, it appears that I was getting frequent crashes of lmtpd this morning and imapd crashed at one point as well (I’m assuming that’s when the mailboxes.db database got corrupted).
Has anyone else seen these errors? The mail server only has 1.5GB of RAM, so I’ll likely upgrade that tomorrow, but I would think that the mail server could function on 1.5GB or less of RAM (albeit slowly when under stress).

[morgant]

Of course, mailbfr saved the day and was able to reconstruct the database, but I’m still getting errors such as:
[code]Jun 19 18:02:51 postoffice imaps[2158]: SQUAT failed
Jun 19 18:02:51 postoffice imaps[2158]: SQUAT failed to open index file[/code]
According to [url=http://www.irbs.net/internet/info-cyrus/0502/0268.html]this[/url], that means it’s running out of file descriptors (or too many files are open). I’ve not seen those before, is that likely just because everyone’s mail clients are trying to re-sync to the server?
[b]UPDATE:[/b] According to “Re: cyrus connection timeout” [url=http://blog.gmane.org/gmane.mail.imap.cyrus/day=20051011]here[/url], these SQUAT errors may not be important.
Can you even up the number of open files in Mac OS X Server? I have a vague recollection that it’s hard-coded into Darwin at compile-time, so you’d have to rebuild the kernel (obviously not an option).

[morgant]

Josh,
Thanks for the correction, as usual. Exactly the info I was looking for.
Of course, the mail was recovered and the mail service has been back up for a few days now, I’m considering this issue closed (the other errors I was seeing are not necessarily related to the database having been corrupted).
The SQUAT errors went away once the server caught up on trying to re-index all of mailboxes. I have noticed that SQUAT indexing is turned off on all the mailboxes (atleast in SirAdmin.app, but I should probably verify in cyradm), so I’ll give sysctl a try and start re-enabling SQUAT indexing on the mailboxes (a few at a time).
Thanks again!

[Author]

Posts