Mail having IO errors, tied to LDAP. At wits end.

[esmith]

Our mail server is crashing about once a day, give or take. The console log will typically look like this prior to, then during the crash. Server is a Quad Xeon at 10.4.10

[code]Sep 25 16:58:53 zappa ntpd[328]: sendto(17.254.0.31): Bad file descriptor
Sep 25 17:17:57 zappa DirectoryService[82]: Search connection failure: During an attempt to bind to [127.0.0.1] LDAP server.
Sep 25 17:17:57 zappa DirectoryService[82]: Search connection failure: Disabled future attempts to bind to [127.0.0.1] LDAP server for next 0 seconds.
Sep 25 17:19:19 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:19 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:19 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:19 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:19 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:19 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:19 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:20 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:20 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:20 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:20 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:20 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:20 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:20 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:20 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:20 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:20 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 25 17:19:20 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
2007-09-25 17:20:59.386 Server Admin[11887] Server got an empty response (servermgr_info may have crashed)
2007-09-25 17:20:59.386 Server Admin[11887] Disabling multithreaded transactions for zappa.local
2007-09-25 17:20:59.386 Server Admin[11887] Server got an empty response (servermgr_info relay may have crashed)
Sep 25 17:21:20 zappa DirectoryService[82]: Search connection failure: During an attempt to bind to [127.0.0.1] LDAP server.
Sep 25 17:21:20 zappa DirectoryService[82]: Search connection failure: Disabled future attempts to bind to [127.0.0.1] LDAP server for next 0 seconds.
Sep 25 17:21:50 zappa imaps[29671]: AOD: authentication error: cannot find user: john (0)
DSOpenNode(): dsOpenDirNode(“/LDAPv3/127.0.0.1”) == -14002
Sep 25 17:22:35 zappa /System/Library/CoreServices/mcxd.app/Contents/MacOS/mcxd: DSOpenNode(): dsOpenDirNode(“/LDAPv3/127.0.0.1”) == -14002
Sep 25 17:22:52 zappa DirectoryService[82]: Search connection failure: During an attempt to bind to [10.1.1.6] LDAP server.
Sep 25 17:22:52 zappa DirectoryService[82]: Search connection failure: Disabled future attempts to bind to [10.1.1.6] LDAP server for next 0 seconds.
Sep 25 17:24:27 zappa imaps[21391]: IOERROR: stat /var/imap/mailboxes.db: Resource temporarily unavailable
Sep 25 17:24:27 zappa imaps[21391]: DBERROR: error fetching user.bob.Drafts: cyrusdb error[/code]

We’re not using Kerberos at the moment for anything in an effort to try and narrow down where the problem may lie. (blocked at the server firewall)

After it starts erroring out, stopping mail, rebuilding the clients and mail DB and firing it back up has it back on its feet and running in two minutes. However, these crashes are driving me insane.

I tried rebuilding the slapd index last night and that didn’t seem to make any difference. (per the [url=https://www.afp548.com/forum/viewtopic.php?forum=18&showtopic=16535]thread[/url] here)

The GSSAPI errors began after we cut off the Kerberos port, but we saw piles of them today and they didn’t adversely affect service. The crashes that we’d experienced in the past happened without any GSSAPI errors in the logs.

The last slapd.log entries before it dies are as follows

[code]Sep 25 17:22:50 zappa slapd[66]: <= bdb_substring_candidates: (apple-mcxflags) index_param failed (18)\n Sep 25 17:22:50 zappa slapd[66]: SASL [conn=946] Failure: GSSAPI Error: Miscellaneous failure (No principal in keytab matches desired name)\n Sep 25 17:29:41 zappa slapd[66]: <= bdb_substring_candidates: (apple-mcxflags) index_param failed (18)\n Sep 25 17:32:09 zappa slapd[66]: SASL [conn=1006] Failure: GSSAPI Error: Miscellaneous failure (No principal in keytab matches desired name)\n Sep 25 17:32:28 zappa slapd[66]: <= bdb_substring_candidates: (apple-mcxflags) index_param failed (18)\n [/code] Any suggestions? Servers weren't my thing prior to this, so go slow. 🙂

[esmith]

Here are some more error logs if they help. (If there are other logs that would be more useful, please let me know which ones and I’ll be happy to post them here)

The crashes are always timed with when someone logs in to the VPN. I feel like there’s an underlying authentication problem somewhere in here, but I can’t track it down.

The imapd crashlog looks like this:

[code]Date/Time: 2007-09-26 13:07:06.873 -0400
OS Version: 10.4.10 (Build 8R2218b)
Report Version: 4

Command: imapd
Path: /usr/bin/cyrus/bin/imapd
Parent: master [21541]

Version: ??? (???)

PID: 22146
Thread: 0

Exception: EXC_BAD_ACCESS (0x0001)
Codes: KERN_PROTECTION_FAILURE (0x0002) at 0x00000a08

Thread 0 Crashed:
0 imapd 0x000814e9 0x1000 + 525545
1 imapd 0x00051acb 0x1000 + 330443
2 imapd 0x00052358 0x1000 + 332632
3 imapd 0x00004448 0x1000 + 13384
4 imapd 0x0000ae94 0x1000 + 40596
5 imapd 0x000195be 0x1000 + 99774
6 imapd 0x00002656 0x1000 + 5718
7 imapd 0x00001e8e 0x1000 + 3726
8 imapd 0x00001db5 0x1000 + 3509

Thread 0 crashed with X86 Thread State (32-bit):
eax: 0x00000002 ebx: 0x000814d0 ecx: 0x00000002 edx: 0x00000a00
edi: 0xbfffec58 esi: 0x00423ed0 ebp: 0xbfffebc8 esp: 0xbfffeb10
ss: 0x0000001f efl: 0x00010206 eip: 0x000814e9 cs: 0x00000017
ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037

Binary Images Description:
0x1000 – 0x17afff imapd /usr/bin/cyrus/bin/imapd
0x1b2000 – 0x1b4fff apop.so /usr/lib/sasl2/apop.so
0x1b8000 – 0x1c0fff digestmd5WebDAV.so /usr/lib/sasl2/digestmd5WebDAV.so
0x1c4000 – 0x1c5fff libanonymous.2.so /usr/lib/sasl2/libanonymous.2.so
0x1c9000 – 0x1cbfff libcrammd5.2.so /usr/lib/sasl2/libcrammd5.2.so
0x1cf000 – 0x1d7fff libdigestmd5.2.so /usr/lib/sasl2/libdigestmd5.2.so
0x1db000 – 0x1defff libgssapiv2.2.so /usr/lib/sasl2/libgssapiv2.2.so
0x1e2000 – 0x1e4fff login.so /usr/lib/sasl2/login.so
0x1e8000 – 0x1edfff libntlm.so /usr/lib/sasl2/libntlm.so
0x1f1000 – 0x1f3fff libplain.2.so /usr/lib/sasl2/libplain.2.so
0x1f7000 – 0x1fafff mschapv2.so /usr/lib/sasl2/mschapv2.so
0x3a7000 – 0x3affff libotp.2.so /usr/lib/sasl2/libotp.2.so
0x3b9000 – 0x3bbfff pwauxprop.so /usr/lib/sasl2/pwauxprop.so
0x3d6000 – 0x3d8fff shadow_auxprop.so /usr/lib/sasl2/shadow_auxprop.so
0x3dc000 – 0x3defff smb_lm.so /usr/lib/sasl2/smb_lm.so
0x3e2000 – 0x3e4fff smb_nt.so /usr/lib/sasl2/smb_nt.so
0x3e8000 – 0x3ebfff smb_ntlmv2.so /usr/lib/sasl2/smb_ntlmv2.so
0x505000 – 0x541fff dhx.so /usr/lib/sasl2/dhx.so
0x8fe00000 – 0x8fe4afff dyld 46.12 /usr/lib/dyld
0x90000000 – 0x90171fff libSystem.B.dylib /usr/lib/libSystem.B.dylib
0x901c1000 – 0x901c3fff libmathCommon.A.dylib /usr/lib/system/libmathCommon.A.dylib
0x9080b000 – 0x908d3fff com.apple.CoreFoundation 6.4.7 (368.28) /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
0x90911000 – 0x90911fff com.apple.CoreServices 10.4 (???) /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
0x90913000 – 0x90a07fff libicucore.A.dylib /usr/lib/libicucore.A.dylib
0x90a57000 – 0x90ad6fff libobjc.A.dylib /usr/lib/libobjc.A.dylib
0x90aff000 – 0x90b63fff libstdc++.6.dylib /usr/lib/libstdc++.6.dylib
0x90bd2000 – 0x90bd9fff libgcc_s.1.dylib /usr/lib/libgcc_s.1.dylib
0x90bde000 – 0x90c51fff com.apple.framework.IOKit 1.4.8 (???) /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
0x90c66000 – 0x90c78fff libauto.dylib /usr/lib/libauto.dylib
0x90c7e000 – 0x90f24fff com.apple.CoreServices.CarbonCore 682.26 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0x90f67000 – 0x90fcffff com.apple.CoreServices.OSServices 4.1 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
0x91008000 – 0x91047fff com.apple.CFNetwork 129.21 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
0x9105a000 – 0x9106afff com.apple.WebServices 1.1.3 (1.1.0) /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/WebServicesCore.framework/Versions/A/WebServicesCore
0x91075000 – 0x910f4fff com.apple.SearchKit 1.0.5 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
0x9112e000 – 0x9114cfff com.apple.Metadata 10.4.4 (121.36) /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
0x91158000 – 0x91166fff libz.1.dylib /usr/lib/libz.1.dylib
0x91169000 – 0x91308fff com.apple.security 4.5.2 (29774) /System/Library/Frameworks/Security.framework/Versions/A/Security
0x91406000 – 0x9140efff com.apple.DiskArbitration 2.1.1 /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
0x91415000 – 0x9141cfff libbsm.dylib /usr/lib/libbsm.dylib
0x91420000 – 0x91446fff com.apple.SystemConfiguration 1.8.6 /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
0x91948000 – 0x919fafff libcrypto.0.9.7.dylib /usr/lib/libcrypto.0.9.7.dylib
0x91be5000 – 0x91cc3fff libxml2.2.dylib /usr/lib/libxml2.2.dylib
0x9262b000 – 0x92718fff libiconv.2.dylib /usr/lib/libiconv.2.dylib
0x927d8000 – 0x92a08fff com.apple.Foundation 6.4.8 (567.29) /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
0x931ef000 – 0x9320afff com.apple.DirectoryService.Framework 3.3 /System/Library/Frameworks/DirectoryService.framework/Versions/A/DirectoryService
0x949ce000 – 0x949ddfff libsasl2.2.dylib /usr/lib/libsasl2.2.dylib
0x949e1000 – 0x94a06fff libssl.0.9.7.dylib /usr/lib/libssl.0.9.7.dylib
0x94a12000 – 0x94a2ffff libresolv.9.dylib /usr/lib/libresolv.9.dylib
0x94bda000 – 0x94cb2fff edu.mit.Kerberos 5.5.25 /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
0x95bc6000 – 0x95be4fff com.apple.DirectoryService.PasswordServerFramework 2.1.2 /System/Library/PrivateFrameworks/PasswordServer.framework/Versions/A/PasswordServer
0x9c268000 – 0x9c26bfff libwrap.7.dylib /usr/lib/libwrap.7.dylib[/code]

Here’s a sample from system.log when a user logs into the VPN.

[code]ep 26 14:13:44 zappa vpnd[51]: Incoming call… Address given to client = 10.1.1.218\n
Sep 26 14:13:44 zappa pppd[28738]: pppd 2.4.2 (Apple version 233-16) started by root, uid 0
Sep 26 14:13:44 zappa pppd[28738]: L2TP incoming call in progress
Sep 26 14:13:46 zappa pppd[28738]: L2TP connection established.
Sep 26 14:13:46 zappa pppd[28738]: Connect: ppp4 <--> socket[34:18]
Sep 26 14:13:47 zappa pppd[28738]: DSAccessControl plugin: User ‘joe’ authorized for access\n
Sep 26 14:13:47 zappa pppd[28738]: Unsupported protocol 0x8057 received
Sep 26 14:13:47 zappa pppd[28738]: local IP address 10.1.1.6
Sep 26 14:13:47 zappa pppd[28738]: remote IP address 10.1.1.218
Sep 26 14:13:49 zappa DirectoryService[82]: Search connection failure: During an attempt to bind to [127.0.0.1] LDAP server.
Sep 26 14:13:49 zappa DirectoryService[82]: Search connection failure: Disabled future attempts to bind to [127.0.0.1] LDAP server for next 0 seconds.
Sep 26 14:13:49 zappa servermgrd: servermgr_dns: Reloaded named
Sep 26 14:13:50 zappa imaps[25981]: starttls: SSLv3 with cipher RC4-MD5 (128/128 bits new) no authentication
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:53 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:54 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:55 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:55 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:55 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:55 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:55 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:55 zappa DirectoryService[82]: GSSAPI Error: Miscellaneous failure (Server not found in Kerberos database)
Sep 26 14:13:55 zappa imaps[25981]: login: bda196.bis.na.blackberry.com [216.9.xxx.xxx bob plaintext+TLS User logged in[/code]

[esmith]

While it might be premature it believe we have isolated the offending service.

The server has now hit the first 36 hour window of stability in two weeks.

As per [url=https://www.afp548.com/forum/viewtopic.php?showtopic=14427]this thread[/url] the problem was related to the VPN service.

Moving VPN to another box instantly fixed the problem.

Just a heads up for anyone else that might run into this.

[mcrispin]

Hi, I work with the person (esmith) who first made this post.

We run an Intel Xeon Quad 2.0 XServe w/ 4GB of RAM as an OD master.

VPN users are around 20 total, but no more than 6-7 are on at any given time.

Mail users are around 50.

We suspect this was also a problem in the previous server configuration which was entirely PPC, however due to a lack of documentation from those who came before us (!) we can’t confirm.

As a matter of full-disclosure, VPN services were temporarily moved to a PPC box with results as described above. As a new router will soon be installed, VPN services are expected to be handled from that hardware, thus the PPC box will be elminiated from the network.

– Thanks!

[Author]

Posts