Server hangs after a few hours of inactivity
The server is a fresh installed 10.6.2 server with app. 1300 user records, 35 group records, 70 computer records and 25 computer group records. It is not in production yet, so it's only accessed by one administrator.
Only five services active: AFP, DNS, OD master, VPN and Web. DNS working as it should.
When the admin has changed anything in WGM the server tends to hang after a period of one to a few hours. During a portscan all supposed ports shows up. All activities that demand interaction with DirectoryService hang that process.
Seconds before the hang occurs the DS debug log shows the first errors:
2010-01-12 18:36:36 CET - T[0x0000000104781000] - mbr_syscall - Dispatch - Lookup - user/computer ID 2016662 - succeeded emle0301
2010-01-12 18:36:36 CET - T[0x0000000104781000] - mbr_syscall - Dispatch - Membership - is user emle0301 member of group GID 0 = false - TTL 120
2010-01-12 18:36:36 CET - T[0x0000000104781000] - Client: mds, PID: 64, API: mbr_syscall, Server Used : delivered kauth result 0x0000000105BD9930
2010-01-12 18:36:36 CET - T[0x0000000104781000] - Client: mdworker, PID: 24470, API: mbr_syscall, Server Used : process kauth result 0x0000000105BD8CC0
2010-01-12 18:36:36 CET - T[0x0000000104781000] - mbr_syscall - Dispatch - Lookup - user/computer ID 89 - succeeded _spotlight
2010-01-12 18:36:36 CET - T[0x0000000104781000] - mbr_syscall - Dispatch - Membership - is user _spotlight member of group GID 0 = false - TTL 120
2010-01-12 18:36:36 CET - T[0x0000000104781000] - Client: mdworker, PID: 24470, API: mbr_syscall, Server Used : delivered kauth result 0x0000000105BD8CC0
2010-01-12 18:36:36 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsFindDirNodes(), Server Used : DAC : Dir Ref 16787685 : Data buffer size = 1024
2010-01-12 18:36:36 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsFindDirNodes(), Server Used : DAR : 1 : Dir Ref = 16787685 : Requested nodename = /Search
2010-01-12 18:36:36 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsFindDirNodes(), Server Used : DAR : 2 : Dir Ref = 16787685 : Result code = 0
2010-01-12 18:36:36 CET - T[0x0000000102481000] - Client: Requesting dsOpenDirNode with PID = 20842, UID = 92, and EUID = 92
2010-01-12 18:36:36 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsOpenDirNode(), Search Used : DAC : Dir Ref = 16787685 : Node Name = /Search
2010-01-12 18:36:36 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsOpenDirNode(), Search Used : DAR : Dir Ref = 16787685 : Node Ref = 33596808 : Result code = 0
2010-01-12 18:36:36 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsDoPlugInCustomCall(), Search Used : DAC : Node Ref = 33596808 : Request Code = 1000
2010-01-12 18:36:36 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsDoPlugInCustomCall(), Search Used : DAR : Node Ref = 33596808 : Request Code = 1000 : Result code = -14292
2010-01-12 18:36:36 CET - T[0x0000000102481000] - Plug-in call "dsDoPlugInCustomCall()" failed with error = -14292.
2010-01-12 18:36:36 CET - T[0x0000000102481000] - Port: 30135 Call: dsDoPlugInCustomCall() == -14292
After a bunch of similar errors we get another type of error:
Plug-in call "dsDoAttributeValueSearchWithData()" failed with error = -14138.
The problem seems to be key collisions when CCachePlugin tries to AddEntryToCacheWithKeys
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsGetDirNodeInfo(), Search Used : DAC : Node Ref = 33597058 : Requested Attrs = dsAttrTypeStandard:SearchPath : Attr Type Only Flag = 0
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsGetDirNodeInfo(), Search Used : DAR : Node Ref = 33597058 : Result code = 0
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsDoPlugInCustomCall(), Search Used : DAC : Node Ref = 33597058 : Request Code = 666
2010-01-12 18:38:35 CET - T[0x0000000102481000] - CSearchPlugin::DoPluginCustomCall eDSCustomCallSearchSubNodesUnavailable - 0 nodes unreachable
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsDoPlugInCustomCall(), Search Used : DAR : Node Ref = 33597058 : Request Code = 666 : Result code = 0
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsCloseDirNode(), Search Used : DAC : Node Ref = 33597058
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Client: SecurityAgent, PID: 20842, API: dsCloseDirNode(), Search Used : DAR : Node Ref = 33597058 : Result code = 0
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Client: ipfw, PID: 28939, API: libinfo, Server Used : libinfomig DAC : Procedure Request = getprotobynumber
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Client: ipfw, PID: 28939, API: libinfo, Server Used : libinfomig DAR : Procedure = getprotobynumber (13) : Result code = 0
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Client: ipfw, PID: 28939, API: libinfo, Server Used : libinfomig DAC : Procedure = getprotobynumber (13)
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Internal Dispatch, API: dsDoAttributeValueSearchWithData(), Search Used : DAC : 1 : Node Ref = 33554434 : Requested Attr Type = dsAttrTypeNative:number : Attr Match String = 17 : Attr Pattern Match:8449 = eDSiExact : Requested Rec Types = dsRecTypeStandard:Protocols
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Internal Dispatch, API: dsDoAttributeValueSearchWithData(), Search Used : DAC : 2 : Node Ref = 33554434 : Requested Attrs = dsAttrTypeStandard:AppleMetaNodeLocation;dsAttrTypeStandard:RecordName;dsAttrTypeNative:number : Attr Type Only Flag = 0 : Record Count Limit = 1 : Continue Data = 0
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Internal Dispatch, API: dsDoAttributeValueSearchWithData(), Local Used : DAC : 1 : Node Ref = 33555114 : Requested Attr Type = dsAttrTypeNative:number : Attr Match String = 17 : Attr Pattern Match:8449 = eDSiExact : Requested Rec Types = dsRecTypeStandard:Protocols
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Internal Dispatch, API: dsDoAttributeValueSearchWithData(), Local Used : DAC : 2 : Node Ref = 33555114 : Requested Attrs = dsAttrTypeStandard:AppleMetaNodeLocation;dsAttrTypeStandard:RecordName;dsAttrTypeNative:number : Attr Type Only Flag = 0 : Record Count Limit = 1 : Continue Data = 0
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Internal Dispatch, API: dsDoAttributeValueSearchWithData(), Local Used : DAR : Node Ref = 33555114 : Number of Found Records = 0 : Continue Data = 0 : Result code = 0
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Internal Dispatch, API: dsDoAttributeValueSearchWithData(), BSD Used : DAC : 1 : Node Ref = 33555115 : Requested Attr Type = dsAttrTypeNative:number : Attr Match String = 17 : Attr Pattern Match:8449 = eDSiExact : Requested Rec Types = dsRecTypeStandard:Protocols
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Internal Dispatch, API: dsDoAttributeValueSearchWithData(), BSD Used : DAC : 2 : Node Ref = 33555115 : Requested Attrs = dsAttrTypeStandard:AppleMetaNodeLocation;dsAttrTypeStandard:RecordName;dsAttrTypeNative:number : Attr Type Only Flag = 0 : Record Count Limit = 1 : Continue Data = 0
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Internal Dispatch, API: dsDoAttributeValueSearchWithData(), BSD Used : DAR : Node Ref = 33555115 : Number of Found Records = 1 : Continue Data = 7269 : Result code = 0
2010-01-12 18:38:35 CET - T[0x0000000102481000] - Internal Dispatch, API: dsDoAttributeValueSearchWithData(), Search Used : DAR : Node Ref = 33554434 : Number of Found Records = 1 : Continue Data = 0 : Result code = 0
2010-01-12 18:38:35 CET - T[0x0000000102481000] - CCachePlugin::AddEntryToCacheWithKeys - Entry NOT added for record 0x0000000105AABA10 with key p_name:udp - collision
2010-01-12 18:38:35 CET - T[0x0000000102481000] - CCachePlugin::AddEntryToCacheWithKeys - Entry 0x0000000105E05C00 added for record 0x0000000105AABA10 with key p_proto:17 - TTL 3600
2
I'm not familiar enough with the internals of DirectoryService to understand what's gone wrong - and how to fix it.
We already tried reinstalling and importing the records mentioned above.
Any hint how to advance would be appreciated.
Tycho