Intermittent OS X Server / AD error

[DSoderholm]

I have an XServe set up for file sharing with SMB, on a Windows network. I’ve bound the AD plugin to the domain, and set Samba up as a domain member. Users log on with Active Directory credentials, and setting permissions etc works perfectly.

The problem is, every now and again it will simply stop authenticating users, and needs a restart. A user will attempt to open a share, and be told ‘username does not exist’ or ‘access denied’, even if they were opening the same share five minutes earlier. This suddenly starts happening after about 12 days’ uptime, and only a reboot will solve the problem. Restarting Samba services sometimes fixes it for a few minutes, but then the problem reoccurs. It seems as if Samba stops checking against the AD, but there’s nothing in any log files to indicate what is going on. Does anyone else have experience of this?

(I’ve tried setting a Cron job to restart the server once a week, but it doesn’t seem to work and even if it did, it wouldn’t be ideal). Any input greatly appreciated.

[RobertHammen]

Yes, we have this same exact issue. It seems to be something Apple is aware of – there was a “Best Practices/check these three things if you have this problem” in a session at WWDC done by Nicole Jacque of Apple. (BTW, I checked her guidelines and everything’s fine on our xServe/OS X Server, and still have the same intermittent issue).

You don’t have to reboot the entire server – as you noted, just stopping and restarting the Windows service doesn’t fix the issue – the temporary workaround is to kill/restart all instances of the “DirectoryService” process – this will boot all of your Windows connections off, but not disrupt AFP, NetBoot, Software Update Server, et. al.. Here’s to hoping the rewrite of DirectoryService in Leopard is the eventual fix for this.

In researching this problem it seems like it was a samba issue. At what version of OS X Server was your computer joined to AD? We started with (gulp) 10.4. It’s been postulated that unbinding from AD, deleting the computer out of the AD domain, and then re-joining/re-binding might “fix” this issue. However, with a number of sharepoints with permissions set using ACL’s from the Windows side, this didn’t appeal to us. I’m just about done setting up a new Intel-based xServe/RAID (to replace the G5 one we leased almost 3 years ago), so I’m in hopes that this *might* address the issue.

By any chance do you have access to the AD side of the house, to debug what’s going on when authentication is failing? I do…

[DSoderholm]

We got the server around 10.4.3 time, tried to join it to AD, failed, left it for ages, finally succeeded at 10.4.9, and it’s now 10.4.10. I’m the administrator on the Windows machines too, so I have full access to the AD. Nothing ever shows up in standard Windows server event logs, but I haven’t got round to enabling super-verbose debugging output to dig deeper.

I’ve been reluctant to delete it from the AD and re-joining it (considering it took me six months to get it working in the first place and I’m still not sure how I did it). At least knowing which process is buggy helps, but avoiding it entirely would of course be best. Some people mumbled that it might be lookupd, so I set a cron job to restart that and smbd every few days, but that didn’t solve it. If I see it happen again I’ll try restarting DirectoryService, but at the moment I’m just rebooting it once a week before it starts going belly-up, just to keep users from complaining. I’m going away for a fortnight soon though >.< I'll look through memberd too, and see if I can figure out how that might be related - I'm more of a Windows guy so I'm not overly familiar with all the OS X Server processes. Thanks for the suggestions! Incidentally, is there a written-down version of those guidelines Robert mentioned?

[nicole]

There’s two possibilities here: either your users are failing authentication, which is a DS/winbindd problem, or they’re failing authoriziation, which is likely a memberd problem. As a test, when the problem happens, try authenticating with an AD user account using dirt with ntlm authentication:

dirt -a nt -u -p

If this fails, you have an authentication problem, and the steps I talked about at WWDC should be able to help you.

If it succeeds, your authentication is ok, but you’re being blocked by permissions. Assuming your permissions are set such that the user in question should have access, the culprit is probably memberd. Resetting the memberd cache on the fileserver ought to fix this: memberd -r. You can also fix this issue for a specific user by running id for that user on the fileserver, as this will refresh the memberd cache for that user. Once you “id” the user, they should be able to login again.

Hope that helps.

[RobertHammen]

[QUOTE][u]Quote by: DSoderholm[/u][p]Incidentally, is there a written-down version of those guidelines Robert mentioned?[/p][/QUOTE]

Here are my notes. Sorry if anything is incorrect, I was trying to keep up:

Active Directory Plug-in troubleshooting
1) Binding fails at “Step 5” = enable DS debug logging sudo killall -USR1 DirectoryService

touch /Library/Preferences/DirectoryService/.DSLogAtStart

/Library/Logs/DirectoryService/DirectoryService.debug.log

kpasswd operates over UDP on Tiger (can do either in Leopard)
Windows clients can use TCP instead

Port 464 UDP blocked? Check firewall.
Too much Kerberos PAC data? Use different admin account
Non-routable DNS address for dc? disable checkbox
Network settings that interfere with UDP? fix network settings

2) Issues with authenticating SMB to DirectoryService

AD user authentication to SMB services fails
Non-AD users are able to authenticate via SMB
Authentication to other services (AFP, etc.) works

First – check which authentication is working
dirt -u username -p password
dirt -u username -p password -a nt (NTLM)

dsconfigad -sso (all services to auth against AD
check /etc/smb.conf
workgroup = NETBIOS name of AD
security = ADS (active directory server) – DON’T USE DOMAIN
netbios name = computer account that you use when you bound to AD in Directory Access
use spnego = yes – enables negotiation of authentication protocols
realm = Kerberos name of your AD domain – ALL CAPS (Kerberos = case-sensitive)

Verify that winbindd is running (2 instances) – if not, start it /usr/sbin/windbinddd -s /Library/Preferences/DirectoryService/winbindd.conf

Make sure both AD AND samba are using the same password

/Library/Preferences/DirectoryService/ActiveDirectory.plist

AD Computer Password q
base64 password – need to be root to access

Dump the secrets.tdb file (/var/db/samba

tdbdump
data = “”

If they don’t match – reset samba

net -f changesecretpw

(will prompt you for a password – use the base64-decoded AD plugin)

3)Mac OS X Server bound to AD
Home Directory path in AD pointing to share on OS X Server
Home Directory shares located on non-boot volume (i.e. xServe RAID)

Symptoms:
clients can’t automount home dirs
home dirs don’t automagically get created
home dirs get created in the wrong place
Virtual home shares don’t work
AFP logins fail

\\Server\Share\Username -> /Network/Servers/server.fqdn.com/share/username

if home dirs not on the boot drive

/Network/Servers/server.fqdn.com/Volumes/volumename/share/username

create a symbolic link:
ln -s /Volumes/Storage/Homes /Homes

[RobertHammen]

[QUOTE][u]Quote by: nicole[/u][p]There’s two possibilities here: either your users are failing authentication, which is a DS/winbindd problem, or they’re failing authoriziation, which is likely a memberd problem. As a test, when the problem happens, try authenticating with an AD user account using dirt with ntlm authentication:

dirt -a nt -u -p

If this fails, you have an authentication problem, and the steps I talked about at WWDC should be able to help you.

If it succeeds, your authentication is ok, but you’re being blocked by permissions. Assuming your permissions are set such that the user in question should have access, the culprit is probably memberd. Resetting the memberd cache on the fileserver ought to fix this: memberd -r. You can also fix this issue for a specific user by running id for that user on the fileserver, as this will refresh the memberd cache for that user. Once you “id” the user, they should be able to login again.

Hope that helps.

[/p][/QUOTE]

The exact same problem still occurs on our new bound-at-10.4.9 Intel Xserve. It’s getting pretty frustrating.

When the problem happens authentication fails. It seems there are two problems:

#1) The machine password authentication against WINS starts failing (resetting nmbd (SIGHUP) seems to fix the problem temporarily). We have WINS registration set to 4 weeks in the smb.conf, so this shouldn’t be happening.

#2) Once this happens, user authentication starts to fail. Only SIGHUP’ing DirectoryService seems to fix it temporarily.

Sometimes this problem occurs and it fixes itself. Otherwise it’s multiple times per day. Doesn’t seem to be consistent that it happens after x amount of uptime (the only consistent thing is to run both commands and the problem is gone, at least temporarily). It’s getting pretty frustrating.

Would love to get to the bottom of this issue, as it’s been occurring under 10.4.x for a couple of years across two different Xserves (G5 and Intel)…

[DSoderholm]

It’s still happening here, and all I can do at the moment is kill -9 DirectoryService whenever it pops up (or preemptively, once a week or so). I haven’t yet had a chance to try out some of the other ideas mentioned in this thread, but I hope to next week. I did idly wonder if it had to do with Kerberos ticket length or something like that, and checked through the Group Policies on the Windows domain to see if there was anything there, but nothing jumped out at me. It’s largely under control at the moment (and killing DirectoryService instead of rebooting means that for the first time in ages, I have a server uptime of more than two weeks). Still, it’s a problem I wish wasn’t there…

[jeffbevans]

I have been having this same issue at a clients site as well. I did find this post:

https://www.afp548.com/forum/viewtopic.php?showtopic=16099

and I wonder if running the script that is in that post is the solution. Read and let me know what you think. I don’t mean to hijack this post, but I have some relevant questions as well. I am a windows guy (keep the comments to yourself, peanut gallery :)) and don’t have too much experience (read none) on Mac servers. I have one 10.4.10 server joined to AD successfully but get these intermittent access denied messages as reported in this post. Now, in the post I point out above, it is stated that this issue was fixed as of 10.4.9, I believe. As stated, I am at 10.4.10 and still experiencing this issue. The xserve I am working on does exhibit the issue as highlighted in the post I referenced above. I am planning on running the recommended procedure soon and will report back my results here.

Still, even though I have a successful bind to AD, I am not sure that everything is setup as it should be. When I look in Server Admin I don’t have a little green light beside Open Directory. In addition, when I click on Open Directory and go to the Overview tab, Kerberos is listed as stopped. I don’t know how to turn this on, I don’t know if I even need it on. When I have Open Directory selected, the option to start the service is still greyed out.

I say I have a successful bind to AD because I am able to add AD users and groups to the ACL’s on folders located on the Mac server. The fact that the lookup to AD works for that leads me to believe that my binding is successful. Is this correct? How does anyone else’s Open Directory service look?

thanks
Jeff

[Author]

Posts