AppleFileServer hogging CPU

[hetjan]

Hi,

I have a server that is giving me grief. The server is one of three XServes, a G5 Cluster unit, running mostly AFP.

There are two processes in the process list called AppleFileServer. One is launched by launchd with an ID of 311. The other has an ID of 312 and its parent process is AppleFileServer (311). The 312 process, run by root, uses 0.0% of CPU, running 1 thread and is behaving OK. The 312 process, run by root, is using anywhere from 85% to 110% CPU time with 42 threads and a LOT more memory than the 311 process.

I tried upgrading to 10.4.2 Server (8C39) with no luck. I tried starting DNS and making all clients plus this server bind to the IP of the directory server (another server running 10.3.9); without any positive effect.

Do you have any pointers for me?

A.

[hetjan]

lookupd is also taking 40+% CPU time

[dmc]

On our xServe’s I have found 4 things what will drive up the CPU utilization of the AppleFileServer process.

1. “corrupt” .DS_Store files
2. filled AFP share (above 90% full)
3. samba
4. large number of files in a folder (> 1000)

Number 1 – I have *no* idea why this happens, or if it’s common, but it’s happend to us twice in the last year. Hunting down the problematic .DS_Store file is next to impossible. You can find the workstation sometimes by looking at the packets (tcpdump) – there will be one ore flooding the server with traffic for no obvious reason. But that doesn’t help unless you know which .DS_Store files(s) are freaking it out. My fix is to log all workstations off and then run a find command and delete all .DS_Store files on all AFP mounts. I run this command from each share:

(run this at your own risk!)
sudo find . -name .DS_Store -exec rm {} \;

Number 2 – do a df -h and if any volume is over 90% full you have problems. I think our 1.4 TB volumes freak out around 92-95% full. I now have a script that actively watches server space.

Number 3 – If a PC starts drilling our server really hard on a GB connection it will drive up the AFP process and slow everybody down. Our fix? Moved all the PC’s to 100 BaseT.

Number 4 – if you have a folder with more than 1000 items (roughly) and people open it up often, your AFP CPU usage will rise as you describe.

Hope something in there helps.

[Anonymous]

I’ve been experiencing the same issue. The problem only surfaced after the networking was connected to a new router.

At first, I thought it was the server box (running 10.2.8 @ 100 full duplex). I’ve since upgraded to a new box (10.4.2 @ 1000base-T) but the problem is still there.

I’ve watched top to see when it starts freaking out. CPU usage is normal if an OS 9 machine connects. However, as soon as an OS X machine connects (irregardless of how many), it goes psycho.

I do have several folders with more than 1000 items but, like I mentioned above, the problem only surfaced after moving to new routing. SMB is turned on. The volume has 74% free space. I’ve not searched for/deleted any .DS_Store files yet.

Any other ideas?

Thanks,
Brian

[Anonymous]

I’ve been experiencing the same problem.

I have noticed that AppleFileServer runs at 90% of CPU only when Illustrator 10 is being run on the client machines. I think there is some bug in Illustrator that causes this.

[Anonymous]

OK, I turned off “calculate folder size” on the client machines and that worked…for a week.

It started happening again today at 3:00 in the afternoon. No server side changes have been made. Nothing in any logs that helped.

My network guys are running some sniffers to see if there’s any excess network traffic to that port.

I spoke with an Apple System Engineer about this and he didn’t have a surefire solution. He said that it sounded like there might be a lot of retransmitted packets between the server and the network gear.

I hope someone can enlighten me with a solution for this soon. It’s giving our Windows admins something to tease the Mac guy about.

[macmanjc]

I am having the same problem but I have found a way, at least in my setup, to avoid it. Unfortunately, I think that the solution raises more questions than it answers.

First, the setup: 1 older G4/766MHz acting as the OD Master w/ DHCP and DNS running. 1 first generation XServe G4/1GHZ acting as an AFP/SMB file server and running FileMaker Server 7v3. 1 XServe G5/2x2GHz acting as a Mail server and running Retrospect 6 for backups.

I installed OS X 10.4.2 clean on each of these machines on Saturday and configured them as described above over the weekend. As I was logging into the clients via Remote Desktop and logging into the server to test connectivity I noticed that the activity lights on the XServe G4 were maxed out. I immediately ran Activity Monitor and noticed the same thing that everyone else is reporting: AppleFileServer was running as two separate processes, one of which was running at between 85%-100% of CPU capacity.

I immediately kicked the clients and it settled down. Since I needed to get the clients up and running for Monday morning, I decided to configure all of the clients to bind to the Open Directory server and get their mobile accounts working, and then troubleshoot the AFP issue.

Here is where it gets interesting!

I found that any client that logs into the server as part of the Login process (via an alias to the share point in Accounts -> Login Items) did not cause AFP to go crazy. In fact, after getting all 21 users configured and attached to the server, it was running at about 5% CPU capacity.

Just to see what would happen, I ejected the share point on one of the machines and then logged into it through the CMD-K Connect To interface, and even though it still authenticated using Kerberos, it still instantly caused the usage to skyrocket! I then ejected the share point and tried going in through the Network icon, and the same thing happened: instant CPU peak.

So…. I’m curious if any of the people who have reported this problem are using the same setup that I am, and getting different results, or are your users logging in through one of the Connect methods?

Also, one caveat to this:

I needed to mount the two sharepoints from the XServe G4 on the XServe G5 so that Retrospect can back them up. I decided to try the path of least resistance first, and used the CMD-K Connect To method to select the file server, and then authenticated through Kerberos… and they mounted with no problem! No strange CPU spikes when connected manually to the XServe G5. So then I tried the same thing from the G4/766 server, and the usage went up to about 65%; not as bad as when a client connected, but still more than when the other XServe connected.

So, is it something to do with the XServe hardware? Something to do with the difference between OS X Server and OS X client? (BTW, the clients that are connecting are almost all 10.3.9, but I did connect with two clients that are running 10.4.2 and had the same issue).

I hope all of this long-winded information is helpful to someone out there!

-John

P.S. I am also posting this to the following thread, since these topics are virtually identical:
https://www.afp548.com/forum/viewtopic.php?forum=25&showtopic=7322&mode=&onlytopic=&show=10&page=2

[gary]

How do you stop the Server from creating .DS_Store files on Network Mounts?

[Anonymous]

My setup is significantly different. No OD, DHCP & DNS are provided via Windows domain controllers.

The problem has appeared on: a G4/1.25 MDD file server running Server 10.2.8, a G4/1.25 MDD file server running Server 10.3.9 and a Xserve G5/2.0 file server running 10.4.2. It does happen most frequently on the Xserve. OS 9 clients can connect with no issues but the first OS X client will freak the server out.

TCPdump doesn’t show anything obviously wrong (at least to me – not that I’m the world’s greatest at deciphering this info) except for packets with bad checksums. It looks like the packets are exceeding standard MTU of 1500 (does the packet header add bytes?) and the switch/router may be stripping parts out. Some of the packets look garbled when viewed through any type of packet sniffer.

The only thing I can think of at this point is it’s either the 10.3.9 clients or something misconfigured on the switches and/or router.

I’d really like to find a solution soon, I’m running out of ideas.

[macmanjc]

[QUOTE BY= Brian B] TCPdump doesn’t show anything obviously wrong (at least to me – not that I’m the world’s greatest at deciphering this info) except for packets with bad checksums. It looks like the packets are exceeding standard MTU of 1500 (does the packet header add bytes?) and the switch/router may be stripping parts out. Some of the packets look garbled when viewed through any type of packet sniffer.

The only thing I can think of at this point is it’s either the 10.3.9 clients or something misconfigured on the switches and/or router.
[/QUOTE]

Brian,
Have you taken a look at this thread?
https://www.afp548.com/forum/viewtopic.php?forum=25&showtopic=7322&mode=&onlytopic=&show=10&page=1

Reading through it there is a lot of discussion about probable network causes for this issue (see post by chiefgeek on down). He suggests using Helios LANTest to test the client/server connection. I personally have not been onsite at my client’s since I saw his post, but I am definitely going to try this when I am in next time, since it might indeed show some interesting problem areas in the network.

Also, are you using any of the same networking equipment that the other posters are referring to?

-John

[Anonymous]

John,

I’ve read that thread too. The only thing suggested that I haven’t tried is LANTest.

All of our networking gear is Cisco, I’d have to doublecheck models.

A restart resolved the issue this week. That’s never “fixed” the problem before. Maybe the problem was a fluke this time.

[gary]

We’ve been having this problem as well,
I notice that AppleFileServer spikes when one of our clients (we use the xserve as an OD Master and people login with network home drives) will open an MS Office program. When you get that splash screen that says “configuring office components,” applefileserver spikes to around 60% with a SINGLE USER logged in. With an entire class, it spikes to over 100% and the kids think the computer freezes. This is 10.4.2 Tiger Server with 10.4.2 clients. Anybody report the same?

[Tarny]

I have 2 customers that are experiencing high utilization of the CPU by the second AppleFileServer process. Both customers have Mac OS X Server 10.4.4. One customer is updating to 10.4.5 tomorrow.

I see that there hasn’t been a lot of traffic on this thread since October of last year. Has the problem “gone away” for everyone? Or is there a solution?

Using top I can identify that the AFS process is taking about 10-20% of the CPU when there are very few connections and it is just sort of idling. Once the actual work starts, it will average over 95% utilization.

Here’s some more information about the servers and environments. Both customers have xServes. Both have Open Directory Masters. One has the file services on the ODM, the other has an ODM plus replicas and two file servers connected to the directory services. One customer only uses built in hard drives, the other has an xServe RAID connected to the two file servers. In both instances the networking infrastructure seems solid, gigabit ethernet, switches. Note, the CPU utilization and apparent slow down of service to the users occurs even if the users are connecting via wireless connections. Both customers have extensive wireless, but the wired connections have slow service too and the CPU still seems to be working excessively hard.

Lot’s of good suggestions in this thread and we are working our way through them, but I note that nothing appears to be a “silver bullet.”

Tarny

[Anonymous]

I’ve had this exact issue with our school server for a long time. When certain applications were launched, the AFP process would go nuts and everything would grind to a halt. It got to the point where it would happen when just a couple of machines started Classic at the same exact time. I tried everything and nothing worked. Until…

I archived the OD database, reinstalled the server software on to a new startup volume, updated, then restored the OD database. I had to manually merge the database back in because my old setup didn’t have Kerberos turned on. In the end, the server is once again performing great.

I have a feeling this is a problem for servers that have been upgraded over the years and not freshly installed.

[jchang]

I’m having the same problem with two home directory servers. I decided to call Apple Enterprise Support.

They told me that they know about AFP problem and gave me three work around.

1. Limit 100 users per server.
2. Disable the Guess Access.
3. Limit the network mount / automount to 1 per server.

[Author]

Posts