CatSearch starting then server crawling….

[Moofo]

Once in a while I get these entries in the AFP server logfile (Access)

IP 10.0.1.106 – – [06/Dec/2006:11:49:25 -0500] “CatSearch starting Studio” 0 0 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:25 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:26 -0500] “CatSearch starting Studio” 0 0 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:26 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:26 -0500] “CatSearch starting Studio” 0 0 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:26 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:26 -0500] “CatSearch starting Studio” 0 0 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:26 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:26 -0500] “CatSearch starting Studio” 0 0 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:26 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:26 -0500] “CatSearch starting Studio” 0 0 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:26 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:26 -0500] “CatSearch starting Studio” 0 0 0
IP 10.0.1.106 – – [06/Dec/2006:11:49:26 -0500] “CatSearch starting Studio” 0 -2147475456 0

And then it kills the rest of the users on the server (Spinning beach ball of death for everyone), disks are thrashing, and there is nothing ellse to do than pull the plug and restart

The server is brand new, Xserve Intel Xeon

[Moofo]

For the find, I don’t know, but that was what I was thinking as well.

I will try to delete the .ds_store files though… Anyy shell command to help ?

I submitted the problem to Apple, they should get back to me.

[cohey]

I am having the exact same issue on a very similar setup. Anyone have any luck finding a solution?

[Moofo]

Updates:

The setup is a small (6 machines) graphic studio. I asked them no to do anymore searches on the server and it seems to have solved the problem. Since it is an unnaceptable solution, I ran some test by the weekend.

I first tried to do searches on smallers network volumes (Such as the one I use to put admin tools) and had no problems.

I then tried to do network search on the Studio and it would really hamper the performance of the (server) machine. The local console would stutter but the CPU was not so bad. I imagine if you do many search at the same time, it would jam at a one point.

The thing is, the network volume was once a whole disk. I mean: The root of this network volume was the root of a disk. I trashed the .ds_store file at the root of the network volume and it seems to have helped a bit.

I will run some more tests by the 27 and will get back to you all. I think there is a bug.

BTW, I have an Apple Case number. They are waiting for my input.

Oh and Cohey, are you using Mirrored Drives ? What is your config. Don’t hesitate to email me again. I will answer you.

[cohey]

Thanks for your reply. We have a little larger scale setup.

[b]Setup:[/b]
We are using a Dual 2.3 G5 Xserve with 4GB RAM running 10.4.8 server (8L127). The volumes are stored on a directly connected Xserve RAID. This server is basically just running AFP. I typically have about 65 users simultaneously connected but the file sizes they are using/copying are very small (typically 300k-500k). Client systems are G4s/G5s running 10.4.x.

[b]Problem:[/b]
If a user does any kind of search on a network volume from the Finder, the side of the RAID where the volume is located indicates max activity, the server response slows for all users, some users get the “spinning beach ball,” the server CPU activity significantly increases. The search seems to never finish or return any results. If the user cancels the search, the symptoms continue for 10+ minutes. The user cannot eject the network volume. The only way to stop the symptoms is to determine which user initiated the search and manually disconnect them via Server Admin. Problem replicates on small ( 20GB ) and large ( 600+GB ) volumes. I have also tried deleting the .DS_Store files with no resolution.

[b]AFP Log sample:[/b]
IP 10.3.10.11 – – [22/Dec/2006:12:52:07 -0600] “CatSearch starting Advertising” 0 0 0
IP 10.3.10.11 – – [22/Dec/2006:12:52:07 -0600] “CatSearch starting Advertising” 0 -2147475424 0
IP 10.3.10.11 – – [22/Dec/2006:12:52:07 -0600] “CatSearch starting Advertising” 0 0 0
IP 10.3.10.11 – – [22/Dec/2006:12:52:07 -0600] “CatSearch starting Advertising” 0 -2147475424 0
IP 10.3.10.11 – – [22/Dec/2006:12:52:11 -0600] “CatSearch starting Advertising” 0 0 -5009

Our users usually don’t use Finder searching but once a few of them do (especially at the same time) it brings everything to a crawl. I would also like them to be able to use Finder searching if they need to. Let me know if you have any ideas or need any more info.

Thanks!

[Moofo]

I emailed back the Apple Enterprise support with my findings (My previous post). I got an auto answer that they won’t be back until the new year.

I sent them the address of this thread.

Our hardware setup is almost the same. I have less RAM, and I’m using two of the internal drives as a raid (750 GB each).

I will try to work on the problem on the 27th. I suggest you file a bug report with Apple enterprise.

BTW, were you surprised not to have RAID 5 in the machine ? I have a fun tale about this.

Oh, I think I partially know a cause. If you run XBench on your server, you’ll be really surprised at the Disk ratings. I don’t know if it’s because of the software mirror, or a bug in the disk controller, but the ratings are really low. like 25 % of a G5 Dual 2 Ghz….

Can you test ?

Needless to say, I’m happy to have found someone with the same problem.

[Moofo]

Well. I’m kinda stuck.

I can’t reproduce the problem anymore !

I think trashing all the folders starting with a dot at the root of my volumes helped fix the problem. I trashed the .ds_store at the root of the volume as well.

[cohey]

My problem seems to have gone away as well. I had noticed a ‘build_hd_index’ process being kicked off by Apple Remote Desktop that was causing some other issues so I disabled indexing on all of my ARD Admin machines to prevent the process from starting on the server every night. I don’t know if it was coincidental or not but the catsearch problem is now not occurring. If I see the problem again or get any more insight on what was causing it I will post it up here.

Thanks for the help!

[Moofo]

I known now that it was definitely related to search on server volumes. Even by doing lots of search on a network volume, the problem does not occur anymore.

[Moofo]

I found something, and I submitted it to the Apple Enterprise support….

I first thought it was retrospects fault but…

I login in the machine at 10 Am, everything is fine…
At 13h30, the Kernel-Task is taking 1.69 Gb (out of 2 gb) of Real Memory and the server is swapping out, like crazy…

I log off, log back in: same problem, the used up memory is “Wired”

Everything is back OK after restart

Turns out, someone in the graphic studio did a file search on the server, then it started crawling…

I did some testing, I cannot reproduce the problem. Why the hell is it doing this !

[Moofo]

Called Apple Enterprise again…

Confirmed that kernel_task process is taking up all the memory after a file search from a client machine sometimes.

I had a pretty good confirmation since:

I was logged into the machine with remote desktop. I saw the memory use of Kernel_Task Jumping ! At one point, my window stopped responding, that’s when my phone rang with a dreaded “The server’s dead” phone call.

I restarted, and after the restart, it turned out that the user’s search windows was still opened. I opened the Activity monitor and….

The memory use jumped to 1.14 Gb in seconds !

The AFP log file was full of those lines (Notice the -5009)

IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:11 -0500] “CatSearch starting Studio” 0 0 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:11 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:11 -0500] “CatSearch starting Studio” 0 0 -5009
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:11 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:11 -0500] “CatSearch starting Studio” 0 0 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:11 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:12 -0500] “CatSearch starting Studio” 0 0 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:12 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:12 -0500] “CatSearch starting Studio” 0 0 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:12 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:13 -0500] “CatSearch starting Studio” 0 0 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:13 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:13 -0500] “CatSearch starting Studio” 0 0 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:13 -0500] “CatSearch starting Studio” 0 -2147475456 0

Well… I sent those lines to Apple. Should get a call back tomorrow. Confidentially, there is supposedly a patch in the works.

Sorry for being so chatty… 😉

[jpbuse]

I’m glad I saw this thread… I’m seeing the exact same issues on my 10.4.8 xServe G5 dual 2.3Ghz server. I’m not sure if my users are doing a Finder style search but perhaps other programs are doing the searching as well. Things seemed to be working just fine until I moved all the core data from one sServe Raid to another. I’ve had these symtpoms before though and like you, a restart of the AFP server (if possible) fixes the slowness for a limited amount of time.

Is anyone running OS X Server Universal yet? Wondering if the AFP server in that build experiences all these fun issues.

[Moofo]

Hem !

If you read my first post, I’m on an Intel Xeon Xserve so I run the universal Mac OS X.

[jpbuse]

Oops. I did read that actually, just didn’t sink in. I was hoping Universal would’ve fixed stuff magically but obviously not 🙁 Any word from Apple Ent yet on a fix/patch? I’ll most likely open a case on my own and reference this thread.

[Moofo]

I only know they are working on it 🙁

[Author]

Posts