[Moofo]

Just so everyone knows.

The issue is really solved on the Intel OS X server as far as I know. The studio where the Xserve is being used is doing lots of search and everything seems to be fine now. In fact, I had to turn off the machine one time: to clean the vents !

[Moofo]

Remembre the Quadra 950 ? It’s even worst than that.

When the fans runs full speed, it’s really noisy. It’s like a 747…

You definitely don’t want it in the same room as you.

[Moofo]

I wonder if installing Mac OS X Server universal would solve that problem on PPC machines.

BTW: OS X server 10.4.9 is out.

[Moofo]

I tried the fix.

It works ! I was able to do lots of search without Kernel Task gobbling it all up ! Yay !

[Moofo]

I have users in my studio using the latest version of ID and it doesn’t make a difference.

Try the following:

Log in SSH to your server from a machine and run top “top -ovsize” It will sort the output by VirtualMemory Size

Then from the same machine or another, mount a volume on your server and do a file find.

While the search is running, look at the top output, especially the RSIZE column.

There you have your proof. I don’t know if InDesign is using the search function to look for artwork and related file, but I think it has to do with the problem.

Apple is really slow and not very talky for what I would call a major issue.

Look there

[url]http://discussions.apple.com/thread.jspa?messageID=4014503&#4014503[/url]
[url]http://discussions.apple.com/message.jspa?messageID=3995835#3995835[/url]

[Moofo]

Yes Retrospect (Not the client, the full app) is running on the server.

[url=http://forums.dantz.com/ubbthreads/showflat.php?Cat=0&Board=Desktopworkgrupx&Number=93657&Searchpage=1&Main=93657&Words=+Moofo&topic=&Search=true#Post93657]I first thought it was retrospect[/url] (My thread at EMC)

However, further analysis of logfiles and continuous Top Output proved to demonstrate a problem with the AFP server.

Just for your info, I reproduced the problem without Retrospect running on the server. I doubt it has anything to do with it.

And yes I’m P***** Apple Enterprise is working on it since december. I got a very rude phone call from them today. I guess I was being too inquisitive.

I gave them:

Screenshot of activity monitor before it happens
Screenshot of activity monitor while it happens
Countinuous top output (fresh from a reboot) from a “Clean state” to a “Problem state”
Countinuous AFP output (fresh from a reboot) from a “Clean state” to a “Problem state” with one user doing search.

And still “The engineer are working on it” I directed them to this thread and I’m sure they are ignoring it.

Is your real memory usage of “Kernel Task” jumping to inordinate amounts ?

[Moofo]

I don’t think it has anything to do with the problem. ARD index is a totally different beast.

I disabled the indexing by simply adding the drive were the data is to the “Privacy” tab in the Spotlight System preferences pane.

I reproduced the problem many times now, and one of those time, I took continuous Top output as well as AFP log, screenshots of Activity Monitor before, and while the problem is occuring.

Sent that to Apple Engineering. There is definitely a problem with AFP server.

They should come back to me soon.

[Moofo]

I only know they are working on it 🙁

[Moofo]

Hem !

If you read my first post, I’m on an Intel Xeon Xserve so I run the universal Mac OS X.

[Moofo]

Called Apple Enterprise again…

Confirmed that kernel_task process is taking up all the memory after a file search from a client machine sometimes.

I had a pretty good confirmation since:

I was logged into the machine with remote desktop. I saw the memory use of Kernel_Task Jumping ! At one point, my window stopped responding, that’s when my phone rang with a dreaded “The server’s dead” phone call.

I restarted, and after the restart, it turned out that the user’s search windows was still opened. I opened the Activity monitor and….

The memory use jumped to 1.14 Gb in seconds !

The AFP log file was full of those lines (Notice the -5009)

IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:11 -0500] “CatSearch starting Studio” 0 0 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:11 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:11 -0500] “CatSearch starting Studio” 0 0 -5009
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:11 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:11 -0500] “CatSearch starting Studio” 0 0 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:11 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:12 -0500] “CatSearch starting Studio” 0 0 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:12 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:12 -0500] “CatSearch starting Studio” 0 0 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:12 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:13 -0500] “CatSearch starting Studio” 0 0 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:13 -0500] “CatSearch starting Studio” 0 -2147475456 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:13 -0500] “CatSearch starting Studio” 0 0 0
IP fe80::20a:95ff:fecf:6698 – – [24/Jan/2007:16:49:13 -0500] “CatSearch starting Studio” 0 -2147475456 0

Well… I sent those lines to Apple. Should get a call back tomorrow. Confidentially, there is supposedly a patch in the works.

Sorry for being so chatty… 😉

[Moofo]

I found something, and I submitted it to the Apple Enterprise support….

I first thought it was retrospects fault but…

I login in the machine at 10 Am, everything is fine…
At 13h30, the Kernel-Task is taking 1.69 Gb (out of 2 gb) of Real Memory and the server is swapping out, like crazy…

I log off, log back in: same problem, the used up memory is “Wired”

Everything is back OK after restart

Turns out, someone in the graphic studio did a file search on the server, then it started crawling…

I did some testing, I cannot reproduce the problem. Why the hell is it doing this !

[Moofo]

I known now that it was definitely related to search on server volumes. Even by doing lots of search on a network volume, the problem does not occur anymore.

[Moofo]

Well. I’m kinda stuck.

I can’t reproduce the problem anymore !

I think trashing all the folders starting with a dot at the root of my volumes helped fix the problem. I trashed the .ds_store at the root of the volume as well.

[Moofo]

I emailed back the Apple Enterprise support with my findings (My previous post). I got an auto answer that they won’t be back until the new year.

I sent them the address of this thread.

Our hardware setup is almost the same. I have less RAM, and I’m using two of the internal drives as a raid (750 GB each).

I will try to work on the problem on the 27th. I suggest you file a bug report with Apple enterprise.

BTW, were you surprised not to have RAID 5 in the machine ? I have a fun tale about this.

Oh, I think I partially know a cause. If you run XBench on your server, you’ll be really surprised at the Disk ratings. I don’t know if it’s because of the software mirror, or a bug in the disk controller, but the ratings are really low. like 25 % of a G5 Dual 2 Ghz….

Can you test ?

Needless to say, I’m happy to have found someone with the same problem.

[Moofo]

Updates:

The setup is a small (6 machines) graphic studio. I asked them no to do anymore searches on the server and it seems to have solved the problem. Since it is an unnaceptable solution, I ran some test by the weekend.

I first tried to do searches on smallers network volumes (Such as the one I use to put admin tools) and had no problems.

I then tried to do network search on the Studio and it would really hamper the performance of the (server) machine. The local console would stutter but the CPU was not so bad. I imagine if you do many search at the same time, it would jam at a one point.

The thing is, the network volume was once a whole disk. I mean: The root of this network volume was the root of a disk. I trashed the .ds_store file at the root of the network volume and it seems to have helped a bit.

I will run some more tests by the 27 and will get back to you all. I think there is a bug.

BTW, I have an Apple Case number. They are waiting for my input.

Oh and Cohey, are you using Mirrored Drives ? What is your config. Don’t hesitate to email me again. I will answer you.

[Author]

Posts