Home › Forums › OS X Server and Client Discussion › Questions and Answers › Some Basic launchd questions/Apache log rotation
- This topic has 20 replies, 3 voices, and was last updated 17 years, 2 months ago by
khiltd.
-
AuthorPosts
-
January 21, 2008 at 2:48 pm #371177
Bill Eccles
ParticipantGentleones,
I’ve found the [url=http://developer.apple.com/macosx/launchd.html]article at Apple[/url] which describes how launchd works, and it does a pretty good job. Since, however, I’ve never created a launchd job, I’ve got some basic questions which were not answered by this article.
First, if I’m just in need of running a basic shell script, where’s the best place to put that shell script? Is there a preferred directory for that type of thing, especially considering it’s a system-wide, needs-to-run-even-when-no-users-are-logged-in script?
Second, my plist should go into /Library/LaunchDaemons, right?
By way of background, I find the log rotation scheme offered by Apache/Apache2 to be annoying, so I’m working on making a script that rotates the httpd logs nicely in the same way that all other logs are rotated with the added catch that I have to update AWStats before rotating logs on through. If anybody has done this kind of thing before, I’d love to hear of it.
Thanks,
BillJanuary 23, 2008 at 3:33 pm #371219Bill Eccles
ParticipantWell, that was interesting.
I created a script, below, and a .plist, also below, and placed them in /usr/local/bin and /Library/LaunchAgents, respectively. When I issue
[code]launchctl load /Library/LaunchAgents/net.eccles.rolllogs[/code]
it causes launchd (and I don’t know if it’s a new process or launchd with a PID of 1) to consume 100% of a CPU forever. No new programs can run because nothing can get launched. So I have to 5-second-hold-the-power-button reboot, and the machine boots fine. But when I try to login, again, launchd goes out to launch… er, “lunch.” (Great. Now I’m hungry.)I had a few episodes earlier in the day where I ran the script manually and it ran fine except that it griped because it didn’t have sufficient privileges to modify the /var/log/apache2 directory contents. So I ran it as sudo and it went off into la la land, never to return. Eventually, I rebooted and tried again, but this time it worked. A “top” showed that gzip was consuming CPU, as might be expected.
Is there something in my plist which would send launchctl out to lunch? Is there something in my script which looks crazy?
Thanks in advance,
Billrolllogs.sh
[code]
#!/bin/bash# Update statistics for AWStats…
cd /Library/WebServer/awstats/wwwroot/cgi-bin
for i in `ls *.conf | sed -e ‘s/awstats.//’ -e ‘s/.conf//’` ; do
echo -n “${i}”;
/Library/WebServer/awstats/wwwroot/cgi-bin/awstats.pl -update -config=”$i”
donecd /var/log/apache2/
for i in `ls *_log` ; do
echo “$i”;
if [ -f “${i}” ]; then
echo -n ” ${i}”
if [ -x /usr/bin/gzip ]; then gzext=”.gz”; else gzext=””; fi
if [ -f “${i}.6${gzext}” ]; then mv -f “${i}.6${gzext}” “${i}.7${gzext}”; fi
if [ -f “${i}.5${gzext}” ]; then mv -f “${i}.5${gzext}” “${i}.6${gzext}”; fi
if [ -f “${i}.4${gzext}” ]; then mv -f “${i}.4${gzext}” “${i}.5${gzext}”; fi
if [ -f “${i}.3${gzext}” ]; then mv -f “${i}.3${gzext}” “${i}.4${gzext}”; fi
if [ -f “${i}.2${gzext}” ]; then mv -f “${i}.2${gzext}” “${i}.3${gzext}”; fi
if [ -f “${i}.1${gzext}” ]; then mv -f “${i}.1${gzext}” “${i}.2${gzext}”; fi
if [ -f “${i}.0” ]; then mv -f “${i}.0” “${i}.1” && if [ -x /usr/bin/gzip ]; then gzip -9 “${i}.1”; fi; fi
if [ -f “${i}” ]; then mv -f “${i}” “${i}.0”; fi
fiapachectl graceful
done
[/code]
[/code]
and the plist, net.eccles.rolllogs.plist
[code]
Label
net.eccles.rolllogs
ProgramArguments
/usr/local/bin/rolllogs.sh
StartCalendarInterval
Day
0
Hour
3
Minute
15
January 23, 2008 at 4:21 pm #371221Bill Eccles
ParticipantIt’s not PID 1 that goes to 100%. It’s #108, so I don’t know what’s going on.
I tried “launchd bash” per Apple’s recommendation, but every time I try that, launchd only tells me that it’s not meant to be run directly.
/Bill
January 24, 2008 at 4:17 pm #371242Bill Eccles
ParticipantIt’s another instance of launchd.
Right now on this machine I have two instances of launchd running:
1 ?? 0:05.56 /sbin/launchd
114 ?? 0:02.15 /sbin/launchdI’m not where I can send this machine into hysteria right now (it’s my primary work machine), but if I get a chance, I’ll try launchctl and see if it’s 114 that sucks up a CPU or if launchctl causes a new launchd to launch and suck up a CPU.
In either case, it seems to me that I’ve got something very, very wrong in either my script or my plist. Yesterday, I tried to set launchd logging level to debug, but still nothing showed up in system.log. (Evidence from Apple seems to suggest that there should be another log for launchd somewhere, but I can’t find it.) And, though I tried to “launchd bash” per Apple’s suggestion that doesn’t work–it just gripes that it can’t be run directly.
Yikes…
January 24, 2008 at 4:58 pm #371248khiltd
ParticipantWell that script is basically two completely unyielding loops, both of which fire off other CPU hogging processes repeatedly. I’d suggest making friends with nice and renice to lower your scheduling priority.
January 24, 2008 at 5:11 pm #371250Bill Eccles
ParticipantOK, if I understand the plist correctly, that’s something I can do in the launchd plist for this.
Since it’s a shell script, I presume there’s no automatic loop unrolling or anything like that, though, right? So only one instance of awstats or one instance of gzip would be running, correct?
In either case, when I run this script manually, CPU is never above 30% for the entire time that the thing is running, and I can see each of the perl and gzip processes being executed, so I guess that begs the question, why is a launchd process hogging the CPU? If I use launchd to fire off the script, does it somehow hide all of what it’s doing, such as the perl and gzipping? Or should I be able to see them running as they’re fired off?
I don’t even think my script gets to being run, quite frankly.
January 24, 2008 at 5:40 pm #371252Bill Eccles
ParticipantSuspicion confirmed. The script is never getting called, and yet launchd takes up 100% CPU and even bounces back and forth between CPUs.
I replaced my complex-ish rolllogs.sh script with this:
[code]
#!/bin/bash
logger TestMessage
[/code]And it works fine if I merely sudo /usr/local/bin/rolllogs.sh.
But if I launchd load net.eccles.rolllogs.plist, launchd goes out to lunch.
Does anybody have any idea what’s happening?
Thanks,
BillJanuary 24, 2008 at 5:44 pm #371253khiltd
ParticipantYes, there is a “Nice” key as well as the “HardResourceLimits” set of keys. I’m not certain that anything specified in either of these will apply to processes other than the shell in which your script is being executed, so while your script might get a lower priority, the tools it fires off might not. I haven’t really played with it so I don’t know.
Loop unrolling is an optimization technique which strives to increase execution speed by eliminating superfluous conditional branch instructions. Such a feature would not be beneficial in this case even if it did exist, but no, shell scripts are interpreted, not compiled, and the commands contained within them are not issued asynchronously unless a given command explicitly forks or daemonizes itself. vpnd is one such tool which does this regardless of the command line options it is supplied, and so it does not get along well with launchd at all. What awstats.pl does is something which might warrant further investigation, but log analyzers are not usually very resource friendly.
It should be fairly simple to determine whether or not your script is running. Just send some output to stdout and watch the logs.
January 24, 2008 at 8:09 pm #371257Bill Eccles
ParticipantSee above… We must have been typing at about the same time.
🙂
January 24, 2008 at 9:04 pm #371258khiltd
ParticipantSo you replaced the entire contents of your script file with that single line? If a launchd job does not run for at least 30 seconds then launchd assumes that it has died and attempts to restart it repeatedly. If you do nothing but log a single line, then it’s not going to work out very well. Similar behavior will be observed if you fork or daemonize as stated in the article:
[i]”Jobs run from launchd should not duplicate launchd functionality; for instance, they should not use chroot(2). Furthermore, they should not do the things normally required of daemon processes, such as detaching from the terminal they are initially attached to. The only things that are strictly prohibited, however, are fork()/exit() combinations (including indirect methods, such as the daemon(3) library call). A server which attempts to run itself as a daemon in this way will seem to have finished running, potentially leading to launchd respawning it, or disabling the service.”[/i]
Try something like
[code]#! /bin/sh
while true
do
echo “Pointless script running…”
sleep 10
done[/code]
January 24, 2008 at 10:55 pm #371260Bill Eccles
ParticipantWell, I’m not on the machine where I attempt to figure this kind of stuff out, but it seems to me that if launchd were running and rerunning the script, shouldn’t I see the result of the logging statement in the log repeatedly? It does make an entry in system.log when I run the script manually, so I know it functions as expected.
January 25, 2008 at 5:32 am #371273khiltd
ParticipantWell, we’re even further from that machine, but attempting to sort it out on another one will at least tell you whether it’s something generally wrong with that specific system or if it’s something you’re doing incorrectly. A more generalized troubleshooting approach may very well be in order if launchd is indeed failing to run a simple job.
January 25, 2008 at 1:34 pm #371277Bill Eccles
ParticipantActually, you raise an interesting point. This is the second of two machines that I’ve tried this on. The first is an OSX Server box, 10.5.1, and this one, the second, is a vanilla OSX box, 10.5.1 also. Both had the same results, except when I horked the OSX Server box, I was 20 miles away from it and couldn’t get to it to reset it!
Yikes… And again I say, Yikes…
January 25, 2008 at 2:22 pm #371279Bill Eccles
ParticipantPerhaps… perhaps if you (or someone else) have a script and accompanying plist that I could experiment with, one that you know works on your machine, I could try that. I’m still unconvinced that my plist is completely correct, that there’s not something odd in it that launchd is hanging up on.
Thanks,
BillJanuary 25, 2008 at 5:04 pm #371282khiltd
ParticipantThe script I pasted above works, and Lingon can build a working plist for you with a short series of clicks. What you have there looks alright to me, but it’s possible you might have some garbage characters in the file or something depending on what you created it with.
-
AuthorPosts
- You must be logged in to reply to this topic.
Comments are closed