Home Forums OS X Server and Client Discussion Questions and Answers Some Basic launchd questions/Apache log rotation

Viewing 15 posts - 1 through 15 (of 19 total)
  • Author
    Posts
  • #371177
    Bill Eccles
    Participant

    Gentleones,

    I’ve found the [url=http://developer.apple.com/macosx/launchd.html]article at Apple[/url] which describes how launchd works, and it does a pretty good job. Since, however, I’ve never created a launchd job, I’ve got some basic questions which were not answered by this article.

    First, if I’m just in need of running a basic shell script, where’s the best place to put that shell script? Is there a preferred directory for that type of thing, especially considering it’s a system-wide, needs-to-run-even-when-no-users-are-logged-in script?

    Second, my plist should go into /Library/LaunchDaemons, right?

    By way of background, I find the log rotation scheme offered by Apache/Apache2 to be annoying, so I’m working on making a script that rotates the httpd logs nicely in the same way that all other logs are rotated with the added catch that I have to update AWStats before rotating logs on through. If anybody has done this kind of thing before, I’d love to hear of it.

    Thanks,
    Bill

    #371219
    Bill Eccles
    Participant

    Well, that was interesting.

    I created a script, below, and a .plist, also below, and placed them in /usr/local/bin and /Library/LaunchAgents, respectively. When I issue
    [code]launchctl load /Library/LaunchAgents/net.eccles.rolllogs[/code]
    it causes launchd (and I don’t know if it’s a new process or launchd with a PID of 1) to consume 100% of a CPU forever. No new programs can run because nothing can get launched. So I have to 5-second-hold-the-power-button reboot, and the machine boots fine. But when I try to login, again, launchd goes out to launch… er, “lunch.” (Great. Now I’m hungry.)

    I had a few episodes earlier in the day where I ran the script manually and it ran fine except that it griped because it didn’t have sufficient privileges to modify the /var/log/apache2 directory contents. So I ran it as sudo and it went off into la la land, never to return. Eventually, I rebooted and tried again, but this time it worked. A “top” showed that gzip was consuming CPU, as might be expected.

    Is there something in my plist which would send launchctl out to lunch? Is there something in my script which looks crazy?

    Thanks in advance,
    Bill

    rolllogs.sh
    [code]
    #!/bin/bash

    # Update statistics for AWStats…
    cd /Library/WebServer/awstats/wwwroot/cgi-bin
    for i in `ls *.conf | sed -e ‘s/awstats.//’ -e ‘s/.conf//’` ; do
    echo -n “${i}”;
    /Library/WebServer/awstats/wwwroot/cgi-bin/awstats.pl -update -config=”$i”
    done

    cd /var/log/apache2/

    for i in `ls *_log` ; do
    echo “$i”;
    if [ -f “${i}” ]; then
    echo -n ” ${i}”
    if [ -x /usr/bin/gzip ]; then gzext=”.gz”; else gzext=””; fi
    if [ -f “${i}.6${gzext}” ]; then mv -f “${i}.6${gzext}” “${i}.7${gzext}”; fi
    if [ -f “${i}.5${gzext}” ]; then mv -f “${i}.5${gzext}” “${i}.6${gzext}”; fi
    if [ -f “${i}.4${gzext}” ]; then mv -f “${i}.4${gzext}” “${i}.5${gzext}”; fi
    if [ -f “${i}.3${gzext}” ]; then mv -f “${i}.3${gzext}” “${i}.4${gzext}”; fi
    if [ -f “${i}.2${gzext}” ]; then mv -f “${i}.2${gzext}” “${i}.3${gzext}”; fi
    if [ -f “${i}.1${gzext}” ]; then mv -f “${i}.1${gzext}” “${i}.2${gzext}”; fi
    if [ -f “${i}.0” ]; then mv -f “${i}.0” “${i}.1” && if [ -x /usr/bin/gzip ]; then gzip -9 “${i}.1”; fi; fi
    if [ -f “${i}” ]; then mv -f “${i}” “${i}.0”; fi
    fi

    apachectl graceful

    done
    [/code]
    and the plist, net.eccles.rolllogs.plist
    [code]


    Label
    net.eccles.rolllogs
    ProgramArguments

    /usr/local/bin/rolllogs.sh

    StartCalendarInterval

    Day
    0
    Hour
    3
    Minute
    15

    [/code]

    #371221
    Bill Eccles
    Participant

    It’s not PID 1 that goes to 100%. It’s #108, so I don’t know what’s going on.

    I tried “launchd bash” per Apple’s recommendation, but every time I try that, launchd only tells me that it’s not meant to be run directly.

    /Bill

    #371242
    Bill Eccles
    Participant

    It’s another instance of launchd.

    Right now on this machine I have two instances of launchd running:

    1 ?? 0:05.56 /sbin/launchd
    114 ?? 0:02.15 /sbin/launchd

    I’m not where I can send this machine into hysteria right now (it’s my primary work machine), but if I get a chance, I’ll try launchctl and see if it’s 114 that sucks up a CPU or if launchctl causes a new launchd to launch and suck up a CPU.

    In either case, it seems to me that I’ve got something very, very wrong in either my script or my plist. Yesterday, I tried to set launchd logging level to debug, but still nothing showed up in system.log. (Evidence from Apple seems to suggest that there should be another log for launchd somewhere, but I can’t find it.) And, though I tried to “launchd bash” per Apple’s suggestion that doesn’t work–it just gripes that it can’t be run directly.

    Yikes…

    #371248
    khiltd
    Participant

    Well that script is basically two completely unyielding loops, both of which fire off other CPU hogging processes repeatedly. I’d suggest making friends with nice and renice to lower your scheduling priority.

    #371250
    Bill Eccles
    Participant

    OK, if I understand the plist correctly, that’s something I can do in the launchd plist for this.

    Since it’s a shell script, I presume there’s no automatic loop unrolling or anything like that, though, right? So only one instance of awstats or one instance of gzip would be running, correct?

    In either case, when I run this script manually, CPU is never above 30% for the entire time that the thing is running, and I can see each of the perl and gzip processes being executed, so I guess that begs the question, why is a launchd process hogging the CPU? If I use launchd to fire off the script, does it somehow hide all of what it’s doing, such as the perl and gzipping? Or should I be able to see them running as they’re fired off?

    I don’t even think my script gets to being run, quite frankly.

    #371252
    Bill Eccles
    Participant

    Suspicion confirmed. The script is never getting called, and yet launchd takes up 100% CPU and even bounces back and forth between CPUs.

    I replaced my complex-ish rolllogs.sh script with this:

    [code]
    #!/bin/bash
    logger TestMessage
    [/code]

    And it works fine if I merely sudo /usr/local/bin/rolllogs.sh.

    But if I launchd load net.eccles.rolllogs.plist, launchd goes out to lunch.

    Does anybody have any idea what’s happening?

    Thanks,
    Bill

    #371253
    khiltd
    Participant

    Yes, there is a “Nice” key as well as the “HardResourceLimits” set of keys. I’m not certain that anything specified in either of these will apply to processes other than the shell in which your script is being executed, so while your script might get a lower priority, the tools it fires off might not. I haven’t really played with it so I don’t know.

    Loop unrolling is an optimization technique which strives to increase execution speed by eliminating superfluous conditional branch instructions. Such a feature would not be beneficial in this case even if it did exist, but no, shell scripts are interpreted, not compiled, and the commands contained within them are not issued asynchronously unless a given command explicitly forks or daemonizes itself. vpnd is one such tool which does this regardless of the command line options it is supplied, and so it does not get along well with launchd at all. What awstats.pl does is something which might warrant further investigation, but log analyzers are not usually very resource friendly.

    It should be fairly simple to determine whether or not your script is running. Just send some output to stdout and watch the logs.

    #371257
    Bill Eccles
    Participant

    See above… We must have been typing at about the same time.

    🙂

    #371258
    khiltd
    Participant

    So you replaced the entire contents of your script file with that single line? If a launchd job does not run for at least 30 seconds then launchd assumes that it has died and attempts to restart it repeatedly. If you do nothing but log a single line, then it’s not going to work out very well. Similar behavior will be observed if you fork or daemonize as stated in the article:

    [i]”Jobs run from launchd should not duplicate launchd functionality; for instance, they should not use chroot(2). Furthermore, they should not do the things normally required of daemon processes, such as detaching from the terminal they are initially attached to. The only things that are strictly prohibited, however, are fork()/exit() combinations (including indirect methods, such as the daemon(3) library call). A server which attempts to run itself as a daemon in this way will seem to have finished running, potentially leading to launchd respawning it, or disabling the service.”[/i]

    Try something like

    [code]#! /bin/sh

    while true

    do

    echo “Pointless script running…”

    sleep 10

    done[/code]

    #371260
    Bill Eccles
    Participant

    Well, I’m not on the machine where I attempt to figure this kind of stuff out, but it seems to me that if launchd were running and rerunning the script, shouldn’t I see the result of the logging statement in the log repeatedly? It does make an entry in system.log when I run the script manually, so I know it functions as expected.

    #371273
    khiltd
    Participant

    Well, we’re even further from that machine, but attempting to sort it out on another one will at least tell you whether it’s something generally wrong with that specific system or if it’s something you’re doing incorrectly. A more generalized troubleshooting approach may very well be in order if launchd is indeed failing to run a simple job.

    #371277
    Bill Eccles
    Participant

    Actually, you raise an interesting point. This is the second of two machines that I’ve tried this on. The first is an OSX Server box, 10.5.1, and this one, the second, is a vanilla OSX box, 10.5.1 also. Both had the same results, except when I horked the OSX Server box, I was 20 miles away from it and couldn’t get to it to reset it!

    Yikes… And again I say, Yikes…

    #371279
    Bill Eccles
    Participant

    Perhaps… perhaps if you (or someone else) have a script and accompanying plist that I could experiment with, one that you know works on your machine, I could try that. I’m still unconvinced that my plist is completely correct, that there’s not something odd in it that launchd is hanging up on.

    Thanks,
    Bill

    #371282
    khiltd
    Participant

    The script I pasted above works, and Lingon can build a working plist for you with a short series of clicks. What you have there looks alright to me, but it’s possible you might have some garbage characters in the file or something depending on what you created it with.

Viewing 15 posts - 1 through 15 (of 19 total)
  • You must be logged in to reply to this topic.

Comments are closed