rotatelog on OS X Server (Archive every 7 days)(awstats makes my head ache)

[l008com]

OK after finding a really easy to follow how to on some obscure mac site, I was finally able to get awstats up and running on my server. The docs awstats come with just plain suck. So now I have awstats up and running and I have the next logical problem. Combined log format logs take up a TON of space, and my most popular site gets 8000+ page views a day. So of course I NEED log rotation. The problem is, when you rotate your logs, you loose all the traffic (in stats) that happened between that last awstats update and the log rotation. What awstats tells you to do is to have awstats run right before it rotates the logs. It gives you very brief instructions on how to do this, which don’t seem to apply to OS X at all. Now, I have a cron script that updates my awstats currently every day at midnight. The Server Admin app will let me set the number of days to archive the logs. But it doesn’t tell me when exactly it does it. I’m not sure which avenue to go down at this point. But the best one I can think of is to disable the automatic log rotation in Server Admin, and then figure out the commands to run to manually rotate the logs, and stick that code right in my awstats updater script, so every day, at midnight, one by one, every site’s awstats will be updated, then that sites logs will be rotated. There may very well be a better solution, however I’m not aware of it.

[puxuradude6]

Since noone is commenting on your problem I’ll throw in my thoughts.

What I found out by reading the daily.out log file from the periodic program that apple runs with cron is that it does rotate the log files at that time which is 3.15. I set the server admin to rotate the logs once a day since I can’t figure out how to do the calculations otherwise. The log file my OS X server creates is names access.log.[numbers of seconds from 1970]. If you set the log file to rotate every week you’ll see that the next day awstats increases the number of seconds, therefore missing the log files. I tried doing a calculation using the day variable but awstats can’t handle those so I’m out of luck.

I now added a cron task that reads my access.log.[insert crappy second counting here] 5 minutes before the log is rotated and will see and wait if this works out.

I am really curious how the guys fmro afp548 set this up themselves. There are no tutorials online, neither can you find *any* examples of the config file online. Awstats themselves seem pretty ignorant about it as well. They added the seconds counting (which is obviously only for mac) as a variable but give no examples or references about how to use it properly in an OS X server environment.

I’ll get back to you about how my little cron job is doing, realising I’m missing 5 minutes of logs every day but I guess you can’t risk awstats reading the log file while OS X is deleting it…

[puxuradude6]

Since noone is commenting on your problem I’ll throw in my thoughts.

What I found out by reading the daily.out log file from the periodic program that apple runs with cron is that it does rotate the log files at that time which is 3.15. I set the server admin to rotate the logs once a day since I can’t figure out how to do the calculations otherwise. The log file my OS X server creates is names access.log.[numbers of seconds from 1970]. If you set the log file to rotate every week you’ll see that the next day awstats increases the number of seconds, therefore missing the log files. I tried doing a calculation using the day variable but awstats can’t handle those so I’m out of luck.

I now added a cron task that reads my access.log.[insert crappy second counting here] 5 minutes before the log is rotated and will see and wait if this works out.

I am really curious how the guys fmro afp548 set this up themselves. There are no tutorials online, neither can you find *any* examples of the config file online. Awstats themselves seem pretty ignorant about it as well. They added the seconds counting (which is obviously only for mac) as a variable but give no examples or references about how to use it properly in an OS X server environment.

I’ll get back to you about how my little cron job is doing, realising I’m missing 5 minutes of logs every day but I guess you can’t risk awstats reading the log file while OS X is deleting it…

[AaronAdams]

I’m not sure if this is an acceptable solution for you, but I’ll offer it up.

Instead of rotating the logs, I use awstats to purge log records it has already processed. In the /etc/awstats/awstats.sitename.conf file, there are two sections to control this:

# AWStats can purge log file, after analyzing it. Note that AWStats is able
# to detect new lines in a log file, to process only them, so you can launch
# AWStats as often as you want, even with this parameter to 0.
# With 0, no purge is made, so you must use a scheduled task or a web server
# that make this purge frequently.
# With 1, the purge of the log file is made each time AWStats update is ran.
# This parameter doesn't work with IIS (This web server doesn't let its log
# file to be purged).
# Change : Effective for new updates only
# Possible values: 0 or 1
# Default: 0
# 
PurgeLogFile=1

# When PurgeLogFile is setup to 1, AWStats will clean your log file after
# processing it. You can however keep an archive file (saved in "DirData") of
# all processed log records by setting this to 1 (For example if you want to
# use another log analyzer).
# This parameter is not used if PurgeLogFile=0
# Change : Effective for new updates only
# Possible values: 0 or 1
# Default: 0
#
ArchiveLogRecords=1

This keeps my access_log small and provides an archive that I can compress and save for later if I need it.

[bubbleboy]

My head is aching and about to explode.

I keep getting an error when I try to do an awstats update that says I don’t have the right log file type defined. I’ve tried just using combined but OSX’s combined format doesn’t seem to match. I tried just copying and pasting the log format string that shows up in the Server Admin (then escaping the quotes) and it doesn’t seem to like that either.
Anyone been through this and have a fix?

Your AWStats LogFormat parameter is:
%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"
This means each line in your web server log file need to have the following personalized log format:
%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"
And this is an example of records AWStats found in your log file (the record number 50 in your log):
24.8.179.250 – – [13/Aug/2005:04:58:50 -0600] "GET /surveys/labmembers/ HTTP/1.0" 200 639

[Kyle]

[QUOTE BY= bubbleboy] My head is aching and about to explode.

I keep getting an error when I try to do an awstats update that says I don’t have the right log file type defined. I’ve tried just using combined but OSX’s combined format doesn’t seem to match. I tried just copying and pasting the log format string that shows up in the Server Admin (then escaping the quotes) and it doesn’t seem to like that either.
Anyone been through this and have a fix?

Your AWStats LogFormat parameter is:
%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”
This means each line in your web server log file need to have the following personalized log format:
%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”
And this is an example of records AWStats found in your log file (the record number 50 in your log):
24.8.179.250 – – [13/Aug/2005:04:58:50 -0600] “GET /surveys/labmembers/ HTTP/1.0” 200 639[/QUOTE]

This is just a wild guess…
From what I can see on your error statement, it looks to me like there could be a problem on %t
You see, there is a spaec between 58:50 and -0600 and there’s nothing saying to group that with “square bracets” (if that’s the correct term).
Also, it does have more terms on the syntax than in the log, so I think you have to remove the extras.

So maybe the correct line would be this instead:
%h %l %u [%t] “%r” %s %b

Let me know if it works.

[sky]

just ran the autoconfig script for 6.5… buggy! beware!

[Anonymous]

Thank you for a very helpful tip! awstats now seems to read the rotated logs which have the system time appended to their names. We shall see how this goes over the next few weeks.

[QUOTE BY= macshome] We roll your logs weekly and retain one week of archive. We read them with the logresolvemerge.pl tool, which sorts and reads the logs. It’s in the tools dir of your awstats deployment. Your log directive in the config file then looks something like:

LogFile="/somepath/awstats/tools/logresolvemerge.pl /var/log/httpd/access_log.* |"

Or thereabouts.[/QUOTE]

[TvE]

[QUOTE][u]Quote by: macshome[/u]

We roll your logs weekly and retain one week of archive. We read them with the logresolvemerge.pl tool, which sorts and reads the logs. It’s in the tools dir of your awstats deployment. Your log directive in the config file then looks something like:

LogFile="/somepath/awstats/tools/logresolvemerge.pl /var/log/httpd/access_log.* |"

Or thereabouts.

[/QUOTE]

I must be missing something essential, since I am unable to get the above to wotk (I have corrected /somepath to a meaningfull path 😉

My error is:
[code]testserver:/var/log/httpd ladmin$ sudo /usr/local/awstats/tools/awstats_updateall.pl now
Password:
Running ‘”/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -update -config=testserver -configdir=”/usr/local/awstats/wwwroot/cgi-bin”‘ to update config testserver
Update for config “/usr/local/awstats/wwwroot/cgi-bin/awstats.testserver.conf”
With data in log file “/usr/local/awstats/tools/logresolvemerge.pl /var/log/httpd/access_log.* |”…
: bad interpreter: No such file or directory.pl: /usr/bin/perl
Phase 1 : First bypass old records, searching new record…
Direct access to last remembered record is out of file.
So searching it from beginning of log file…
Error: Command for pipe ‘/usr/local/awstats/tools/logresolvemerge.pl /var/log/httpd/access_log.* |’ failed
Setup (‘/usr/local/awstats/wwwroot/cgi-bin/awstats.testserver.conf’ file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in ‘docs’ directory).
[/code]

AWStats have been installed (and is otherwise working) according to this nice article:
[url]https://www.afp548.com/article.php?story=20060823205258972[/url]

[TvE]

Hmm – according to this very interesting post: https://sourceforge.net/forum/message.php?msg_id=3518899 if you “just” change the line endings from DOS to UNIX on logresolvemerge.pl it solves the problem (good error description ;-P)

Now the output of my command is totally different:
[quote]testserver:/usr/local/awstats/tools ladmin$ sudo /usr/local/awstats/tools/awstats_updateall.pl now
Running ‘”/usr/local/awstats/wwwroot/cgi-bin/awstats.pl” -update -config=testserver -configdir=”/usr/local/awstats/wwwroot/cgi-bin”‘ to update config testserver
Update for config “/usr/local/awstats/wwwroot/cgi-bin/awstats.testserver.conf”
With data in log file “/usr/local/awstats/tools/logresolvemerge.pl /var/log/httpd/access_log.* |”…
zcat: /var/log/httpd/access_log.1150848000.gz.Z: No such file or directory
Phase 1 : First bypass old records, searching new record…
Direct access to last remembered record has fallen on another record.
So searching new records from beginning of log file…
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)…
Jumped lines in file: 0
Parsed lines in file: 316
Found 0 dropped records,
Found 0 corrupted records,
Found 120 old records,
Found 196 new qualified records.

testserver:/usr/local/awstats/tools ladmin$ [/quote]

Now I just have to remember where I read the solution to the next problem:
zcat: /var/log/httpd/access_log.1150848000.gz.Z: No such file or directory

[TvE]

Which was easy:
logresolvemerge.pl needs a fix on line 87:
From: my $zcat = ‘zcat’;
To: my $zcat = ‘gzcat’;
Reason is explained here: [url]http://www.osxfaq.com/tips/unix-tricks/week106/monday.ws[/url]

[Author]

Posts