Keeping logs of all your traffic
Although we’ve mentioned AWStats a number of times before, we’ve never really gotten into how to set it up. While this isn’t exceptionally hard, it’s nice to have someone go there before you if it’s your first time out. So, here’s a rundown of installing AWStats on OS X Server 10.4.
Take special note of step 16 which tells you to create a web realm for your AWStats install. Otherwise you stand a good chance of getting bitten by some of the AWStats exploits that crop up from time to time. The Apple documentation on this should make it quite easy for you to set up the realm. These instructions were created using 10.4.7 server which currently runs perl version 5.8.6.
1. Download the latest version of AWStats from http://awstats.sourceforge.net/– at the time of writing, this was 6.5. You can download either the .zip file or the .tar.gz.
2. Extract the awstats file and move the awstats folder into /usr/local/. Change the name of the folder to just awstats
sudo mv awstats-6.5 /usr/local/awstats
3. Change into the /usr/local/awstats/tools folder and run the following command :
sudo perl awstats_configure.pl
4. Enter “Y” when prompted to continue install from NON standard directory.
5. Enter “Y” when prompted to created a new config file. Give your web site name or profile name when prompted – ie SuperDupaHostaStats or your.website.here
6. Press enter to finish the initial config.
7. We now need to verify that the main parameters in your newly created configuration file match our needs. The newly created configuration file lives here /usr/local/awstats/wwwroot/cgi-bin/awstats.SuperDupaHostaStats.conf with whatever name you entered in step 5 substituted for SuperDupaHostaStats.
8. Change the LogFile line to:
LogFile="/var/log/httpd/access_log"
9. Check that the SiteDomain variable is set to your domain you wish to generate stats for (by default set to the name entered in step 5).
10. We now need to change the location that the awstats_updateall.pl script looks for our config files. Edit the file /usr/local/awstats/tools/awstats_updateall.pl and change the DIRCONFIG variable from etc/awstats to /usr/local/awstats/wwwroot/cgi-bin.
11. Create the awstats database directory sudo mkdir /var/lib and sudo mkdir /var/lib/awstats. Assign ownership of this directory to the www user:
sudo chown www /var/lib/awstats
12. Change your web server logging to the combined format → Using server admin, edit the logging section of your website and select “combined” as the logging format. This is probably the most important step, if this is not done, the awstats script that parses the access_log file will fail with errors like “Your log file /var/log/httpd/access_log must have a bad format or LogFormat parameter setup does not match this format.” Note that if you already have a access_log file with data in it, you will probably need to archive it and start the stats from the time you change the log format.
13. You should now be able to run the awstats_updateall.pl script to create the awstats database. Run the following command
sudo /usr/local/awstats/tools/awstats_updateall.pl now
If you receive errors about log format being incorrect and you have changed the log format in the web server module of Server Admin, and make sure:
  a. You have restarted the web server and
  b. That you don’t have the “old” log format logs still present in the access_log file. If you do, you will need to remove them.
14. If the awstats database was successfully created, you should see some output like:
Running '"/usr/local/awstats/wwwroot/cgi-bin/awstats.pl" -update -config=SuperDupaHostaStats -configdir="/usr/local/awstats/wwwroot/cgi-bin"' to update config SuperDupaHostaStats Update for config "/usr/local/awstats/wwwroot/cgi-bin/awstats.SuperDupaHostaStats.conf" With data in log file "/var/log/httpd/access_log"... Phase 1 : First bypass old records, searching new record... Searching new records from beginning of log file... Phase 2 : Now process new records (Flush history on disk after 20000 hosts)... Jumped lines in file: 0 Parsed lines in file: 58 Found 0 dropped records, Found 49 corrupted records, Found 0 old records, Found 9 new qualified records.
15. You can then check your awstats page by browsing to http://your.web.server/awstats/awstats.pl?config=SuperDupaHostaStats
Of course, replacing your.web.server with the name/ip address of your web server and SuperDupaHostaStats with the name you entered in step 5.
16. Once you have verified that awstats are working ok, you will need to create a realm so that the awstats data is at least password protected (very easy to do with server admin, so I won’t go into detail on how to do that). You will also want to setup a cron job so that the awstats database is updated every day at least.
17. Create cron job for awstats database. Enter the following command:
sudo crontab -e
This will open roots crontab entry using the vi text editor (unless you have changed the default crontab editor). Enter editing mode (by pressing “i”) and enter the following:
0 0 * * * /usr/local/awstats/tools/awstats_updateall.pl now
This will make awstats update the database once a day every day at 12:00 am. You can change the above settings according to your preferences – ie update awstats database every hour or whatever. Check here for a good quick reference to using crontab http://www.adminschoice.com/docs/crontab.htm
18. As noted in the original AFP548 article reviewing awstats https://www.afp548.com/articles/reviews/awstats.html you will need to disable performance cache for your web server.
Thats it!!! All things going to plan, you should have awstats running successfully on your machine.
As an added bonus, this setup can also be used to generate stats for virtual sites as well.
19. When you create your virtual site, specify an unique access and error log file for each site (using Server Admin), remember to change the logging format to “combined”.
20. For each virtual site, create your awstats config file:
sudo cp /usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf<br /> /usr/local/awstats/wwwroot/cgi-bin/awstats.your.virtual.site.conf
21. Modify the settings in this file as per steps 8 and 9 above, noting that the log file will be different than in step 8, it will match what you entered in server admin in step 19.
22. Thats it!! You can manually run sudo /usr/local/awstats/tools/awstats_updateall.pl now to generate the awstats database for your virtual site or just wait for the cron job you setup earlier to run its course. You can access the awstats for your virtual site at http://your.virtual.site/awstats/awstats.pl?config=your.virtual.site of course the config=your.virtual.site must match the name given to the new awstats config file created in step 20 (without the awstats and the conf).
crontabs are soooooo passe…
here is a sample LaunchDaemon plist
in my case – I call it
edu.csupomona.class.awstats-update.plist
shove it into /System/Library/LaunchDaemons
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://
http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>edu.csupomona.class.awstats-update</string>
<key>ProgramArguments</key>
<array>
<string>/Library/Scripts/awstats-update.sh</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>ServiceDescription</key>
<string>awstats updates</string>
<key>StartInterval</key>
<integer>3600</integer>
</dict>
</plist>
the script it calls is here….
http://www.class.csupomona.edu is my config/site file…
LOGGER=$(which logger)
${LOGGER} -t AWSTATS "Starting awstats"
${LOGGER} -t AWSTATS "Rebuilding the db"
/Library/WebServer/awstats/wwwroot/cgi-bin/awstats.pl \
-config=http://www.class.csupomona.edu \
-update
${LOGGER} -t AWSTATS "db update complete"
${LOGGER} -t AWSTATS "Building web pages"
/Library/WebServer/awstats/tools/awstats_buildstaticpages.pl \
-config=http://www.class.csupomona.edu \
-update \
-dir=/Groups/CLASS/statistics \
-awstatsprog=/Library/WebServer/awstats/wwwroot/cgi-bin/awstats.pl
${LOGGER} -t AWSTATS "Completed awstats run"
I put this script in /Library/Scripts
lock it down
chown it to root and chmod it to 700
I keep awstats in /Library/WebServer/awstats not in /usr/local/awstats
Make sure your mods to /etc/httpd/httpd.conf reflect your appropriate
directories…
the configure perl script do mod your apache config…
you can visit my stats page for awstats for my web boxen at
http://www.class.csupomona.edu/awstats/awstats.pl?
config=http://www.class.csupomona.edu
or
http://www.class.csupomona.edu/statistics/
awstats.http://www.class.csupomona.edu.html
I have successfully set up awstats to scan/read PureFTPd if you have
questions about that email me here or post more comments
Just what I needed to get started on analyzing my httpd log files.
One thing that I do not know how to deal with is how to get AWStats to look
at my archived logfiles.
I know that the filename is EG "access_log.1158624000" the latter digits
being a unix timestamp (seconds from 1/1-1970) and I assume the the
periodic scripts are responsible for actually rolling the files (to *.gz at some
time later I think)
What is the best way to be able to use the archiving scripts as well as having
all the log files analyzed from AWStat before being gz’ed – or what is the
correct procedure in general…?
I am getting a 403 error when I try and access
http://www.domain.com/awstats/awstats.pl?config=www.domain.com
I did follow the steps any one have any idea how to fix this?
this is what is in my log..
[Tue Oct 10 16:36:36 2006] [error] [client 65.173.238.2] (13)Permission
denied: access to /awstats/awstats.pl failed because search permissions are
missing on a component of the path
lol..
found my error when reading the forums for awstats when I moved the awstats
directory to /usr/local the permissions were 700, I changed it to 755 and all is
good.
working for me now.. thanks.
Worked like a charm for me. Thanks for the writeup.
—
Matthew Kosterman • Chief Executive Geek • DeltaQuest Imaging, Inc.
. . . broke the Web option in Server Admin.
Now I get a gray dot with "Not Available" in the status
display and no access to settings.
servermgrd throws a nil reference error when trying to do anything regarding the Web service.
Web services are still running but servermgrd can’t apparently
see them. And yes, I tried a restart, among other things.
Any ideas for me?
—
Paul E. Compton — University of Oregon
[email protected]
Okay I follow this guide step by step..
But when I get to step 13
(sudo /usr/local/awstats/tools/awstats_updateall.pl now) I get this error:
Running ‘"/usr/local/awstats/wwwroot/cgi-bin/awstats.pl" -update –
config=domain.com -configdir="/usr/local/awstats/wwwroot/cgi-bin"’ to
update config domain.com
sh: line 1: /usr/local/awstats/wwwroot/cgi-bin/awstats.pl: Permission denied
Should I change permissions on some folder(s)?
Sorry for the late addition. How CAN I set up realms for vhosts? I can manage the .htaccess, however it’s unclear how I set it up.