Producing Better Web Statistics with Report Magic

—by David O’Donnell, atropos@afp548.com

25 September 2002—Updated 12 January 2003

If you host Web sites, chances are that at some point, your clients have asked for usage statistics. Or, perhaps, you’ve been interested in making some sense of all those lines in your logs yourself. Regardless of the prompt, collecting and analyzing Web statistics is “a good thing.” There are a number of Web statistics analysis programs out there, some of which are fairly pricey. In keeping with the Open Source mindset, we recommend Analog and have an article on setting it up on Mac OS X Server.

Analog does a great job by itself, but I’ve always hankered for something more. The answer to that yearning can be found in Report Magic by Wadsack-Allen. Report Magic takes Analog’s output and produces some stunning visuals, as well as cogent textual analysis. While it may not be earth-shattering, I personally find it pretty interesting to know that, e.g., as of 25 September 2002 AFP548.com had been visited by people on 33,514 unique Internet hosts since it went live on 16 January 2002. Now, if only some more of you visitors would write articles or post in the eBBS… :-)

Installing Report Magic is a fairly involved and lengthy process. The steps outlined below are a result of reading the documentation, trial, and a fair amount of error. If you find any errors in the instructions—or learn that newer versions of support packages than the ones listed work—please let me know!

One of the secrets to success in this process is an up-to-date Perl CPAN installaton. If you haven’t updated CPAN lately, or are working with a fresh Jaguar installation, you should do so now. A new Jaguar (and thus CPAN) installation will require more time, but you will appreciate the effort in the end. New CPANners should install copies of NcFTP and Lynx in advance. (See my UNIX Tool Chest article for more information on these useful utilities.)

Report Magic needs several system libraries to be present for it to work. Apple kindly has included one of them—zlib 1.1.3, which provides compression hooks—with Mac OS X Server 10.2. The others will need to be obtained and installed.

First, we need to install libpng. This library provides support for generating images in the PNG format. PNG is relatively new, but has several advantages over GIF: it isn’t encumbered by unpleasant licensing restrictions imposed by self-serving megacorporations, it is a fairly compact standard, it can generate 8- or 24-bit compressed images, and it supports alpha-channel transparency (à la Photoshop). In fact, were it not for the fact that Microsoft cannot seem to write a decent PNG-support function into its browsers (virtually everyone else can…), there would be zero reason to use GIF images at all any more. Enough of my soapbox ranting; Report Magic wants to be able to make PNG images, so we need to get and install the library.

As of this writing, the latest stable build of libpng is version 1.2.4. Retrieve it from download.sourceforge.net and place the resulting source in a directory you’ve already created for the Report Magic project:

mkdir -p reportmagic 
cd reportmagic 
curl -O http://download.sourceforge.net/libpng/libpng-1.2.4.tar.gz 
gnutar zxvf libpng-.1.2.4.tar.gz 
cd libpng-1.2.4

Libpng wants to know where zlib is, and by default it looks in some fairly standard locations. Unfortunately, they aren’t where Apple has placed its libraries, so we need to modify the supplied Mac OS X makefile after copying it into the default makefile.

cp scripts/makefile.macosx makefile

Edit the makefile with your favourite command-line editor, and search for the line where ZLIBLIB is set. Change the definition to the following:

ZLIBLIB=/usr/lib

Next, change the definition for ZLIBINC:

ZLIBINC=/usr/include

You may also want to change the definition for MANPATH to keep libpng’s manpages in a location recognized by default under Mac OS X 10.2:

MANPATH=/usr/share/man

Create the library using the following commands:

make test 
sudo make install 
sudo ranlib /usr/local/lib/libpng.a

That’s one down?

To give Report Magic the ability to draw JPEG graphics, we’ll need to install libjpeg. The latest version, 6b, installs cleanly and without modification under Mac OS X Server 10.2:

cd .. ; 
#only needed if you’re still in libpng’s directory 
curl -O ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6b.tar.gz 
gnutar zxvf jpegsrc.v6b.tar.gz cd jpeg-6b 
sudo ./configure 
sudo make 
sudo make install 
sudo make install-lib

We obtained libpng and libjpeg for one purpose: so they could be called by GD (libgd), an open source library which allows programmers to easily generate PNG, JPEG, and WBMP images from many different programming languages. Libgd is very important to Report Magic. The latest version, 2.0.1, does not currently compile cleanly under Mac OS X (sadly) but it is also clearly labeled as beta software. We’ll get and install version 1.8.4, the recommended version.

cd .. ; 
#only needed if you’re still in libjpeg’s directory 
curl -O http://www.boutell.com/gd/http/gd-1.8.4.tar.gz 
gnutar zxvf gd-1.8.4.tar.gz 
cd gd-1.8.4

GD is really designed to work in an X-Windows environment. If you have X-Windows installed on your Server, more power to you; for our purposes, though, we need to remove references to it from the Makefile in GD’s source tree. We also need to tell GD where to look for Apple’s libraries and the ones we’ve just installed, and explicitly tell it we have libjpeg support.

Open Makefile in your favourite command-line editor and search for the CFLAGS declaration. If it isn’t already in the line, add -DHAVE_LIBJPEG to it. Search for the LIBS declaration and likewise ensure the line contains -ljpeg.

Now search for the INCLUDEDIRS declaration. Remove references to X Windows and FreeType, and add references to the /usr/include and /usr/local/include directories. The line should end up looking like follows:

INCLUDEDIRS=-I. -I/usr/local/include -I/usr/include

Likewise, we will need to modify the LIBDIRS declaration. Remove references to X Windows and FreeType, and include references to /usr/lib and /usr/local/lib. Our line ended up like this:

LIBDIRS=-L. -L/usr/local/lib -L/usr/lib/

Save the file and execute these commands:

sudo make 
sudo make install

You’re done with GD. Now we can turn to the Perl bits! Remember that Report Magic is written in Perl; as such, it expects certain Perl support routines to be there. They aren’t included by default (big surprise!) but can all be installed using the CPAN network. Jump into CPAN:

sudo perl -MCPAN -e shell

Report Magic uses the GD library to draw the pretty pictures it uses, so unsurprisingly it relies on the Perl GD library to call the system-level GD library. Unfortunately, the folks on CPAN have made the default GD installation for GD version 2.0.1—the beta software we mentioned above that doesn’t install under Mac OS X. Luckily, an older release that works with libgd 1.8.4 is out there. Tell CPAN to install it:

install L/LD/LDS/GD-1.41.tar.gz

When prompted, tell GD that you have JPEG and PNG support, but no others.

Next, tell CPAN to install the GD::Text module. The latest version was 0.83 when I installed Report Magic; use the command

install GD::Text

to install it. If you have problems, you can use

force install M/MV/MVERB/GDTextUtil-0.80.tar.gz

to install an older version.

Going along with GD text support is support for 2D and 3D graphs:

install GD::Graph install GD::Graph3d

One of the peculiarities (to my mind) of Report Magic is that it uses a Windows-style “.ini” configuration file format. Perl has a module that supports INI files, believe it or not, but the current version doesn’t work under Mac OS X 10.2 (perhaps the OS rebels :). Regardless, an older version does work—as may newer versions, but this one achieves the desired result:

force install W/WA/WADG/Config-IniFiles-2.22.tar.gz

We need to force install it because CPAN won’t install modules that fail above a certain percentage of tests; the tests that this fails are (as far as I can tell!) irrelevant to Report Magic actually working.

Finally, install these three modules:

install HTML::TagSet 
install HTML::Parser 
install File::Spec

You can now exit CPAN. Our work here is done.

Now we can get and install Report Magic!

cd .. ; 
# just to make sure we’re not still in GD’s directory 
curl -O http://www.reportmagic.org/rmagic-2.15.tar.gz 
gnutar zxvf rmagic-2.15.tar.gz 
cd rmagic-2.15 
sudo perl Install.PL -no_modules

Note that we use the -no_modules flag to prevent Report Magic’s installer from trying to update our Perl modules, which would screw everything up.

When all’s said and done, you’ve got a lovely copy of Report Magic in /usr/local/bin/rmagic-2.15. You might want to change the directory’s name to something less version-dependent, like rmagic, but it’s a matter of personal preference.

Configuring Report Magic from this point forward doesn’t deviate from the supplied documentation, so I’ll leave that task to you:

cd /usr/local/bin/$RMAGIC-DIRECTORY ; 
# rmagic-2.15 or whatever you called it 
open docs/overview.html

Scroll down to “Configuration” and have fun! If you’d like to see AFP548.com’s Report Magic output, click here.