AFP548 - Topic: Is Bayesian filtering turned on?

This topic has 41 replies, 18 voices, and was last updated 18 years, 1 month ago by filipp.

Viewing 15 posts - 1 through 15 (of 41 total)

1 2 3 →

Author

Posts
October 1, 2005 at 8:40 pm #363442

thinbits
Participant

I’ve been dutifully training the SA filters by feeding it batches of spam and ham (in mbox format, 200+ at a time), however I don’t think it’s actually filtering anything.

SA is catching a lot of spam, and the X-Spam-Status header is being added with all the tests that ran. However, none of the 700+ spams I just looked at (caught and uncaught) have BAYES_XX in the X-Spam-Status header.

My /private/etc/mail/spamassassin/local.cf file has auto_learn 1 and use_bayes 1, among other settings.

Shouldn’t I be seeing some indication of the bayes filters in the mail headers?

Thanks,
Dave

October 10, 2005 at 2:07 am #363545

thinbits
Participant

I still have not figured out why I’m not getting any bayesian filtering. I tried a test run of SA from the command line on a batch of known spam with the -D debug switch on.

su – clamav -c “spamassassin -D –test –local –mbox /Users/dave/spam3.mbox > spam.test”

debug: bayes: 2187 tie-ing to DB file R/O /var/clamav/.spamassassin/bayes_toks
debug: bayes: 2187 tie-ing to DB file R/O /var/clamav/.spamassassin/bayes_seen
debug: bayes corpus size: nspam = 4310, nham = 3017
debug: tests= BAYES_99,FORGED_RCVD_HELO,HTML_80_90,HTML_MESSAGE,, [lots of other stuff]…

So, when run manually, I can see that bayesian filtering works. It just doesn’t seem to be engaged when mail is normally received by postfix. I have yet to see a BAYES_XX tag added to any spam processed by postfix/amavis.

Any thoughts? Is there some way I can enable SA debug output when called from amavisd?

Thanks,
Dave

October 10, 2005 at 12:23 pm #363550

Anonymous
Guest

ping ping ping.

bump bump bump.

I just noticed the exact same thing. Why are we supposed to train our anti-spam software when it has no effect?

October 10, 2005 at 6:51 pm #363555

thinbits
Participant

I’ve added the “debug” arg to the launchd plist for amavisd, and turned on the sa_debug thing at the bottom of the amavisd.conf file. I now get a lot more logging from amavisd, but I still don’t see any debug logging from SA anyplace I’ve looked.

I’m suspecting there is some sort of problem with SA finding the correct bayes databases. Possibly it’s being run as a user other than clamav, or some sort of path or ownership problem. It would probably be obvious if I could figure out how to get SA to output debug logs when called from amavisd.

Dave

October 13, 2005 at 12:55 pm #363601

Anonymous
Guest

/private/var/amavis/.spamassassin/bayes_journal
/private/var/amavis/.spamassassin/bayes_seen
/private/var/amavis/.spamassassin/bayes_toks

Thar she blows, thinbits.

Those are the amavis users’ bayes files for SA.

On my system the file permissions look kosher. All owned by clamav:clamav and rw for it.

October 13, 2005 at 7:26 pm #363615

Anonymous
Guest

We’ve having the same exact problem. No Bayes on 10.4.2 server! Very strange. Has anyone been able to fix this?

October 14, 2005 at 3:21 pm #363624

thinbits
Participant

That’s interesting… I did a locate on bayes_seen and got the following:

/private/var/amavis/.spamassassin/bayes_seen
/private/var/clamav/.spamassassin/bayes_seen

Looking in those directories I see:

dave% sudo ls -l /private/var/amavis/.spamassassin/
-rw——- 1 clamav clamav 671744 Oct 14 08:13 auto-whitelist
-rw——- 1 clamav clamav 172032 Oct 14 08:03 bayes_seen
-rw——- 1 clamav clamav 1318912 Oct 14 08:03 bayes_toks

dave% sudo ls -l /private/var/clamav/.spamassassin/
-rw——- 1 clamav clamav 655360 Oct 14 01:00 bayes_seen
-rw——- 1 clamav clamav 5349376 Oct 14 01:00 bayes_toks
-rw-r–r– 1 clamav clamav 1175 Oct 9 18:32 user_prefs

From what I can tell, /private/var/clamav is where the results of learn_junk_mail go, (or at least that’s what happens on my machine). I’m wondering if it actually pulling from the /private/var/amavis directory when processing real mail. That would explain the lack of filtering…

I’ll tinker with this today and report my results. I’m hoping we are on to something…

Dave

October 15, 2005 at 12:24 am #363636

thinbits
Participant

I stopped mail, moved /var/amavis/.spamassassin aside and replaced it with a symlink to /var/clamav/.spamassassin.

The next two mails that came in got a BAYES_99 tag. Yay!

So, it appears there is some sort of disconnect between the database path learning takes and what the mail server uses. I’ll let this run for the weeked and see how it turns out.

October 17, 2005 at 4:09 pm #363659

Anonymous
Guest

This is a huge bug. One of the features they advertise doesn’t even work! Do you think a symlink would work? Let me know how this is working for you.

October 18, 2005 at 4:07 am #363664

Anonymous
Guest

This is a huge bug.

Way to go, Thinbits. You earned yourself a high-five. Someone want to blog this somewhere? I hope its fixed in 10.4.3

October 18, 2005 at 2:23 pm #363670

thinbits
Participant

Since placing the symlink, I’m getting almost 100% accuracy with spam filtering. It’s quite amazing! I’m not used to my Inbox not being full every morning.

I reported this as bug # 4305231 at bugreport.apple.com

Maybe if enough people report the bug it will get fixed for 10.4.3

October 18, 2005 at 8:56 pm #363691

Anonymous
Guest

Hmm. I’ve “moved /var/amavis/.spamassassin aside and replaced it with a symlink to /var/clamav/.spamassassin” but no effect. In fact, on my 10.4.2 machine, it seems like there is alot more info in amvis than clamav. Any suggestions?

October 19, 2005 at 1:15 am #363696

pblolo
Participant

[QUOTE BY= thinbits]So, it appears there is some sort of disconnect between the database path learning takes and what the mail server uses. I’ll let this run for the weeked and see how it turns out.[/QUOTE]

Strange, I checked my server, I have only the good .spamassassin folder, which explains why it works fine here also.

Did you install in a special way ?

Here : 10.4.2 server, clean install.

October 19, 2005 at 5:13 pm #363712

aderium
Participant

I only have the .spamassassin in /va/amavis when i move it os .spamassassin_old it recreates it but none in /var/clamav

Also I noticed in /etc a /etc/spam with some files like amavisd.conf what is that there for ?

Thaks

P.S. talking about bugs anybody able to filter the logsi in serveradmin mail portion ?

October 19, 2005 at 6:40 pm #363713

Anonymous
Guest

[QUOTE BY= Aderium] I only have the .spamassassin in /va/amavis when i move it os .spamassassin_old it recreates it but none in /var/clamav

Also I noticed in /etc a /etc/spam with some files like amavisd.conf what is that there for ?

Thaks

P.S. talking about bugs anybody able to filter the logsi in serveradmin mail portion ?
[/QUOTE]

I was just about to submit that to apple. No, I cannot either. Filters work everywhere else that I have tried but not in mail.
Author

Posts