Adaptive Firewall Rules with afctl

[l008com]

Someone finally turned me on to the afctl program and I was thrilled. I quickly made a script on my site that will automatically block abusers for an hour. Once I turned this script on, my server’s cpu dropped to near idle while it continued to pump out tens of thousands of legit page views per day. I was very happy.

BUT now I have a big problem. afctl generated temporary firewall rules start at 01700 and increment by 5 with every new rule. Well after about 4 days of running this script, my rule numbers have gotten up to and past number 12300, which is the range of a bunch of the default firewall rules. afctl is now making rules with higher numbers than this (meaning they come after them) and the afctl rules now don’t DO anything.

Server Admin doesn’t even let you delete these afctl created rules. Is there some way I can manage these rules? Ideally, the best way to have it run is to have it pick the lowest available rule number higher than 01700. That way I would have to have over 10,000 rules at once for me to have any conflicts. Normally I only have 30-50 rules at once, maybe a few more during heavy traffic.

Does anybody have any insight on this matter?

[khiltd]

Looks like these are all the configuration options you get with that tool:

http://developer.apple.com/documentation/Darwin/Reference/ManPages/man5/af.plist.5.html#//apple_ref/doc/man/5/af.plist

[l008com]

I’ve seen that before. The only thing there that looks like it could possibly be helpful is the default_set. I say possibly because I’m not sure what it means by rule set. Can I somehow put all my afctl rules into group B, and have that whole group run in a certain order within my main list of firewall rules?

[l008com]

The rules are expiring when they should. The problem is that all new rules are incremented by 5. So even though the rules started at 1700, right now i have about 200 active dynamic rules that are numbered from 9570 to 9825. Every new rule is +5, even though there are no more rules between 1700 and 9570. Once I get up to rule 12300, my dynamic rules start to come after the included default firewall rules.

Here this will make it easy to see:
http://img71.imageshack.us/img71/7927/picture1kc3.png

So after about 3 days of running this script, new rules get to 12300 and above. Then the dynamic rules stop working.
I need some way to keep all of afctl’s between 1700 and 12300. There aren’t that many rules at any one time. They all expire after 60 minutes.

And they are all bots. Other people’s scripts that are set up to load my home page repeatedly. Repeatedly can mean once every 5 minutes, or it could mean 10 times per second. Lots of different ways but in the end this script works amazingly well, right up until the rules get too high.

[l008com]

SO

If I disable my script, or in other words if I don’t run afctl for about two hours… long enough for all rules to expire, and then some…

Then if I turn the script back on, it will start again 1700.
So how can I reset the rule numbers, without stopping using afctl?

[l008com]

Well I found a solution, but it’s not great. I run the following commands daily (nightly).

sudo rm /var/db/af/blacklist;
sudo ipfw delete set 17;
sudo /usr/libexec/afctl;

This deletes any memory afctl has of it’s rules. Then it manually deletes all the rules it’s made. Then it recreates it’s database file.

This will make your rules start over every night so you won’t get ‘rule number overflow’ headaches.

OF COURSE the whole point of afctl is auto-expiring firewall rules. So if you’re going to do this, I might as well have my server firewall addresses directly to ipfw instead of bothering with afctl. I’m going to leave it using afctl now only because its already set up and running. At least I can be away from my server now without having a rule number overflow which for several different reasons brings my server to it’s knees.

[bguillory]

l008com, thanks for the insight into what was happening. I initially found this problem because afctl was removing my production firewall rules. I got entries in my /var/log/system.log like this:
Nov 18 01:33:13 mac com.apple.afctl[32862]: ip6fw: rule 12554: setsockopt(IPV6_FW_DEL): Invalid argument

Here is the script that I came up with to work around the problem:

[code]
#!/bin/sh

# Brad Guillory 18 November 2009
# Initial Version, detects current afctl rule number and resets afctl black
# list if a threshold is breached.

#By default on Mac OS X Server 10.5 emond(8) is configured to detect repeated
#auth failures (likely a brute force attack). When it detects an attack
#emond calls afctl to temporarily blacklist (and hopefully block) that IP
#
#This works great unless emond detects enough attacks that afctl is always
#blocking at least one IP. Because afctl in turn uses ipfw rules to block
#the blacklisted IPs afctl needs to use firewall rule numbers to do accounting
#with ipfw. Internally ipfw rules are applied in order; afctl rules start out
#at a relatively low number (by default 1700) and “regular” rules start at
#12300. If afctl’s rule number grow so they collide with the “regular” rules
#afctl blacklist addresses are no longer blocked and (perhaps worse) when
#afctl goes to remove its rules it may inadvertently remove a “regular” rule
#causing a denial of service.
#There is lots of room for rules between 1700 and 12300 but afctl rule numbers
#increment by 5 and each blacklist address consumes multiple consecutive rules
#
#To address this problem I have written this script to detect when the afctl
#current rule number has grown too large. When it has it clears the afctl
#blacklist and reset afctl. (Thank you to [l008com] he posted the method
#here: https://www.afp548.com/forum/viewtopic.php?showtopic=21220)
#

#set -x

AFCTL_SET=17
THRESHOLD=10000
#NOTE: A THRESHOLD of 10000 allows for 460 additional rules before we get
#into trouble. Because afctl is making multiple rules for each IP address
#this gets cut down to less than 50 detected incidences.
#So we should probably run AT LEAST every 30 minutes. To be safe I suggest
#running every 5 minutes.

CURRENTRULE=`sudo ipfw -ST list | fgrep ” set $AFCTL_SET ” | cut -d’ ‘ -f 1 | sort -n | tail -1`

if [ 0″$CURRENTRULE” -gt “$THRESHOLD” ]; then
echo “Highest afctl rule is $CURRENTRULE, resetting afctl…”
sudo rm /var/db/af/blacklist;
sudo ipfw delete set $AFCTL_SET;
sudo /usr/libexec/afctl;
fi
[/code]

[jgardner]

Here’s a version that resets the rule numbering, but keeps the existing blacklisted hosts and TTLs intact.

[code]
#!/bin/sh
BF=/var/db/af/blacklist
MAX=`/usr/bin/cut -f3 $BF | /usr/bin/sort -n | /usr/bin/tail -1`
if [ $MAX -gt 12000 ]
then
/bin/mv $BF $BF.old
/sbin/ipfw delete set 17
/usr/libexec/afctl
CT=`/bin/date +%s`
for IP in `/usr/bin/cut -f1 $BF.old | /usr/bin/grep -v “#”`
do
EXP=`/usr/bin/grep $IP $BF.old | /usr/bin/cut -f2 | /usr/bin/cut -f1 -d’.’`
TTL=`/bin/expr $EXP / 60 – $CT / 60`
/usr/libexec/afctl -a $IP -t $TTL
done
fi
[/code]

[Author]

Posts