Recently, one of my clients’ dev servers was made publicly visible for running some tests, and almost immediately we observed fake users1 registering on the system, somehow cracking the image CAPTCHA challenges present on all anonymously accessible forms on the site.
That the attacks were automated was evident from the rapid rate of unceasing incoming requests. We figured the attackers might be using one of several inexpensive online CAPTCHA solving services, many of which use a pool of actual humans in the backend to process the images. Faced with such a MO, it becomes near pointless to increase the CAPTCHA complexity or switch to some other CAPTCHA method, since what thwarts a professional CAPTCHA solver (who has presumably accrued a good deal of skill from solving thousands of them) is more than likely to confound a legitimate end-user as well.
Our next tack was to trace the origin of the IPs from which the attacks were originating. As it turned out, most attacks were coming from just a handful of addresses, which we traced back to certain locations in China. Of the remainder, a few could be traced back to regions in the EU. Great, so we had something resembling a small, but well-spread botnet to deal with!
What works in favor of sysadmins trying to fend off these pesky bloodsuckers is that spammers like to haul their manure by the truckload, since their operation turns a profit only when spam delivered at INCREDIBLY high volumes. Hence, if you’re under attack from a spammer’s botnet, chances are you’re not the only one, and that there have been several reported incidents citing the same sources previously. These incident reports (both manual and automated) are collected by several anti-spam organizations online, which lets them prepare blacklists of well known sources.
We searched a few online databases which publish suspicious IPs, and found the list at Stop Forum Spam to be one of the most comprehensive. In fact, it included every single IP we had identified as a threat, based on our server access logs. One of their published resources is a single, massive list2 of banned IPs, which is updated daily. They also publish incremental addendums to this list, updated hourly, making their data perfectly suited for building and updating our own blacklist.
Most linux systems use iptables as the default interface to netfilter (packet filtering framework inside linux 2.4+ kernels), which works well enough for managing a handful of serial trapdoor rules. In this particular situation, however, the standard one-rule-per-ip configuration scheme would fail miserably, since we have a HUGE list of IPs to block3. With the usual configuration method, each incoming packet would have to fall through a chain of over a quarter million rules before it could be allowed to pass through. This would render the server completely unusable4. This is where ipset comes to the rescue. To quote a veteran on the Gentoo forums:
It (ipset) lets you create huge lists of ip addresses and/or ports (with tens of thousands of entries or more) which are stored in a tiny piece of ram with extreme efficiency. In your iptables rules, you can then simply refer to the lists by name, and the entire list is checked with remarkable speed and in a single netfilter rule. Also, you can change the contents of the list while the firewall is running.
ipset needs to be enabled at the kernel level (as a module or built-in), and most sane kernels have it enabled already. If your distribution provides the ipset package from within its official repositories, then its default kernel should already have ipset enabled. If not, you’ll need to recompile your kernel with ipset enabled, and then build and install ipset userland tools from source. See the ipset installation page for more details.
Once installed, our firewall setup boils down to 2 main steps:
- An initial import of the latest complete list of banned IPs from Stop Forum Spam, containing more than 380,000 (at the time of writing) entries, and setting up firewall rules to filter incoming packets through this list.
- A cron job that periodically updates this list with addtional entries published in a separate incremental list. Entries are updated while the firewall is live and running. The new entries are automatically included while filtering packets thereon.
In this stage we download a complete list and import it into a blacklist managed by ipset.
Note: The last two commands would need to be run as root.
$ wget http://www.stopforumspam.com/downloads/bannedips.zip
$ unzip bannedips.zip
$ ipset create blacklist hash:ip hashsize 4096 maxelem 1048576
$ sed s/,/\\n/g ./bannedips.csv |while read i; do ipset add blacklist $i; done #This will take a while.
Now that the blacklist is ready, we need to include this in our iptables chains:
# /etc/sysconfig/iptables (on RHEL-like systems)
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [210:32206]
:TCP - [0:0]
:UDP - [0:0]
-A INPUT -p tcp -m set --match-set blacklist src -m tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -p tcp -m set --match-set blacklist src -m tcp --dport 443 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -p tcp -m set --match-set blacklist src -m tcp --dport 22 -j REJECT --reject-with icmp-port-unreachable
If you have read and implemented my previous post on securing a linux server, or otherwise already have some rules present in your
iptables config file, you should take care to insert the new rules at the right location for them to take effect. Putting them right at the beginning would work in all cases, but wouldn’t be very efficient since these filters would be processed for EVERY incoming packet, even if it is from a trusted source or from an already established connection. As an example, here’s what the configuration on our sytems looks like:
Finally we need a script to load the ipset list at boot, just before the firewall is brought online, and to save the list to disk during shutdown. Such a script is already available for RHEL-like systems and is installed using the following steps5:
$ cd /etc/rc.d/init.d/
$ wget http://toad-one.org/ipset/ipset
$ chmod 755 ipset
$ chkconfig --add ipset
$ cd /etc/sysconfig
$ wget http://toad-one.org/ipset/ipset-config
This script will save all ipset lists to
/etc/sysconfig/ipset during shutdown, and restore them prior to firewall initialization during boot.
Note: This list, though enormous, takes only about a second to load, as opposed to the lengthy processing time taken during the initial import.
The final stuff to setup is a cron job that updates the main blacklist with an incremental set of new IPs, updated hourly on the Stop Forum Spam site. This list actually contains IPs collected over the last 24 hours, so it is usually sufficient to run the update once or twice daily. We have included the following script in
/etc/cron.daily. You may choose a higher frequency, but you need to update at least once daily if you wish to use their hourly list.
curl -O http://www.stopforumspam.com/downloads/listed_ip_1.zip
curl -O http://www.stopforumspam.com/downloads/listed_ip_1.md5
echo ' listed_ip_1.zip' >> ./listed_ip_1.md5
md5sum -c ./listed_ip_1.md5
if [ $? -eq 0 ];
unzip -o listed_ip_1.zip
while read i; do ipset -quiet add blacklist $i; done < ./listed_ip_1.txt
echo "Failed download. Checksum does not match."
rm -f ./listed_ip_1.*
That’s it. Say goodbye to spambots (mostly).
- Determined to be fake by their profile content.
- Containing nearly 400,000 entries.
- Stop Forum Spam also publishes a list of networks in CIDR format which, though more compact, cannot cover all the individual banned IPs. This list is more useful for banning entire networks identified as highly toxic. The list of individual IPs we have used is a superset of the IPs represented in this list.
- See here for some benchmarks with just a few thousand rules.
- See this thread for a Gentoo initscript.