Using blocklist.de with fail2ban

Anyone who runs a server with open ports knows that systems with questionable intent will come knocking. Sometimes thousands of them. fail2ban is software that that checks your server logs and detects multiple failures, for example 5 failed SSH logins in a row, and bans the source IP address a period of time, e.g. for an hour. This helps prevent password-guessing and brute force attacks. It might be useful to share information about those questionable IP addresses with others so that we can block them proactively.

One such list of IP addresses that I found is blocklist.de. Since I am primarily concerned with systems that are trying to SSH into my system, I looked specifically at their SSH blocklist:
All IP addresses which have been reported within the last 48 hours as having run attacks on the service SSH.

Implementation details:

I found this forum post on the blocklist site that describes how to add the blocklist to fail2ban:
Import Blocklist to fail2ban (PHP Cronjob)

My ability to read German is minimal, but I do speak code. The code provides 3 things:

1. The fail2ban jail, jailing offending IP addresses for an hour (3600 seconds)
2. A PHP script to retrieve the blocklist of IP addresses and ban each one via the jail
3. A cron task to run the script hourly

In general, the PHP script works, but there’s no need for it to be PHP and it could be improved. I wrote a Bash shell script instead that additionally:

1. Verifies the MD5 checksum of the blocklist file
2. Processes only lines in the blocklist file that match a basic IP address regex

I also left out overwriting an empty.log file with a hyphen (“-“):

exec("echo "-" > /var/log/fail2ban.blocklist.log"); //force reload ip-locks

As far as I can tell that does nothing. I’ll come back to the log file later, though.

The SSH blocklist contained 3614 IP addresses when I checked, which means that on the hour, the script will retrieve the list and run the fail2ban-client 3614 times. How long does that process take? I timed the original PHP version, which took over 8 minutes:

$ sudo sh -c "time /usr/bin/php /etc/fail2ban/blocklist.php"

real    8m19.708s
user    5m0.565s
sys     1m31.705s

Calling the fail2ban-client seems like a strange way to go about this. fail2ban is designed to process log files. I had the idea use the IP blocklist to generate a log file, and then have fail2ban read from that log:

while read IP_ADDRESS
do
    echo $(date +'%b %d %T') $HOSTNAME sshd: $IP_ADDRESS >> /var/log/fail2ban.blocklist.log
done < ssh.txt

I specified the new log file in the fail2ban jail:

logpath  =  /var/log/fail2ban.blocklist.log

And restarted fail2ban:

$ sudo systemctl restart fail2ban

However, fail2ban did not block IPs added to that log. My custom log format did not match any of the failregex lines in /etc/fail2ban/filter.d/sshd.conf

I created a custom filter (relying heavily on fail2ban‘s Developing Filters documentation) named ssh-blocklist.conf that contains a single failregex that matches what I’m writing to the custom log:

ssh-blocklist.conf

[INCLUDES]
before = common.conf

[Definition]
_daemon = sshd
failregex = ^%(__prefix_line)s<HOST>$ 

After restarting fail2ban again, fail2ban started processing new items added to the /var/log/fail2ban.blocklist.log. I confirmed that the addresses were added by reviewing the iptables REJECT entries:

$ sudo iptables -L -w -n | grep REJECT

As it turns out, if you add 3614 log entries to the blocklist log, fail2ban doesn’t process all those instantly either. I haven’t timed it, but anecdotally it appears to take about 30 minutes this way, even longer than using fail2ban-client directly.

This brings up a good question: is the effort worth it? That is, what is gained by spending human time (setup/config) and processor time (hourly cron) to block these 3614 IPs? How many of these IPs will actually attempt an SSH connection with the newly-configured server? It would be interesting to log connection attempts from the blocklist IP addresses. This would require updating associated fail2ban action to add an iptables LOG rule, rather than a DROP rule — a project for another day.

Since fail2ban is already running with a sshd jail, offending IPs would be blocked after n failed attempts anyway. Even if a substantial portion of the blocklist IPs attempt connections, they would be blocked quickly. The extra effort seems rather pointless for a single SSH server, but if the same blocklist were applied at a network firewall in front of hundreds or even thousands of hosts, then makes more sense. Adding 3614 block rules to prevent a handful of connection attempts is one thing, adding 3614 block rules to prevent thousands of connection attempts is another.

The files I created are available at fail2ban-blocklist on GitHub.