{"id":2069,"date":"2017-09-24T23:07:23","date_gmt":"2017-09-25T04:07:23","guid":{"rendered":"http:\/\/osric.com\/chris\/accidental-developer\/?p=2069"},"modified":"2017-09-24T23:07:23","modified_gmt":"2017-09-25T04:07:23","slug":"using-blocklist-de-with-fail2ban","status":"publish","type":"post","link":"https:\/\/osric.com\/chris\/accidental-developer\/2017\/09\/using-blocklist-de-with-fail2ban\/","title":{"rendered":"Using blocklist.de with fail2ban"},"content":{"rendered":"

Anyone who runs a server with open ports knows that systems with questionable intent will come knocking. Sometimes thousands of them. fail2ban<\/a> is software that that checks your server logs and detects multiple failures, for example 5 failed SSH logins in a row, and bans the source IP address a period of time, e.g. for an hour. This helps prevent password-guessing and brute force attacks. It might be useful to share information about those questionable IP addresses with others so that we can block them proactively.<\/p>\n

One such list of IP addresses that I found is blocklist.de<\/a>. Since I am primarily concerned with systems that are trying to SSH into my system, I looked specifically at their SSH blocklist<\/a>:
\nAll IP addresses which have been reported within the last 48 hours as having run attacks on the service SSH.<\/em><\/p>\n

Implementation details:<\/strong><\/p>\n

I found this forum post on the blocklist site that describes how to add the blocklist to fail2ban<\/code>:
\nImport Blocklist to fail2ban (PHP Cronjob)<\/a><\/p>\n

My ability to read German is minimal, but I do speak code. The code provides 3 things:<\/p>\n

    \n
  1. The fail2ban<\/code> jail, jailing offending IP addresses for an hour (3600 seconds)<\/li>\n
  2. A PHP script to retrieve the blocklist of IP addresses and ban each one via the jail<\/li>\n
  3. A cron task to run the script hourly<\/li>\n<\/ol>\n

    In general, the PHP script works, but there’s no need for it to be PHP and it could be improved. I wrote a Bash shell script instead that additionally:<\/p>\n

      \n
    1. Verifies the MD5 checksum of the blocklist file<\/li>\n
    2. Processes only lines in the blocklist file that match a basic IP address regex<\/li>\n<\/ol>\n

      I also left out overwriting an empty.log<\/code> file with a hyphen (“-“):<\/p>\n

      exec(\"echo \"-\" > \/var\/log\/fail2ban.blocklist.log\"); \/\/force reload ip-locks<\/code><\/pre>\n
      As far as I can tell that does nothing. I’ll come back to the log file later, though.<\/p>\n
      The SSH blocklist contained 3614 IP addresses when I checked, which means that on the hour, the script will retrieve the list and run the fail2ban-client<\/code> 3614 times. How long does that process take? I timed the original PHP version, which took over 8 minutes:<\/p>\n
      $ sudo sh -c \"time \/usr\/bin\/php \/etc\/fail2ban\/blocklist.php\"\r\n\r\nreal    8m19.708s\r\nuser    5m0.565s\r\nsys     1m31.705s<\/code><\/pre>\n
      Calling the fail2ban-client<\/code> seems like a strange way to go about this. fail2ban<\/code> is designed to process log files. I had the idea use the IP blocklist to generate a log file, and then have fail2ban<\/code> read from that log:<\/p>\n
      while read IP_ADDRESS\r\ndo\r\n    echo $(date +'%b %d %T') $HOSTNAME sshd: $IP_ADDRESS >> \/var\/log\/fail2ban.blocklist.log\r\ndone < ssh.txt<\/code><\/pre>\n
      I specified the new log file in the fail2ban<\/code> jail:<\/p>\n
      logpath  =  \/var\/log\/fail2ban.blocklist.log<\/code><\/pre>\n
      And restarted fail2ban<\/code>:<\/p>\n
      $ sudo systemctl restart fail2ban<\/code><\/pre>\n
      However, fail2ban<\/code> did not block IPs added to that log. My custom log format did not match any of the failregex<\/code> lines in \/etc\/fail2ban\/filter.d\/sshd.conf<\/code><\/p>\n
      I created a custom filter (relying heavily on fail2ban<\/code>‘s Developing Filters<\/a> documentation) named ssh-blocklist.conf<\/code> that contains a single failregex that matches what I’m writing to the custom log:<\/p>\n
      ssh-blocklist.conf<\/strong><\/p>\n
      [INCLUDES]\r\nbefore = common.conf\r\n\r\n[Definition]\r\n_daemon = sshd\r\nfailregex = ^%(__prefix_line)s<HOST>$ <\/code><\/pre>\n
      After restarting fail2ban<\/code> again, fail2ban<\/code> started processing new items added to the \/var\/log\/fail2ban.blocklist.log<\/code>. I confirmed that the addresses were added by reviewing the iptables REJECT<\/code> entries:<\/p>\n
      $ sudo iptables -L -w -n | grep REJECT<\/code><\/pre>\n
      As it turns out, if you add 3614 log entries to the blocklist log, fail2ban<\/code> doesn’t process all those instantly either. I haven’t timed it, but anecdotally it appears to take about 30 minutes this way, even longer than using fail2ban-client<\/code> directly.<\/p>\n
      This brings up a good question: is the effort worth it? That is, what is gained by spending human time (setup\/config) and processor time (hourly cron) to block these 3614 IPs? How many of these IPs will actually attempt an SSH connection with the newly-configured server? It would be interesting to log connection attempts from the blocklist IP addresses. This would require updating associated fail2ban action to add an iptables LOG<\/code> rule, rather than a DROP<\/code> rule — a project for another day.<\/p>\n
      Since fail2ban<\/code> is already running with a sshd jail, offending IPs would be blocked after n<\/em> failed attempts anyway. Even if a substantial portion of the blocklist IPs attempt connections, they would be blocked quickly. The extra effort seems rather pointless for a single SSH server, but if the same blocklist were applied at a network firewall in front of hundreds or even thousands of hosts, then makes more sense. Adding 3614 block rules to prevent a handful of connection attempts is one thing, adding 3614 block rules to prevent thousands of connection attempts is another.<\/p>\n
      The files I created are available at fail2ban-blocklist<\/a> on GitHub.<\/p>\n","protected":false},"excerpt":{"rendered":"
      Methods of incorporating a public blocklist of IP address from blocklist.de into a fail2ban framework.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[48],"tags":[452,408],"class_list":["post-2069","post","type-post","status-publish","format-standard","hentry","category-security","tag-fail2ban","tag-iptables"],"_links":{"self":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2069","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/comments?post=2069"}],"version-history":[{"count":17,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2069\/revisions"}],"predecessor-version":[{"id":2105,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2069\/revisions\/2105"}],"wp:attachment":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/media?parent=2069"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/categories?post=2069"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/tags?post=2069"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}