Tag: linux

fail2ban fails to ban SSH login failures

fail2ban is one of those magical programs that, in my experience, just works. I’ve inherited many systems with a working fail2ban configuration, and therefore I didn’t know much about configuring it or troubleshooting it.

Summary: by default, fail2ban on CentOS 7 does absolutely nothing!

One of the things that it is reported (falsely!) to do out-of-the-box is to block repeated SSH login failures. According to Protecting SSH with Fail2ban:

Fail2ban should now protect SSH out of the box. If Fail2ban notices six failed login attempts in the last ten minutes, then it blocks that IP for ten minutes.

I wanted to test this, so I set up 2 virtual machines, a victim and an attacker.

On the victim VM:
[ariel]# sudo yum install epel-release
[ariel]# sudo yum install fail2ban
[ariel]# sudo systemctl start fail2ban
[ariel]# sudo tail -f /var/log/fail2ban

On the attacker VM:
[caliban]# sudo yum install epel-release
[caliban]# sudo yum install sshpass
[caliban]# for i in `seq 1 100`; do sshpass -p 'TopSecret!' admin@ariel; done

And then I waited. And waited. And waited.

I confirmed that the defaults described matched what was in my /etc/fail2ban/jails.conf (excerpted):
bantime = 600
findtime = 600
maxretry = 5

In my test, I definitely exceeded that: about 30 failed attempts in 5 minutes. The failures appear in /var/log/secure, but nothing appears in /var/log/fail2ban.log!

From How To Protect SSH With Fail2Ban on CentOS 7 I found the fail2ban-client status command:

[ariel]# fail2ban-client status
Status
|- Number of jail: 0
`- Jail list:

Zero jails! That’s definitely a problem.

As mentioned in the above, I created a file, /etc/fail2ban/jail.local containing the following:
[sshd]
enabled = true

New results:
[ariel]# systemctl restart fail2ban
[ariel]# fail2ban-client status
Status
|- Number of jail: 1
`- Jail list: sshd

That looks better! /var/log/fail2ban.log now has new entries, and the attacker IP address has been banned! Just to confirm I tried to SSH to the machine from the attacker:

[caliban]# ssh admin@ariel
ssh_exchange_identification: Connection closed by remote host

Great! Exactly what I expected to happen.

When I look at the /etc/fail2ban/jails.conf, I do not see enabled = true under the [sshd] section. In fact, part of that file explains that all jails are disabled by default:

# "enabled" enables the jails.
# By default all jails are disabled, and it should stay this way.
# Enable only relevant to your setup jails in your .local or jail.d/*.conf
#
# true: jail will be enabled and log files will get monitored for changes
# false: jail is not enabled
enabled = false

On CentOS 7, fail2ban is configured to work with firewalld. My next post describes using fail2ban with iptables on CentOS 7.

Block an IP address via iptables

I was monitoring the mail logs on a Postfix server and noted repeated failed connection attempts from the same IP address. The source was likely up to no good, and it was making it more difficult to monitor the logs for legitimate connections, so I decided to block it:

iptables -A INPUT -s 123.456.789.101 -j DROP

(IP address changed to protect…the innocent?)

However, the IP address was still making connections:
Dec 2 17:19:05 mercutio postfix/smtpd[15230]: connect from unknown[123.456.789.101]
Dec 2 17:19:06 mercutio postfix/smtpd[15230]: lost connection after AUTH from unknown[123.456.789.101]
Dec 2 17:19:06 mercutio postfix/smtpd[15230]: disconnect from unknown[123.456.789.101]

How is that possible? First I checked iptables to check my sanity and confirm that the rule had been added:

# iptables -L
...
DROP all -- 123.456.789.101 anywhere
...

OK, it’s there. That’s good!

The problem in this case was a different rule that had been added previously. Rules in iptables are processed in order, and no further rules are processed after a matching rule is found. Well above my newly-added rule was this rule:
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:smtp

That rule makes sense for a mail server, but I needed my rule to be inserted before it. I determined which rule it was in the INPUT chain like this:
iptables --line-numbers -L INPUT

It was the 5th rule, so I was able to insert the new rule just above it like this:
iptables -I INPUT 4 -s 123.456.789.101 -j DROP

After that, the offending IP address stopped creating entries in the mail.log.

However, my new rule would disappear after a system restart. Since I am using iptables-persistent, I saved the rules to the config file:
iptables-save > /etc/iptables/rules.v4

To confirm everything worked, I attempted to restart iptables:
# service iptables-persistent restart
Failed to restart iptables-persistent.service: Unit iptables-persistent.service

Apparently the service name changed to netfilter-persistent in Debian 8. The config files are still in the same location, but the service name has changed.

I restarted iptables:
# service netfilter-persistent restart

I checked the rules again and my new rule was there, above the rule allowing connections from any IP on port 25. However, I also noticed the following rule above either of those:
ACCEPT all -- anywhere anywhere

I freaked out. That rule indicates that all traffic from any source on any port should be accepted. That’s the worst firewall rule I’ve ever seen. It basically negates the entire concept of a firewall. It clearly should not be there!

However, using the verbose switch on iptables:
iptables -vL INPUT

I discovered that the rule only applied to the lo interface (loopback). That’s a relief–that rule gets to stay.

iptables and deleting/replacing entries

Whenever I have to reboot my modem [sic] at home, I typically get a new IP address from my ISP.

When that happens, I need to update iptables to allow my new address to connect to the SSH port (port 22) of my jump box (which, fortunately, I have access to from another IP address):

iptables -A INPUT -p tcp -m state --state NEW -s [new IP address] --dport 22 -j ACCEPT

But I don’t want to leave the old entry. How to get rid of it?

The delete (-D) and replace (-R) options require a line number from the chain (e.g. the INPUT chain). To find the line numbers:

iptables -L INPUT --line-numbers

To delete the existing rule and add the new rule:

iptables -D INPUT [line number]
iptables -A INPUT -p tcp -m state --state NEW --dport 22 -s [new IP address] -j ACCEPT

To replace the existing entry:

iptables -R INPUT [line number] -p tcp -m state --state NEW --dport 22 -s [new IP address] -j ACCEPT

Save the updates so they are persistent:

iptables-save > /etc/iptables/rules.v4

(That’s the location for Debian and Ubuntu. This may be different for your distribution.)

Removing exceptions from a list using Bash (with sed and awk)

  • I have a CSV file, a list of 1000+ users and user properties.
  • I have a list of exceptions (users to be excluded from processing), one user per line, about 50 total.

How can I remove the exceptions from the list?

# make a copy of the original list
cp list-of-1000.csv list-of-1000-less-exceptions.csv
# loop through each line in exceptions.txt and remove matching lines from the copy
while read line; do sed -i "/${line}/d" list-of-1000-less-exceptions.csv; done < exceptions.txt

This is a little simplistic and could be a problem if any usernames are subsets of other usernames. (For example, if user ‘bob’ is on the list of exceptions, but the list of users also contains ‘bobb’, both would be deleted.)

In the particular instance I am dealing with, the username is conveniently the first field in the CSV file. This allows me to match the start of the line and the comma following the username:

while read line; do sed -i "/^${line},/d" list-of-1000-less-exceptions.csv; done < exceptions.txt

What if the username was the third field in the CSV instead of the first?

Use awk:
awk -F, -vOFS=, '{print $3,$0}' list-of-exceptions.csv > copy-of-list-of-exceptions.csv

  • -F, sets the field separator to a comma (defaults to whitespace)
  • -vOFS=, sets the Output Field Separator (OFS) to a comma (defaults to a space)
  • $3 prints the third field
  • $0 prints all the fields, with the specified field separator between them

while read line; do sed -i "/^${line},/d" copy-of-1000-less-exceptions.csv; done < exceptions.txt

Now there’s still an extra username in this file. Maybe that doesn’t matter, but maybe it does. There are several ways to remove it–here’s one:

awk -F, -vOFS=, '$1=""; print $0' copy-of-1000-less-exceptions.csv | sed 's/^,//' > list-of-1000-less-exceptions.csv

  • -F, sets the field separator to a comma (defaults to whitespace)
  • -vOFS=, sets the Output Field Separator (OFS) to a comma (defaults to a space)
  • $1="" sets the first field to an empty string
  • print $0 prints all the fields

The result of the awk command has an initial comma on each line. The first field is still there, it’s just set to an empty string. I used sed to remove it.

You could also use sed alone to remove the extra username field:
sed -i 's/^[^,]*,//' copy-of-1000-less-exceptions.csv

WordPress Manual Update Instructions

One of the great things about WordPress is the one-click upgrade procedure. It’s particularly convenient, because WordPress has frequent upgrades and security updates. Without an easy way of upgrading, many users would complain of upgrade fatigue, or would continue running older versions with security flaws.

Of course, not everyone can use the one click upgrade: My host is not configured properly, so I need to upgrade manually. Fortunately, a manual upgrade is relatively painless. Although it is described in some detail at Upgrading WordPress: Manual Update, I’m listing my specific procedures (using the Bash shell) here.

  1. Download the latest version to your home directory:
    wget http://wordpress.org/latest.tar.gz
  2. Remove any old WordPress directory before unpacking the latest:
    rm -r ./wordpress
  3. Unpack the latest version:
    tar -xf wordpress-3.2.tar.gz
  4. Backup the database:
    mysqldump --add-drop-table -h localhost -u [username] -p [database name] | bzip2 -c > [site name].[dd-MMM-yyyy].bak.sql.bz2
  5. Disable plugins. You can do this via the admin interface, or the database:
    UPDATE wp_options SET option_value = 'a:0:{}' WHERE option_name = 'active_plugins';
  6. Remove the wp-admin and wp-includes directories:
    rm -r ./path/to/your/wordpress/wp-admin ./path/to/your/wordpress/wp-includes
  7. Copy only the updated files over to the WordPress install:
    cp -ru ./wordpress/* ./path/to/your/wordpress/
  8. Visit the admin page (which may prompt a database upgrade).
  9. Re-enable any plugins.

Step 6 may seem unnecessary in light of step 7, but in my experience merely updating the wp-admin and wp-includes directories is not enough: there may be old files that are not present in the latest version, but that will cause problems if they still exist.

I’ve found that my reCAPTCHA plugin doesn’t retain its API keys, but you can look them up again on the reCAPTCHA site. Other plugins may have similar issues.

Thanks to Perishable Press for the tip on disabling WordPress plugins via MySQL.