Linux policy based routing

Problem: I have a host that has 2 active network interfaces. One is used as a management port (eth0), one is used as an FTP dropbox (eth1).

Both can route to the Internet, but all connections other than FTP on eth1 are blocked via iptables. The default route uses the interface for the FTP dropbox, but I have a static route configured for the subnet that includes my management and monitoring hosts so that I can SSH to the host and check on host availability, disk space, mail queue, etc.

However, the static route means that I cannot monitor the FTP dropbox, since FTP connection attempts coming in on one interface and IP address are then routed out via the management interface and IP address.

Solution: Use policy-based routing to direct the system to consult a different routing table for connections coming in on the FTP interface.

It sounds easy enough.
Continue reading Linux policy based routing

Docker versus Podman and iptables

I have recently been learning about podman, a tool for running containers that has a command syntax that matches Docker, but that does not require a Docker daemon and which does not require root privileges.

I ran into some unexpected problems publishing ports with Podman, which had to do with my default DROP policy on the iptables FORWARD chain. Below I will demonstrate some of the differences between Docker and Podman in terms of iptables changes, and provide a workaround for Podman.
Continue reading Docker versus Podman and iptables

Re-bind host to FreeIPA

The sudo command on one particular FreeIPA-bound host was taking an exceedingly long time to run. And when it finally ran, it would not accept my current password, but rather my previous password — somehow still cached on the system. It was a strange problem.

Instead of trying to figure out exactly why it was happening, I decided to remove & re-bind the host to my FreeIPA domain.
Continue reading Re-bind host to FreeIPA

Using blocklist.de with fail2ban

Anyone who runs a server with open ports knows that systems with questionable intent will come knocking. Sometimes thousands of them. fail2ban is software that that checks your server logs and detects multiple failures, for example 5 failed SSH logins in a row, and bans the source IP address a period of time, e.g. for an hour. This helps prevent password-guessing and brute force attacks. It might be useful to share information about those questionable IP addresses with others so that we can block them proactively.

One such list of IP addresses that I found is blocklist.de. Since I am primarily concerned with systems that are trying to SSH into my system, I looked specifically at their SSH blocklist:
All IP addresses which have been reported within the last 48 hours as having run attacks on the service SSH.

Implementation details: Continue reading Using blocklist.de with fail2ban

Using fail2ban with iptables instead of firewalld

In the previous post I wrote about the minor configuration changes needed to get fail2ban to actually do something.

I have been working primarily with CentOS 7 and have been using iptables instead of firewalld. Normally, fail2ban works with iptables by default. However, installing fail2ban on CentOS 7 also installs fail2ban-firewalld — which changes that default. Even with a properly configured fail2ban jail, you will not see the expected results. fail2ban will log events as expected, but no traffic will actually be banned.

The fail2ban-firewalld package places a file in /etc/fail2ban/jail.d/00-firewalld.conf. It overrides the default banaction (iptables) and sets it to firewallcmd-ipset.

The top of the 00-firewalld.conf file says:

You can remove this package (along with the empty fail2ban meta-package) if you do not use firewalld

When I tried removing fail2ban-firewalld, it removed fail2ban as a dependency. I have a feeling the referenced fail2ban meta-package may have something to so with that.

I have not yet investigated the meta-package and de-coupling fail2ban-firewalld from fail2ban (see Update below). My solution, for now, has been to move 00-firewalld.conf and restart fail2ban:

$ sudo mv /etc/fail2ban/jail.d/00-firewalld.conf /etc/fail2ban/jail.d/00-firewalld.disabled
$ sudo systemctl restart fail2ban

The default banaction defined in jail.conf is no longer overridden and performs as expected:
banaction = iptables-multiport

Update
According to Fail2ban with FirewallD, The fail2ban package itself is a meta-package that contains several other packages, including fail2ban-firewalld and fail2ban-server. Removing the meta-package will not remove fail2ban-server.

If you’ve already moved 00-firewalld.conf to 00-firewalld.disabled, you’ll get a warning:
warning: file /etc/fail2ban/jail.d/00-firewalld.conf: remove failed: No such file or directory

You can ignore the warning, or remove 00-firewalld.disabled.

Block an IP address via iptables

I was monitoring the mail logs on a Postfix server and noted repeated failed connection attempts from the same IP address. The source was likely up to no good, and it was making it more difficult to monitor the logs for legitimate connections, so I decided to block it:

iptables -A INPUT -s 123.456.789.101 -j DROP

(IP address changed to protect…the innocent?)

However, the IP address was still making connections:
Dec 2 17:19:05 mercutio postfix/smtpd[15230]: connect from unknown[123.456.789.101]
Dec 2 17:19:06 mercutio postfix/smtpd[15230]: lost connection after AUTH from unknown[123.456.789.101]
Dec 2 17:19:06 mercutio postfix/smtpd[15230]: disconnect from unknown[123.456.789.101]

How is that possible? First I checked iptables to check my sanity and confirm that the rule had been added:

# iptables -L
...
DROP all -- 123.456.789.101 anywhere
...

OK, it’s there. That’s good!

The problem in this case was a different rule that had been added previously. Rules in iptables are processed in order, and no further rules are processed after a matching rule is found. Well above my newly-added rule was this rule:
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:smtp

That rule makes sense for a mail server, but I needed my rule to be inserted before it. I determined which rule it was in the INPUT chain like this:
iptables --line-numbers -L INPUT

It was the 5th rule, so I was able to insert the new rule just above it like this:
iptables -I INPUT 4 -s 123.456.789.101 -j DROP

After that, the offending IP address stopped creating entries in the mail.log.

However, my new rule would disappear after a system restart. Since I am using iptables-persistent, I saved the rules to the config file:
iptables-save > /etc/iptables/rules.v4

To confirm everything worked, I attempted to restart iptables:
# service iptables-persistent restart
Failed to restart iptables-persistent.service: Unit iptables-persistent.service

Apparently the service name changed to netfilter-persistent in Debian 8. The config files are still in the same location, but the service name has changed.

I restarted iptables:
# service netfilter-persistent restart

I checked the rules again and my new rule was there, above the rule allowing connections from any IP on port 25. However, I also noticed the following rule above either of those:
ACCEPT all -- anywhere anywhere

I freaked out. That rule indicates that all traffic from any source on any port should be accepted. That’s the worst firewall rule I’ve ever seen. It basically negates the entire concept of a firewall. It clearly should not be there!

However, using the verbose switch on iptables:
iptables -vL INPUT

I discovered that the rule only applied to the lo interface (loopback). That’s a relief–that rule gets to stay.

iptables and deleting/replacing entries

Whenever I have to reboot my modem [sic] at home, I typically get a new IP address from my ISP.

When that happens, I need to update iptables to allow my new address to connect to the SSH port (port 22) of my jump box (which, fortunately, I have access to from another IP address):

iptables -A INPUT -p tcp -m state --state NEW -s [new IP address] --dport 22 -j ACCEPT

But I don’t want to leave the old entry. How to get rid of it?

The delete (-D) and replace (-R) options require a line number from the chain (e.g. the INPUT chain). To find the line numbers:

iptables -L INPUT --line-numbers

To delete the existing rule and add the new rule:

iptables -D INPUT [line number]
iptables -A INPUT -p tcp -m state --state NEW --dport 22 -s [new IP address] -j ACCEPT

To replace the existing entry:

iptables -R INPUT [line number] -p tcp -m state --state NEW --dport 22 -s [new IP address] -j ACCEPT

Save the updates so they are persistent:

iptables-save > /etc/iptables/rules.v4

(That’s the location for Debian and Ubuntu. This may be different for your distribution.)