Using nftables to allow ingress traffic from internet to intranet resources

I recently heard about a case where someone wanted to allow an AWS EC2 instance on a public IP address to access a corporate intranet resource. Their solution was to run an HAProxy server on the corporate network that could proxy traffic from the AWS compute instance to the intranet server.

Whether or not accessing intranet resources from an external host is good idea or violates corporate policy I will leave aside for now, but I thought of 6 additional ways to accomplish the same objective.

What follow is one of those 6 ideas: using Network Address Translation (NAT) on the intermediary host via nftables. I have spent far more time with iptables than I have with nftables. But this seemed like a good opportunity to experiment.

The 3 servers for this example:

  1. The corporate intranet resource, intranet.osric.net or 192.0.2.43
  2. The NAT host, nat.osric.net or 198.51.100.200
  3. The AWS EC2 instance, ec2-203-0-113-155.compute-1.amazonaws.com or 203.0.113.155

The nat.osric.net server is running RHEL 10, although the commands should work on other systems using nftables.

I followed the steps provided by Configuring destination NAT using nftables on nat.osric.net:

sudo nft add table nat
sudo nft -- add chain nat prerouting { type nat hook prerouting priority -100 \; }
sudo nft add chain nat postrouting { type nat hook postrouting priority 100 \; }

The next commands needed the interface name, which I found using nmcli device status (ens5 in this case).

sudo nft add rule nat prerouting iifname ens5 tcp dport { 80, 443 } dnat to 192.0.2.43
sudo nft add rule nat postrouting oifname "ens5" snat to 198.51.100.200
echo "net.ipv4.ip_forward=1" | sudo tee /etc/sysctl.d/95-IPv4-forwarding.conf
sudo sysctl -p /etc/sysctl.d/95-IPv4-forwarding.conf

It worked! Well, with a couple caveats: an issue with the TLS certificate, and a wide-open NAT configuration.

The TLS certificate

From the EC2 instance:

$ curl https://198.51.100.200
curl: (60) SSL: no alternative certificate subject name matches target host name '198.51.100.200'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I could think of a few ways around this:

  • Add an entry to /etc/hosts on the client, i.e. on ec2-203-0-113-155.compute-1.amazonaws.com
  • Ignore the warning, e.g. curl --insecure https://198.51.100.200/
  • Add a subject alternative name (e.g. nat.osric.net) to the TLS cert for intranet.osric.net

The last method, adding a Subject Alternative Name (SAN) sounded like a pain, and also may not be feasible if you don’t manage the TLS certificate for that host. And I never like ignoring security warnings. But editing the /etc/hosts file was a simple solution that worked. Here’s the line I added to the EC2 instance (ec2-203-0-113-155.compute-1.amazonaws.com):

198.51.100.200   intranet.osric.net

After that, I was able to use curl https://intranet.osric.net/ to return content without errors or warnings.

The wide-open NAT

The configuration, as implemented, could allow anyone on the Internet to bypass the network security controls around intranet.osric.net. All they have to do is visit nat.osric.net (and maybe ignore some TLS warnings). This is terrible! Fortunately, we can use nftables to limit NAT to only specific inbound traffic.

How can I review what has already been added to nftables? Fortunately, Quick reference-nftables in 10 minutes was helpful here.

$ sudo nft list tables
table ip nat
$ sudo nft list chains
table ip nat {
        chain prerouting {
                type nat hook prerouting priority dstnat; policy accept;
        }
        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
        }
}
$ sudo nft list ruleset
table ip nat {
        chain prerouting {
                type nat hook prerouting priority dstnat; policy accept;
                iifname "ens5" tcp dport { 80, 443 } dnat to 192.0.2.43
        }

        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                oifname "ens5" snat to 198.51.100.200
        }
}

In order to delete the dnat rule, I needed its handle. To get this, I used the -a flag:

$ sudo nft -a list ruleset
table ip nat { # handle 1
        chain prerouting { # handle 1
                type nat hook prerouting priority dstnat; policy accept;
                iifname "ens5" tcp dport { 80, 443 } dnat to 192.0.2.43 # handle 8
        }

        chain postrouting { # handle 2
                type nat hook postrouting priority srcnat; policy accept;
                oifname "ens5" snat to 198.51.100.200 # handle 9
        }
}

I deleted handle 8:

sudo nft delete rule nat prerouting "handle 8"

And added a new prerouting rule:

sudo nft add rule nat prerouting iifname ens5 ip saddr 203.0.113.155 tcp dport { 80, 443 } dnat to 192.0.2.43

That provides NAT for only the EC2 instance at 203.0.113.155.

The thing that I like about this solution is that it requires no new software. It uses the networking functionality that is already present on the host. However, unless you take additional steps, there is virtually no logging. nftables can log messages to syslog for packets that match the DNAT rule, but even that is very different than the level of logging that a proxy server would provide.

References:

During some initial testing, I was using another EC2 instance as the NAT server and it did not work immediately. That’s because the EC2 instance was not aware of its own public IPv4 address, it was only aware of its private IPv4 address: 172.31.15.224. Removing the SNAT rule for the public IP address and adding a new SNAT rule for 172.31.15.224 solved that.

Linux policy based routing

Problem: I have a host that has 2 active network interfaces. One is used as a management port (eth0), one is used as an FTP dropbox (eth1).

Both can route to the Internet, but all connections other than FTP on eth1 are blocked via iptables. The default route uses the interface for the FTP dropbox, but I have a static route configured for the subnet that includes my management and monitoring hosts so that I can SSH to the host and check on host availability, disk space, mail queue, etc.

However, the static route means that I cannot monitor the FTP dropbox, since FTP connection attempts coming in on one interface and IP address are then routed out via the management interface and IP address.

Solution: Use policy-based routing to direct the system to consult a different routing table for connections coming in on the FTP interface.

It sounds easy enough.
Continue reading Linux policy based routing

Docker versus Podman and iptables

I have recently been learning about podman, a tool for running containers that has a command syntax that matches Docker, but that does not require a Docker daemon and which does not require root privileges.

I ran into some unexpected problems publishing ports with Podman, which had to do with my default DROP policy on the iptables FORWARD chain. Below I will demonstrate some of the differences between Docker and Podman in terms of iptables changes, and provide a workaround for Podman.
Continue reading Docker versus Podman and iptables

Re-bind host to FreeIPA

The sudo command on one particular FreeIPA-bound host was taking an exceedingly long time to run. And when it finally ran, it would not accept my current password, but rather my previous password — somehow still cached on the system. It was a strange problem.

Instead of trying to figure out exactly why it was happening, I decided to remove & re-bind the host to my FreeIPA domain.
Continue reading Re-bind host to FreeIPA

Using blocklist.de with fail2ban

Anyone who runs a server with open ports knows that systems with questionable intent will come knocking. Sometimes thousands of them. fail2ban is software that that checks your server logs and detects multiple failures, for example 5 failed SSH logins in a row, and bans the source IP address a period of time, e.g. for an hour. This helps prevent password-guessing and brute force attacks. It might be useful to share information about those questionable IP addresses with others so that we can block them proactively.

One such list of IP addresses that I found is blocklist.de. Since I am primarily concerned with systems that are trying to SSH into my system, I looked specifically at their SSH blocklist:
All IP addresses which have been reported within the last 48 hours as having run attacks on the service SSH.

Implementation details: Continue reading Using blocklist.de with fail2ban

Using fail2ban with iptables instead of firewalld

In the previous post I wrote about the minor configuration changes needed to get fail2ban to actually do something.

I have been working primarily with CentOS 7 and have been using iptables instead of firewalld. Normally, fail2ban works with iptables by default. However, installing fail2ban on CentOS 7 also installs fail2ban-firewalld — which changes that default. Even with a properly configured fail2ban jail, you will not see the expected results. fail2ban will log events as expected, but no traffic will actually be banned.

The fail2ban-firewalld package places a file in /etc/fail2ban/jail.d/00-firewalld.conf. It overrides the default banaction (iptables) and sets it to firewallcmd-ipset.

The top of the 00-firewalld.conf file says:

You can remove this package (along with the empty fail2ban meta-package) if you do not use firewalld

When I tried removing fail2ban-firewalld, it removed fail2ban as a dependency. I have a feeling the referenced fail2ban meta-package may have something to so with that.

I have not yet investigated the meta-package and de-coupling fail2ban-firewalld from fail2ban (see Update below). My solution, for now, has been to move 00-firewalld.conf and restart fail2ban:

$ sudo mv /etc/fail2ban/jail.d/00-firewalld.conf /etc/fail2ban/jail.d/00-firewalld.disabled
$ sudo systemctl restart fail2ban

The default banaction defined in jail.conf is no longer overridden and performs as expected:
banaction = iptables-multiport

Update
According to Fail2ban with FirewallD, The fail2ban package itself is a meta-package that contains several other packages, including fail2ban-firewalld and fail2ban-server. Removing the meta-package will not remove fail2ban-server.

If you’ve already moved 00-firewalld.conf to 00-firewalld.disabled, you’ll get a warning:
warning: file /etc/fail2ban/jail.d/00-firewalld.conf: remove failed: No such file or directory

You can ignore the warning, or remove 00-firewalld.disabled.

Block an IP address via iptables

I was monitoring the mail logs on a Postfix server and noted repeated failed connection attempts from the same IP address. The source was likely up to no good, and it was making it more difficult to monitor the logs for legitimate connections, so I decided to block it:

iptables -A INPUT -s 123.456.789.101 -j DROP

(IP address changed to protect…the innocent?)

However, the IP address was still making connections:
Dec 2 17:19:05 mercutio postfix/smtpd[15230]: connect from unknown[123.456.789.101]
Dec 2 17:19:06 mercutio postfix/smtpd[15230]: lost connection after AUTH from unknown[123.456.789.101]
Dec 2 17:19:06 mercutio postfix/smtpd[15230]: disconnect from unknown[123.456.789.101]

How is that possible? First I checked iptables to check my sanity and confirm that the rule had been added:

# iptables -L
...
DROP all -- 123.456.789.101 anywhere
...

OK, it’s there. That’s good!

The problem in this case was a different rule that had been added previously. Rules in iptables are processed in order, and no further rules are processed after a matching rule is found. Well above my newly-added rule was this rule:
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:smtp

That rule makes sense for a mail server, but I needed my rule to be inserted before it. I determined which rule it was in the INPUT chain like this:
iptables --line-numbers -L INPUT

It was the 5th rule, so I was able to insert the new rule just above it like this:
iptables -I INPUT 4 -s 123.456.789.101 -j DROP

After that, the offending IP address stopped creating entries in the mail.log.

However, my new rule would disappear after a system restart. Since I am using iptables-persistent, I saved the rules to the config file:
iptables-save > /etc/iptables/rules.v4

To confirm everything worked, I attempted to restart iptables:
# service iptables-persistent restart
Failed to restart iptables-persistent.service: Unit iptables-persistent.service

Apparently the service name changed to netfilter-persistent in Debian 8. The config files are still in the same location, but the service name has changed.

I restarted iptables:
# service netfilter-persistent restart

I checked the rules again and my new rule was there, above the rule allowing connections from any IP on port 25. However, I also noticed the following rule above either of those:
ACCEPT all -- anywhere anywhere

I freaked out. That rule indicates that all traffic from any source on any port should be accepted. That’s the worst firewall rule I’ve ever seen. It basically negates the entire concept of a firewall. It clearly should not be there!

However, using the verbose switch on iptables:
iptables -vL INPUT

I discovered that the rule only applied to the lo interface (loopback). That’s a relief–that rule gets to stay.

iptables and deleting/replacing entries

Whenever I have to reboot my modem [sic] at home, I typically get a new IP address from my ISP.

When that happens, I need to update iptables to allow my new address to connect to the SSH port (port 22) of my jump box (which, fortunately, I have access to from another IP address):

iptables -A INPUT -p tcp -m state --state NEW -s [new IP address] --dport 22 -j ACCEPT

But I don’t want to leave the old entry. How to get rid of it?

The delete (-D) and replace (-R) options require a line number from the chain (e.g. the INPUT chain). To find the line numbers:

iptables -L INPUT --line-numbers

To delete the existing rule and add the new rule:

iptables -D INPUT [line number]
iptables -A INPUT -p tcp -m state --state NEW --dport 22 -s [new IP address] -j ACCEPT

To replace the existing entry:

iptables -R INPUT [line number] -p tcp -m state --state NEW --dport 22 -s [new IP address] -j ACCEPT

Save the updates so they are persistent:

iptables-save > /etc/iptables/rules.v4

(That’s the location for Debian and Ubuntu. This may be different for your distribution.)