Problem: I have a host that has 2 active network interfaces. One is used as a management port (
eth0), one is used as an FTP dropbox (
Both can route to the Internet, but all connections other than FTP on eth1 are blocked via iptables. The default route uses the interface for the FTP dropbox, but I have a static route configured for the subnet that includes my management and monitoring hosts so that I can SSH to the host and check on host availability, disk space, mail queue, etc.
However, the static route means that I cannot monitor the FTP dropbox, since FTP connection attempts coming in on one interface and IP address are then routed out via the management interface and IP address.
Solution: Use policy-based routing to direct the system to consult a different routing table for connections coming in on the FTP interface.
It sounds easy enough.
In some circumstances we want to route packets differently depending not only on destination addresses, but also on other packet fields: source address, IP protocol, transport protocol ports or even packet payload. This task is called ‘policy routing’.
It turns out, policy routing capabilities are quite flexible, but the implementation details are a little complex. Here are the steps I took to make sure that packets associated with inbound connections on eth1 also went out via eth1.
First of all, a few details:
- IPv4 address of monitoring host: 192.168.200.44
- IPv4 address of FTP interface: 192.168.100.9
- IPv4 address of gateway in the subnet containing the FTP interface: 192.168.100.1
Step 1: Mark packets and connections coming in on eth1
For this, I used the iptables MARK and CONNMARK targets (see
sudo iptables -A PREROUTING -t mangle -i eth1 -j MARK --set-mark 1 sudo iptables -A PREROUTING -t mangle -i eth1 -j CONNMARK --save-mark sudo iptables -A OUTPUT -t mangle -j CONNMARK --restore-mark
- The first line sets a 32-bit mark on packets incoming on interface
eth1. (I have also seen this mark referred to as fwmark, nfmark, and Netfilter mark.)
- The second line copies the packet mark to the connection mark for packets incoming on interface
eth1. Since iptables tracks connection state, outbound replies to inbound packets will be treated as part of the same connection. You can view the iptables connection state via
- The third line copies the connection mark to the packet mark for all outbound packets.
This is important, because the packets we are interested in routing are outbound packets.
Step 2: Create a table in the Routing Policy Database (RPDB)
By default, there are 3 tables in the database: local, main, and default. You can confirm this by running
ip rule show:
$ ip rule show 0: from all lookup local 32766: from all lookup main 32767: from all lookup default
You can add your own label by editing
/etc/iproute2/rt_tables. I added the following line:
The label is required, but is otherwise for convenience and does not need to refer to the interface. The entry in
rt_tables is needed for the next step.
Step 3: Add a rule to the RPDB
sudo ip rule add priority 1000 fwmark 0x1 table eth1_table
The priority is an arbitrary between 0 and 32766, exclusive. The lowest priority value has the highest priority, so after this change you can confirm that the new rule will be evaluated after the local lookup, but before the main and default lookups:
$ ip rule show 0: from all lookup local 1000: from all fwmark 0x1 lookup eth1_table 32766: from all lookup main 32767: from all lookup default
The new rule indicates that packets with a mark value of 1 should consult the eth1_table routing table.
Step 4: Add a route to the routing table
sudo ip route add table eth1_table 0.0.0.0/0 via 192.168.100.1 dev eth1 src 192.168.100.9
After you add this route, you may expect to see it when you run
ip route show. However, the main table is displayed by default if you do not specify a table. Instead, use this:
$ ip route show table eth1_table default via 192.168.100.1 dev eth1 src 192.168.100.9
Step 5: update kernel parameter
To make this change persistent, I created
/etc/sysctl.d/10-eth1.conf containing the following line:
Be sure to adjust the permissions as needed, e.g.:
$ chmod 0644 /etc/sysctl.d/10-eth1.conf $ chown root:root /etc/sysctl.d/10-eth1.conf
To load the new value immediately without rebooting:
sudo sysctl -p /etc/sysctl.d/10-eth1.conf
Once that change was made, my monitoring host was successfully able to receive ping replies from and establish FTP connections to the IPv4 address of the host.