Linux policy based routing

Problem: I have a host that has 2 active network interfaces. One is used as a management port (eth0), one is used as an FTP dropbox (eth1).

Both can route to the Internet, but all connections other than FTP on eth1 are blocked via iptables. The default route uses the interface for the FTP dropbox, but I have a static route configured for the subnet that includes my management and monitoring hosts so that I can SSH to the host and check on host availability, disk space, mail queue, etc.

However, the static route means that I cannot monitor the FTP dropbox, since FTP connection attempts coming in on one interface and IP address are then routed out via the management interface and IP address.

Solution: Use policy-based routing to direct the system to consult a different routing table for connections coming in on the FTP interface.

It sounds easy enough.

From man ip-rule:

In some circumstances we want to route packets differently depending not only on destination addresses, but also on other packet fields: source address, IP protocol, transport protocol ports or even packet payload. This task is called ‘policy routing’.

It turns out, policy routing capabilities are quite flexible, but the implementation details are a little complex. Here are the steps I took to make sure that packets associated with inbound connections on eth1 also went out via eth1.

First of all, a few details:

• IPv4 address of monitoring host: 192.168.200.44
• IPv4 address of FTP interface: 192.168.100.9
• IPv4 address of gateway in the subnet containing the FTP interface: 192.168.100.1

Step 1: Mark packets and connections coming in on eth1

For this, I used the iptables MARK and CONNMARK targets (see man iptables-extensions).

sudo iptables -A PREROUTING -t mangle -i eth1 -j MARK --set-mark 1
sudo iptables -A PREROUTING -t mangle -i eth1 -j CONNMARK --save-mark
sudo iptables -A OUTPUT -t mangle -j CONNMARK --restore-mark

1. The first line sets a 32-bit mark on packets incoming on interface eth1. (I have also seen this mark referred to as fwmark, nfmark, and Netfilter mark.)
2. The second line copies the packet mark to the connection mark for packets incoming on interface eth1. Since iptables tracks connection state, outbound replies to inbound packets will be treated as part of the same connection. You can view the iptables connection state via /proc/net/nf_conntrack.
3. The third line copies the connection mark to the packet mark for all outbound packets.

This is important, because the packets we are interested in routing are outbound packets.

Step 2: Create a table in the Routing Policy Database (RPDB)

By default, there are 3 tables in the database: local, main, and default. You can confirm this by running ip rule show:

$ ip rule show
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

You can add your own label by editing /etc/iproute2/rt_tables. I added the following line:

100	eth1_table

The label is required, but is otherwise for convenience and does not need to refer to the interface. The entry in rt_tables is needed for the next step.

Step 3: Add a rule to the RPDB

sudo ip rule add priority 1000 fwmark 0x1 table eth1_table

The priority is an arbitrary value between 0 and 32766, exclusive. The lowest priority value has the highest priority, so after this change you can confirm that the new rule will be evaluated after the local lookup, but before the main and default lookups:

$ ip rule show
0:	from all lookup local
1000:	from all fwmark 0x1 lookup eth1_table
32766:	from all lookup main
32767:	from all lookup default

The new rule indicates that packets with a mark value of 1 should consult the eth1_table routing table.

Step 4: Add a route to the routing table

sudo ip route add table eth1_table 0.0.0.0/0 via 192.168.100.1 dev eth1 src 192.168.100.9

After you add this route, you may expect to see it when you run ip route show. However, the main table is displayed by default if you do not specify a table. Instead, use this:

$ ip route show table eth1_table
default via 192.168.100.1 dev eth1 src 192.168.100.9

Step 5: update kernel parameter net.ipv4.conf.eth1.src_valid_mark

To make this change persistent, I created /etc/sysctl.d/10-eth1.conf containing the following line:

net.ipv4.conf.eth1.src_valid_mark=1

Be sure to adjust the permissions as needed, e.g.:

$ chmod 0644 /etc/sysctl.d/10-eth1.conf
$ chown root:root /etc/sysctl.d/10-eth1.conf

To load the new value immediately without rebooting:

sudo sysctl -p /etc/sysctl.d/10-eth1.conf

Once that change was made, my monitoring host was successfully able to receive ping replies from and establish FTP connections to the IPv4 address of the host.