{"id":2982,"date":"2019-03-05T21:53:20","date_gmt":"2019-03-06T02:53:20","guid":{"rendered":"http:\/\/osric.com\/chris\/accidental-developer\/?p=2982"},"modified":"2021-11-09T09:57:16","modified_gmt":"2021-11-09T14:57:16","slug":"linux-policy-based-routing","status":"publish","type":"post","link":"https:\/\/osric.com\/chris\/accidental-developer\/2019\/03\/linux-policy-based-routing\/","title":{"rendered":"Linux policy based routing"},"content":{"rendered":"<p><strong>Problem:<\/strong> I have a host that has 2 active network interfaces. One is used as a management port (<code>eth0<\/code>), one is used as an FTP dropbox (<code>eth1<\/code>).<\/p>\n<p>Both can route to the Internet, but all connections other than FTP on eth1 are blocked via iptables. The default route uses the interface for the FTP dropbox, but I have a static route configured for the subnet that includes my management and monitoring hosts so that I can SSH to the host and check on host availability, disk space, mail queue, etc.<\/p>\n<p>However, the static route means that I cannot monitor the FTP dropbox, since FTP connection attempts coming in on one interface and IP address are then routed out via the management interface and IP address.<\/p>\n<p><strong>Solution:<\/strong> Use policy-based routing to direct the system to consult a different routing table for connections coming in on the FTP interface.<\/p>\n<p>It <em>sounds<\/em> easy enough.<br \/>\n<!--more--><\/p>\n<p>From <code>man ip-rule<\/code>:<\/p>\n<blockquote><p>In some circumstances we want to route packets differently depending not only on destination addresses, but also on other packet fields: source address, IP protocol, transport protocol ports or even packet payload. This task is called &#8216;policy routing&#8217;.<\/p><\/blockquote>\n<p>It turns out, policy routing capabilities are quite flexible, but the implementation details are a little complex. Here are the steps I took to make sure that packets associated with inbound connections on eth1 also went out via eth1.<\/p>\n<p>First of all, a few details:<\/p>\n<ul>\n<li>IPv4 address of monitoring host: 192.168.200.44<\/li>\n<li>IPv4 address of FTP interface: 192.168.100.9<\/li>\n<li>IPv4 address of gateway in the subnet containing the FTP interface: 192.168.100.1<\/li>\n<\/ul>\n<p><strong>Step 1: Mark packets and connections coming in on eth1<\/strong><\/p>\n<p>For this, I used the iptables MARK and CONNMARK targets (see <code>man iptables-extensions<\/code>).<\/p>\n<pre><code>sudo iptables -A PREROUTING -t mangle -i eth1 -j MARK --set-mark 1\r\nsudo iptables -A PREROUTING -t mangle -i eth1 -j CONNMARK --save-mark\r\nsudo iptables -A OUTPUT -t mangle -j CONNMARK --restore-mark<\/code><\/pre>\n<ol>\n<li>The first line sets a 32-bit mark on packets incoming on interface <code>eth1<\/code>. (I have also seen this mark referred to as fwmark, nfmark, and Netfilter mark.)<\/li>\n<li>The second line copies the packet mark to the connection mark for packets incoming on interface <code>eth1<\/code>. Since iptables tracks connection state, outbound replies to inbound packets will be treated as part of the same connection. You can view the iptables connection state via <code>\/proc\/net\/nf_conntrack<\/code>.<\/li>\n<li>The third line copies the connection mark to the packet mark for all outbound packets.<\/li>\n<\/ol>\n<p>This is important, because the packets we are interested in routing are outbound packets.<\/p>\n<p><strong>Step 2: Create a table in the Routing Policy Database (RPDB)<\/strong><\/p>\n<p>By default, there are 3 tables in the database: <em>local<\/em>, <em>main<\/em>, and <em>default<\/em>. You can confirm this by running <code>ip rule show<\/code>:<\/p>\n<pre><code>$ ip rule show\r\n0:\tfrom all lookup local\r\n32766:\tfrom all lookup main\r\n32767:\tfrom all lookup default<\/code><\/pre>\n<p>You can add your own label by editing <code>\/etc\/iproute2\/rt_tables<\/code>. I added the following line:<\/p>\n<pre><code>100\teth1_table<\/code><\/pre>\n<p>The label is required, but is otherwise for convenience and does not need to refer to the interface. The entry in <code>rt_tables<\/code> is needed for the next step.<\/p>\n<p><strong>Step 3: Add a rule to the RPDB<\/strong><\/p>\n<pre><code>sudo ip rule add priority 1000 fwmark 0x1 table eth1_table<\/code><\/pre>\n<p>The priority is an arbitrary value between 0 and 32766, exclusive. The lowest priority value has the highest priority, so after this change you can confirm that the new rule will be evaluated after the <em>local<\/em> lookup, but before the <em>main<\/em> and <em>default<\/em> lookups:<\/p>\n<pre><code>$ ip rule show\r\n0:\tfrom all lookup local\r\n1000:\tfrom all fwmark 0x1 lookup eth1_table\r\n32766:\tfrom all lookup main\r\n32767:\tfrom all lookup default<\/code><\/pre>\n<p>The new rule indicates that packets with a mark value of 1 should consult the <em>eth1_table<\/em> routing table.<\/p>\n<p><strong>Step 4: Add a route to the routing table<\/strong><\/p>\n<pre><code>sudo ip route add table eth1_table 0.0.0.0\/0 via 192.168.100.1 dev eth1 src 192.168.100.9<\/code><\/pre>\n<p>After you add this route, you may expect to see it when you run <code>ip route show<\/code>. However, the <em>main<\/em> table is displayed by default if you do not specify a table. Instead, use this:<\/p>\n<pre><code>$ ip route show table eth1_table\r\ndefault via 192.168.100.1 dev eth1 src 192.168.100.9<\/code><\/pre>\n<p><strong>Step 5: update kernel parameter <code>net.ipv4.conf.eth1.src_valid_mark<\/code><\/strong><\/p>\n<p>To make this change persistent, I created <code>\/etc\/sysctl.d\/10-eth1.conf<\/code> containing the following line:<\/p>\n<pre><code>net.ipv4.conf.eth1.src_valid_mark=1<\/code><\/pre>\n<p>Be sure to adjust the permissions as needed, e.g.:<\/p>\n<pre><code>$ chmod 0644 \/etc\/sysctl.d\/10-eth1.conf\r\n$ chown root:root \/etc\/sysctl.d\/10-eth1.conf<\/code><\/pre>\n<p>To load the new value immediately without rebooting:<\/p>\n<pre><code>sudo sysctl -p \/etc\/sysctl.d\/10-eth1.conf<\/code><\/pre>\n<p>Once that change was made, my monitoring host was successfully able to receive ping replies from and establish FTP connections to the IPv4 address of the host.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Linux policy routing allows you to use different routing rules based on various criteria. In this example, I mark all packets coming in on a specific network interface and route outbound packets that are part of the same connection accordingly.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[408,534],"class_list":["post-2982","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-iptables","tag-routing"],"_links":{"self":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2982","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/comments?post=2982"}],"version-history":[{"count":10,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2982\/revisions"}],"predecessor-version":[{"id":3561,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2982\/revisions\/3561"}],"wp:attachment":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/media?parent=2982"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/categories?post=2982"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/tags?post=2982"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}