{"id":2935,"date":"2018-12-30T18:46:13","date_gmt":"2018-12-30T23:46:13","guid":{"rendered":"http:\/\/osric.com\/chris\/accidental-developer\/?p=2935"},"modified":"2020-10-05T16:44:09","modified_gmt":"2020-10-05T21:44:09","slug":"docker-versus-podman-and-iptables","status":"publish","type":"post","link":"https:\/\/osric.com\/chris\/accidental-developer\/2018\/12\/docker-versus-podman-and-iptables\/","title":{"rendered":"Docker versus Podman and iptables"},"content":{"rendered":"<p>I have recently been learning about <a href=\"https:\/\/github.com\/containers\/libpod\">podman<\/a>, a tool for running containers that has a command syntax that matches Docker, but that does not require a Docker daemon and which does not require root privileges.<\/p>\n<p>I ran into some unexpected problems publishing ports with Podman, which had to do with my default DROP policy on the iptables FORWARD chain. Below I will demonstrate some of the differences between Docker and Podman in terms of iptables changes, and provide a workaround for Podman.<br \/>\n<!--more--><\/p>\n<p>To test the differences, I used Amazon AWS EC2 t2.nano instances based on the <a href=\"https:\/\/aws.amazon.com\/marketplace\/pp\/B00O7WM7QW\">CentOS 7 (x86_64) &#8211; with Updates HVM<\/a> AMI. In my test I am going to run a mock HTTP server using netcat, so I opened port 8080 to the world in the AWS security group for these EC2 instances.<\/p>\n<p><strong>Docker<\/strong><\/p>\n<p>After launching the EC2 instance, I ran through the following steps to configure the host:<\/p>\n<ul>\n<li><code>sudo yum update<\/code><\/li>\n<li><code>sudo reboot<\/code><\/li>\n<li>Follow steps to <a href=\"https:\/\/docs.docker.com\/install\/linux\/docker-ce\/centos\/\">install Docker on CentOS<\/a>:\n<ul>\n<li><code>sudo yum-config-manager --add-repo https:\/\/download.docker.com\/linux\/centos\/docker-ce.repo<\/code><\/li>\n<li><code>sudo yum install docker-ce<\/code><\/li>\n<\/ul>\n<\/li>\n<li><code>sudo yum install iptables-service<\/code><\/li>\n<\/ul>\n<p>Change the FORWARD chain policy to DROP in <code>\/etc\/sysconfig\/iptables<\/code>:<\/p>\n<pre><code>:FORWARD DROP [0:0]<\/code><\/pre>\n<p>Restart iptables:<\/p>\n<pre><code>sudo systemctl restart iptables<\/code><\/pre>\n<p>I tried running a simple container:<\/p>\n<pre><code>sudo docker run --rm -it --public 8080:80 alpine sh\r\ndocker: Cannot connect to the Docker daemon at unix:\/\/\/var\/run\/docker.sock. Is the docker daemon running?.\r\nSee 'docker run --help'.<\/code><\/pre>\n<p>Oh right, the Docker daemon needed to be running:<\/p>\n<pre><code>sudo systemctl start docker<\/code><\/pre>\n<p>I ran the simple container again, this time successfully:<\/p>\n<pre><code>sudo docker run --rm -it --public 8080:80 alpine sh<\/code><\/pre>\n<p>From inside the container, I started <code>netcat<\/code> listening on port 80:<\/p>\n<pre><code># nc -l -p 80<\/code><\/pre>\n<p>I was then able to connect to the container from the outside world, e.g. my desktop, like so:<\/p>\n<pre><code>curl 3.85.222.191:8080<\/code><\/pre>\n<p>I saw the request header appear via <code>netcat<\/code> within the container:<\/p>\n<pre><code>GET \/ HTTP\/1.1\r\nHost: 3.85.222.191:8080\r\nUser-Agent: curl\/7.56.1\r\nAccept: *\/*<\/code><\/pre>\n<p>From there I entered:<\/p>\n<pre><code>Hello, world\r\n[^d]<\/code><\/pre>\n<p>And confirmed that the response text was received by curl. The entire communication was successful.<\/p>\n<p>Docker sets up some routes so that traffic to a container IP address is handled by the <code>docker0<\/code> interface:<\/p>\n<pre><code>$ route\r\nKernel IP routing table\r\nDestination     Gateway         Genmask         Flags Metric Ref    Use Iface\r\ndefault         ip-172-31-80-1. 0.0.0.0         UG    0      0        0 eth0\r\n172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0\r\n172.31.80.0     0.0.0.0         255.255.240.0   U     0      0        0 eth0<\/code><\/pre>\n<p>Docker makes a lot of changes to <code>iptables<\/code> rules, most of which are filter rules (there are a couple tweaks to NAT rules, which have ignored for now). Primarily I want to focus on the filter rules in the FORWARD chain, since that chain directly affects packets coming in on the <code>eth0<\/code> interface that are destined for the <code>docker0<\/code> interface.<\/p>\n<pre><code>$ sudo iptables -v -L FORWARD\r\nChain FORWARD (policy DROP 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination \r\n   39  2093 DOCKER-USER  all  --  any    any     anywhere             anywhere  \r\n   39  2093 DOCKER-ISOLATION-STAGE-1  all  --  any    any     anywhere             anywhere\r\n    7   621 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED\r\n   13   660 DOCKER     all  --  any    docker0  anywhere             anywhere   \r\n   19   812 ACCEPT     all  --  docker0 !docker0  anywhere             anywhere \r\n    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere  \r\n    0     0 REJECT     all  --  any    any     anywhere             anywhere             reject-with icmp-host-prohibited<\/code><\/pre>\n<p>Let&#8217;s look at what each rule is doing:<\/p>\n<ol>\n<li>All forwarded packets are sent to the DOCKER-USER chain. In this case, the DOCKER-USER chain RETURNs all packets, and so are further processed by the other FORWARD chain rules.<\/li>\n<li>All forwarded packets are sent to the DOCKER-ISOLATION-STAGE-1 chain. This one is a bit more complex, so I&#8217;m going to wave my hands and say that all the packets I sent survived this rule and are RETURNed for further processing.<\/li>\n<li>Any packets forwarded from any interface to the <code>docker0<\/code> interface are accepted if the connection state is RELATED or ESTABLISHED.<\/li>\n<li>Any packets forwarded from any interface to the <code>docker0<\/code> interface are directed to the DOCKER chain. More on this in a minute, but our connection should be ACCEPTed here.<\/li>\n<li>Any packets from the <code>docker0<\/code> interface to any other interface are accepted<\/li>\n<li>Any packets from the <code>docker0<\/code> interface to the <code>docker0<\/code> interface are accepted<\/li>\n<li>All other packets are REJECTed<\/li>\n<\/ul>\n<p>The DOCKER chain is only populated while the container is running, so if you run <code>iptables<\/code> while the container is stopped you may not see any rules here:<\/p>\n<pre><code>$ sudo iptables -v -L DOCKER\r\nChain DOCKER (1 references)\r\n pkts bytes target     prot opt in     out     source               destination \r\n    1    52 ACCEPT     tcp  --  !docker0 docker0  anywhere             ip-172-17-0-2.ec2.internal  tcp dpt:http<\/code><\/pre>\n<p>This rule accepts TCP packets from interfaces other than <code>docker0<\/code> destined for <code>docker0<\/code> on destination port 80.<\/p>\n<p><strong>Podman<\/strong><\/p>\n<p>After launching the EC2 instance, I ran through the following steps to configure the host:<\/p>\n<ul>\n<li><code>sudo yum update<\/code><\/li>\n<li><code>sudo reboot<\/code><\/li>\n<li><code>sudo yum install podman<\/code><\/li>\n<li><code>sudo yum install iptables-services<\/code><\/li>\n<\/ul>\n<p>Change the FORWARD chain policy to DROP in <code>\/etc\/sysconfig\/iptables<\/code>:<\/p>\n<pre><code>:FORWARD DROP [0:0]<\/code><\/pre>\n<p>Restart iptables:<\/p>\n<pre><code>sudo systemctl restart iptables<\/code><\/pre>\n<p>I ran a simple container:<\/p>\n<pre><code>sudo podman run --rm -it --publish 8080:80 alpine sh<\/code><\/pre>\n<p>From inside the container, I started <code>netcat<\/code> listening on port 80:<\/p>\n<pre><code># nc -l -p 80<\/code><\/pre>\n<p>I attempted to connect to the container from the outside world, e.g. my desktop:<\/p>\n<pre><code>curl 18.206.214.125:8080<\/code><\/pre>\n<p>However, the connection failed due to a timeout.<\/p>\n<p>I attempted to connect to the container from its host:<\/p>\n<pre><code>curl 10.88.0.1:8080<\/code><\/pre>\n<p>This worked, and I see the request header appear in the container as expected:<\/p>\n<pre><code>GET \/ HTTP\/1.1\r\nUser-Agent: curl\/7.29.0\r\nHost: 10.88.0.1:8080\r\nAccept: *\/*<\/code><\/pre>\n<p>Why wasn&#8217;t the connection completing from my desktop? I suspected that something wasn&#8217;t routing between the interfaces correctly.<\/p>\n<p>In the routing table, note that the CNI (Container Networking Interface) used by Podman uses a different default range of RFC 1918 IP addresses (10.88.0.0\/16 instead of Docker&#8217;s 172.17.0.0\/16):<\/p>\n<pre><code>$ route\r\nKernel IP routing table\r\nDestination     Gateway         Genmask         Flags Metric Ref    Use Iface\r\ndefault         ip-172-31-80-1. 0.0.0.0         UG    0      0        0 eth0\r\n10.88.0.0       0.0.0.0         255.255.0.0     U     0      0        0 cni0\r\nlink-local      0.0.0.0         255.255.0.0     U     1002   0        0 eth0\r\n172.31.80.0     0.0.0.0         255.255.240.0   U     0      0        0 eth0<\/code><\/pre>\n<p>I looked at the iptables FORWARD chain, where is was clear that all forwarded packets first jump to the CNI-FORWARD chain:<\/p>\n<pre><code>$ sudo iptables -v -L FORWARD\r\nChain FORWARD (policy DROP 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination \r\n    4   212 CNI-FORWARD  all  --  any    any     anywhere             anywhere             \/* CNI firewall plugin rules *\/\r\n    4   212 REJECT     all  --  any    any     anywhere             anywhere             reject-with icmp-host-prohibited<\/code><\/pre>\n<p>Next I looked at the iptables CNI-FORWARD chain:<\/p>\n<pre><code>$ sudo iptables -v -n -L CNI-FORWARD\r\nChain CNI-FORWARD (1 references)\r\n pkts bytes target     prot opt in     out     source               destination \r\n    4   212 CNI-ADMIN  all  --  *      *       0.0.0.0\/0            0.0.0.0\/0            \/* CNI firewall plugin rules *\/\r\n    0     0 ACCEPT     all  --  *      *       0.0.0.0\/0            10.88.0.11           ctstate RELATED,ESTABLISHED\r\n    0     0 ACCEPT     all  --  *      *       10.88.0.11           0.0.0.0\/0<\/code><\/pre>\n<ol>\n<li>The CNI-FORWARD chain first jumps to the CNI-ADMIN chain, which in this case was empty.<\/li>\n<li>Next it ACCEPTs any packets from RELATED or ESTABLISHED connections destined for the container IP address.<\/li>\n<li>Next it ACCEPTS any packets from the container to any destination<\/li>\n<\/ol>\n<p>If none of those rules match, it returns to the FORWARD chain where it hits the REJECT rule. What&#8217;s missing is a rule that allows NEW connection packets destined for the container. There are a variety of ways to do this. At first I tried adding an overly broad rule directly to the FORWARD chain:<\/p>\n<pre><code>sudo iptables -I FORWARD -p tcp ! -i cni0 -o cni0 -j ACCEPT<\/code><\/pre>\n<p>That rule ACCEPTs any TCP packets from an interface other than <code>cni0<\/code> bound for <code>cni0<\/code>. And it worked: after added that rule I was able to reach the container from my desktop via port 8080. However, that rule allows much more than what Docker allows. I removed that rule and added a more specific rule to the CNI-FORWARD chain:<\/p>\n<pre><code>sudo iptables -I CNI-FORWARD -p tcp ! -i cni0 -o cni0 -d 10.88.0.11 --dport 80 -j ACCEPT<\/code><\/pre>\n<p>That also worked, and much more closely resembled what Docker added to iptables.<\/p>\n<p><strong>Why doesn&#8217;t Podman add the necessary iptables rule automatically?<\/strong><\/p>\n<p>Good question. I&#8217;m using the podman package provided via yum from the CentOS extras repository, which provides podman version 0.11.1.1. It may be that future releases will address this. It may also be the case that Podman does not anticipate a default DROP policy on the FORWARD chain (although default DROP is best practice).<\/p>\n<p>In any case, it is useful to know a little bit more about how container tools use iptables. It is also important to note that, at least at this point, Docker is easier to use than Podman.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Published ports worked fine for me in Docker, but not when using Podman, RedHat&#8217;s new container tool with Docker-like command line syntax. Why not? It turned out the reason was due to my default DROP policy for the iptables FORWARD chain. In this post I take a closer look at why this works when using Docker, but not when using Podman.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[451],"tags":[449,408,530],"class_list":["post-2935","post","type-post","status-publish","format-standard","hentry","category-docker","tag-docker","tag-iptables","tag-podman"],"_links":{"self":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2935","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/comments?post=2935"}],"version-history":[{"count":10,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2935\/revisions"}],"predecessor-version":[{"id":3323,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/posts\/2935\/revisions\/3323"}],"wp:attachment":[{"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/media?parent=2935"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/categories?post=2935"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/osric.com\/chris\/accidental-developer\/wp-json\/wp\/v2\/tags?post=2935"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}