Docker versus Podman and iptables

I have recently been learning about podman, a tool for running containers that has a command syntax that matches Docker, but that does not require a Docker daemon and which does not require root privileges.

I ran into some unexpected problems publishing ports with Podman, which had to do with my default DROP policy on the iptables FORWARD chain. Below I will demonstrate some of the differences between Docker and Podman in terms of iptables changes, and provide a workaround for Podman.

To test the differences, I used Amazon AWS EC2 t2.nano instances based on the CentOS 7 (x86_64) – with Updates HVM AMI. In my test I am going to run a mock HTTP server using netcat, so I opened port 8080 to the world in the AWS security group for these EC2 instances.

Docker

After launching the EC2 instance, I ran through the following steps to configure the host:

• sudo yum update
• sudo reboot
• Follow steps to install Docker on CentOS:
  • sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  • sudo yum install docker-ce
• sudo yum install iptables-service

Change the FORWARD chain policy to DROP in /etc/sysconfig/iptables:

:FORWARD DROP [0:0]

Restart iptables:

sudo systemctl restart iptables

I tried running a simple container:

sudo docker run --rm -it --public 8080:80 alpine sh
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.

Oh right, the Docker daemon needed to be running:

sudo systemctl start docker

I ran the simple container again, this time successfully:

sudo docker run --rm -it --public 8080:80 alpine sh

From inside the container, I started netcat listening on port 80:

# nc -l -p 80

I was then able to connect to the container from the outside world, e.g. my desktop, like so:

curl 3.85.222.191:8080

I saw the request header appear via netcat within the container:

GET / HTTP/1.1
Host: 3.85.222.191:8080
User-Agent: curl/7.56.1
Accept: */*

From there I entered:

Hello, world
[^d]

And confirmed that the response text was received by curl. The entire communication was successful.

Docker sets up some routes so that traffic to a container IP address is handled by the docker0 interface:

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         ip-172-31-80-1. 0.0.0.0         UG    0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.31.80.0     0.0.0.0         255.255.240.0   U     0      0        0 eth0

Docker makes a lot of changes to iptables rules, most of which are filter rules (there are a couple tweaks to NAT rules, which have ignored for now). Primarily I want to focus on the filter rules in the FORWARD chain, since that chain directly affects packets coming in on the eth0 interface that are destined for the docker0 interface.

$ sudo iptables -v -L FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination 
   39  2093 DOCKER-USER  all  --  any    any     anywhere             anywhere  
   39  2093 DOCKER-ISOLATION-STAGE-1  all  --  any    any     anywhere             anywhere
    7   621 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
   13   660 DOCKER     all  --  any    docker0  anywhere             anywhere   
   19   812 ACCEPT     all  --  docker0 !docker0  anywhere             anywhere 
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere  
    0     0 REJECT     all  --  any    any     anywhere             anywhere             reject-with icmp-host-prohibited

Let’s look at what each rule is doing:

1. All forwarded packets are sent to the DOCKER-USER chain. In this case, the DOCKER-USER chain RETURNs all packets, and so are further processed by the other FORWARD chain rules.
2. All forwarded packets are sent to the DOCKER-ISOLATION-STAGE-1 chain. This one is a bit more complex, so I’m going to wave my hands and say that all the packets I sent survived this rule and are RETURNed for further processing.
3. Any packets forwarded from any interface to the docker0 interface are accepted if the connection state is RELATED or ESTABLISHED.
4. Any packets forwarded from any interface to the docker0 interface are directed to the DOCKER chain. More on this in a minute, but our connection should be ACCEPTed here.
5. Any packets from the docker0 interface to any other interface are accepted
6. Any packets from the docker0 interface to the docker0 interface are accepted
7. All other packets are REJECTed

The DOCKER chain is only populated while the container is running, so if you run iptables while the container is stopped you may not see any rules here:

$ sudo iptables -v -L DOCKER
Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination 
    1    52 ACCEPT     tcp  --  !docker0 docker0  anywhere             ip-172-17-0-2.ec2.internal  tcp dpt:http

This rule accepts TCP packets from interfaces other than docker0 destined for docker0 on destination port 80.

Podman

After launching the EC2 instance, I ran through the following steps to configure the host:

• sudo yum update
• sudo reboot
• sudo yum install podman
• sudo yum install iptables-services

Change the FORWARD chain policy to DROP in /etc/sysconfig/iptables:

:FORWARD DROP [0:0]

Restart iptables:

sudo systemctl restart iptables

I ran a simple container:

sudo podman run --rm -it --publish 8080:80 alpine sh

From inside the container, I started netcat listening on port 80:

# nc -l -p 80

I attempted to connect to the container from the outside world, e.g. my desktop:

curl 18.206.214.125:8080

However, the connection failed due to a timeout.

I attempted to connect to the container from its host:

curl 10.88.0.1:8080

This worked, and I see the request header appear in the container as expected:

GET / HTTP/1.1
User-Agent: curl/7.29.0
Host: 10.88.0.1:8080
Accept: */*

Why wasn’t the connection completing from my desktop? I suspected that something wasn’t routing between the interfaces correctly.

In the routing table, note that the CNI (Container Networking Interface) used by Podman uses a different default range of RFC 1918 IP addresses (10.88.0.0/16 instead of Docker’s 172.17.0.0/16):

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         ip-172-31-80-1. 0.0.0.0         UG    0      0        0 eth0
10.88.0.0       0.0.0.0         255.255.0.0     U     0      0        0 cni0
link-local      0.0.0.0         255.255.0.0     U     1002   0        0 eth0
172.31.80.0     0.0.0.0         255.255.240.0   U     0      0        0 eth0

I looked at the iptables FORWARD chain, where is was clear that all forwarded packets first jump to the CNI-FORWARD chain:

$ sudo iptables -v -L FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination 
    4   212 CNI-FORWARD  all  --  any    any     anywhere             anywhere             /* CNI firewall plugin rules */
    4   212 REJECT     all  --  any    any     anywhere             anywhere             reject-with icmp-host-prohibited

Next I looked at the iptables CNI-FORWARD chain:

$ sudo iptables -v -n -L CNI-FORWARD
Chain CNI-FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination 
    4   212 CNI-ADMIN  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI firewall plugin rules */
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            10.88.0.11           ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  *      *       10.88.0.11           0.0.0.0/0

1. The CNI-FORWARD chain first jumps to the CNI-ADMIN chain, which in this case was empty.
2. Next it ACCEPTs any packets from RELATED or ESTABLISHED connections destined for the container IP address.
3. Next it ACCEPTS any packets from the container to any destination

If none of those rules match, it returns to the FORWARD chain where it hits the REJECT rule. What’s missing is a rule that allows NEW connection packets destined for the container. There are a variety of ways to do this. At first I tried adding an overly broad rule directly to the FORWARD chain:

sudo iptables -I FORWARD -p tcp ! -i cni0 -o cni0 -j ACCEPT

That rule ACCEPTs any TCP packets from an interface other than cni0 bound for cni0. And it worked: after added that rule I was able to reach the container from my desktop via port 8080. However, that rule allows much more than what Docker allows. I removed that rule and added a more specific rule to the CNI-FORWARD chain:

sudo iptables -I CNI-FORWARD -p tcp ! -i cni0 -o cni0 -d 10.88.0.11 --dport 80 -j ACCEPT

That also worked, and much more closely resembled what Docker added to iptables.

Why doesn’t Podman add the necessary iptables rule automatically?

Good question. I’m using the podman package provided via yum from the CentOS extras repository, which provides podman version 0.11.1.1. It may be that future releases will address this. It may also be the case that Podman does not anticipate a default DROP policy on the FORWARD chain (although default DROP is best practice).

In any case, it is useful to know a little bit more about how container tools use iptables. It is also important to note that, at least at this point, Docker is easier to use than Podman.