Podman container port forwarding has issue with secondary NIC

Question

My title may not be very clear

I have a server running CentOS 8:

eth0 connects to subnet 10.0.3.0/24, having IP 10.0.3.15

eth1 connects to subnet 10.0.103.0/24, having IP 10.0.103.15

As they have different gateways, I have IP rules:

$ ip rule list
0:  from all lookup local
1024:   from all to 172.16.5.0/24 lookup main
5000:   from 10.0.3.0/24 lookup t1
6000:   from 10.0.103.0/24 lookup t2
32766:  from all lookup main
32767:  from all lookup default

and route tables:

$ ip route show table t1
default via 10.0.3.1 dev eth0 proto static metric 100
10.0.3.1 dev eth0 proto static scope link metric 100

$ ip route show table t2
default via 10.0.103.1 dev eth1 proto static metric 101
10.0.103.1 dev eth1 proto static scope link metric 101

The above configuration should work fine at least for the host network, as I can ping both 10.0.3.15 and 10.0.103.15 from another server.

However when I create two redis containers with podman:

podman run -d --name redis1 -p 10.0.3.15:6379:6379 redis:apline
podman run -d --name redis2 -p 10.0.103.15:6379:6379 redis:apline

Only redis1 is accessible with telnet [IP] 6379, telnet 10.0.103.15 6379 gets nowhere, if I use tcpdump in the container redis2 I can't see anything coming in. However if I use telnet 10.88.0.3 6379 while 10.88.0.3 is the internal IP of the redis2 container, I can connect to the redis from the host.

So I suspect maybe iptables NAT rules has issue, but this is my iptables:

$ iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
CNI-HOSTPORT-DNAT  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
CNI-HOSTPORT-MASQ  all  --  0.0.0.0/0            0.0.0.0/0            /* CNI portfwd requiring masquerade */
KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
CNI-97d3d333b7a91bb85f60a8e8  all  --  10.88.0.3            0.0.0.0/0            /* name: "podman" id: "f39d5e41593bc7f949db7bae5d06baa6d72606edec97a818a0842640ac557af5" */
CNI-0131634b11335acac74ae092  all  --  10.88.0.4            0.0.0.0/0            /* name: "podman" id: "65616612a5e5db3ef42cdae3cb40e65390bba2a826f0c7f097e3ed577e38590a" */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
CNI-HOSTPORT-DNAT  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain KUBE-MARK-DROP (1 references)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x8000

Chain KUBE-MARK-MASQ (3 references)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose */ match-set KUBE-LOOP-BACK dst,dst,src
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            mark match ! 0x4000/0x4000
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK xor 0x4000
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ random-fully

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-LOAD-BALANCER  all  --  0.0.0.0/0            0.0.0.0/0            /* Kubernetes service lb portal */ match-set KUBE-LOAD-BALANCER dst,dst
KUBE-MARK-MASQ  all  --  0.0.0.0/0            0.0.0.0/0            /* Kubernetes service cluster ip + port for masquerade purpose */ match-set KUBE-CLUSTER-IP src,dst
KUBE-NODE-PORT  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-CLUSTER-IP dst,dst
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            match-set KUBE-LOAD-BALANCER dst,dst

Chain KUBE-FIREWALL (0 references)
target     prot opt source               destination
KUBE-MARK-DROP  all  --  0.0.0.0/0            0.0.0.0/0

Chain KUBE-NODE-PORT (1 references)
target     prot opt source               destination
RETURN     tcp  --  0.0.0.0/0            0.0.0.0/0            /* Kubernetes nodeport TCP port with externalTrafficPolicy=local */ match-set KUBE-NODE-PORT-LOCAL-TCP dst
KUBE-MARK-MASQ  tcp  --  0.0.0.0/0            0.0.0.0/0            /* Kubernetes nodeport TCP port for masquerade purpose */ match-set KUBE-NODE-PORT-TCP dst

Chain KUBE-LOAD-BALANCER (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            /* Kubernetes service load balancer ip + port with externalTrafficPolicy=local */ match-set KUBE-LOAD-BALANCER-LOCAL dst,dst
KUBE-MARK-MASQ  all  --  0.0.0.0/0            0.0.0.0/0

Chain CNI-HOSTPORT-SETMARK (4 references)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0            /* CNI portfwd masquerade mark */ MARK or 0x2000

Chain CNI-HOSTPORT-MASQ (1 references)
target     prot opt source               destination
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            mark match 0x2000/0x2000

Chain CNI-HOSTPORT-DNAT (2 references)
target     prot opt source               destination
CNI-DN-97d3d333b7a91bb85f60a  tcp  --  0.0.0.0/0            0.0.0.0/0            /* dnat name: "podman" id: "f39d5e41593bc7f949db7bae5d06baa6d72606edec97a818a0842640ac557af5" */ multiport dports 6379
CNI-DN-0131634b11335acac74ae  tcp  --  0.0.0.0/0            0.0.0.0/0            /* dnat name: "podman" id: "65616612a5e5db3ef42cdae3cb40e65390bba2a826f0c7f097e3ed577e38590a" */ multiport dports 6379

Chain CNI-97d3d333b7a91bb85f60a8e8 (1 references)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            10.88.0.0/16         /* name: "podman" id: "f39d5e41593bc7f949db7bae5d06baa6d72606edec97a818a0842640ac557af5" */
MASQUERADE  all  --  0.0.0.0/0           !224.0.0.0/4          /* name: "podman" id: "f39d5e41593bc7f949db7bae5d06baa6d72606edec97a818a0842640ac557af5" */

Chain CNI-DN-97d3d333b7a91bb85f60a (1 references)
target     prot opt source               destination
CNI-HOSTPORT-SETMARK  tcp  --  10.88.0.0/16         10.0.103.15          tcp dpt:6379
CNI-HOSTPORT-SETMARK  tcp  --  127.0.0.1            10.0.103.15          tcp dpt:6379
LOG        tcp  --  0.0.0.0/0            10.0.103.15          tcp dpt:6379 LOG flags 0 level 4 prefix "iptables: "
DNAT       tcp  --  0.0.0.0/0            10.0.103.15          tcp dpt:6379 to:10.88.0.3:6379

Chain CNI-0131634b11335acac74ae092 (1 references)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            10.88.0.0/16         /* name: "podman" id: "65616612a5e5db3ef42cdae3cb40e65390bba2a826f0c7f097e3ed577e38590a" */
MASQUERADE  all  --  0.0.0.0/0           !224.0.0.0/4          /* name: "podman" id: "65616612a5e5db3ef42cdae3cb40e65390bba2a826f0c7f097e3ed577e38590a" */

Chain CNI-DN-0131634b11335acac74ae (1 references)
target     prot opt source               destination
CNI-HOSTPORT-SETMARK  tcp  --  10.88.0.0/16         10.0.3.15            tcp dpt:6379
CNI-HOSTPORT-SETMARK  tcp  --  127.0.0.1            10.0.3.15            tcp dpt:6379
LOG        tcp  --  0.0.0.0/0            10.0.3.15            tcp dpt:6379 LOG flags 0 level 4 prefix "iptables: "
DNAT       tcp  --  0.0.0.0/0            10.0.3.15            tcp dpt:6379 to:10.88.0.4:6379

As you can see, it's identical for 10.88.0.3 (redis2, which has issue) and 10.88.0.4 (redis1, which works)

and nothing special for the broken container in filter table:

$ iptables -n -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  169.254.0.0/16       0.0.0.0/0            tcp dpt:10250
ACCEPT     tcp  --  192.168.0.0/16       0.0.0.0/0            tcp dpt:10250
ACCEPT     tcp  --  172.16.0.0/12        0.0.0.0/0            tcp dpt:10250
ACCEPT     tcp  --  30.0.0.0/8           0.0.0.0/0            tcp dpt:10250
ACCEPT     tcp  --  11.0.0.0/8           0.0.0.0/0            tcp dpt:10250
ACCEPT     tcp  --  9.0.0.0/8            0.0.0.0/0            tcp dpt:10250
ACCEPT     tcp  --  10.0.0.0/8           0.0.0.0/0            tcp dpt:10250
ACCEPT     tcp  --  127.0.0.0/8          0.0.0.0/0            tcp dpt:10250
KUBE-NODE-PORT  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes health check rules */
KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:10250

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
CNI-FORWARD  all  --  0.0.0.0/0            0.0.0.0/0            /* CNI firewall plugin rules */
KUBE-FORWARD  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
DROP       all  -- !127.0.0.0/8          127.0.0.0/8          /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding conntrack rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-NODE-PORT (1 references)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* Kubernetes health check node port */ match-set KUBE-HEALTH-CHECK-NODE-PORT dst

Chain CNI-FORWARD (1 references)
target     prot opt source               destination
CNI-ADMIN  all  --  0.0.0.0/0            0.0.0.0/0            /* CNI firewall plugin admin overrides */
ACCEPT     all  --  0.0.0.0/0            10.88.0.3            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  10.88.0.3            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            10.88.0.4            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  10.88.0.4            0.0.0.0/0

Chain CNI-ADMIN (1 references)
target     prot opt source               destination

What could be wrong?

score 0 · Answer 1 · answered Jul 03 '23 at 17:47

0

Do you see the logs for following rule in dmesg?

LOG        tcp  --  0.0.0.0/0            10.0.103.15          tcp dpt:6379 LOG flags 0 level 4 prefix "iptables: "

answered Jul 03 '23 at 17:47

idle_roamer

1

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jul 03 '23 at 18:06
Yes, I can see logs coming into the host with this rule – Phoenix Jul 04 '23 at 02:05

Podman container port forwarding has issue with secondary NIC

1 Answers1