Routing using iproute2 after NAT

Question

In a Linux system, which acts as a gateway on my lan, I tried to route traffic using iproute2. Moreover, before routing it is necessary to perform the NAT, since the linux machine is connected to a device which allow internet connection and it has 2 IP addresses 172.16.61.1 and 172.16.62.100

I have 2 network cards with the following configuration :

DEVICE=eth3
ONBOOT=yes
BOOTPROTO=none
TYPE=Ethernet
NETMASK=255.255.255.0
IPADDR=192.168.1.150
USERCTL=no
IPV6INIT=no
PEERDNS=yes  

DEVICE=eth4
ONBOOT=yes
BOOTPROTO=none
TYPE=Ethernet
NETMASK=255.255.255.0
IPADDR=172.16.61.2
USERCTL=no
IPV6INIT=no
PEERDNS=yes
GATEWAY=172.16.61.1

and I use the following instruction for ip translation via iptables

/sbin/iptables -t nat -A POSTROUTING -s 192.168.1.0/255.255.255.0 -j SNAT --to-source 172.16.61.2

Using the aforementioned configuration, all computers belonging to the 192.168.1.0/24 network which have 192.168.1.150 as gateway are able to connect to the internet.

If I try to use iproute2 to configure the gateway, then I remove the default gateway from eth4 which assumes the following configuration :

DEVICE=eth4
ONBOOT=yes
BOOTPROTO=none
TYPE=Ethernet
NETMASK=255.255.255.0
IPADDR=172.16.61.2
USERCTL=no
IPV6INIT=no
PEERDNS=yes

AND I performed the following steps:

1. In /etc/iproute2/rt_tables I have added the line 1 route61
2. /sbin/ip route add 172.16.61.0/24 via 172.16.61.1 table route61 proto static
3. /sbin/ip route add default via 172.16.61.1 table route61 proto static
4. /sbin/ip rule add from 172.16.61.0/24 pref 15000 table route61

The output of /sbin/ip route show table route61 is

172.16.61.0/24 via 172.16.61.1 dev eth4  proto static  
default via 172.16.61.1 dev eth4  proto static

The output of /sbin/ip rule show is

0:      from all lookup local 
15000:  from 172.16.61.0/24 lookup route61 
32766:  from all lookup main 
32767:  from all lookup default

but in this case it doesn't work, what am I wrong?

I also tried to use /etc/sysconfig/network-scripts/ifup-routes eth4 with no success

What I need is to understand why I can't get "table based routing" to work as I will have to use it in a context with multiple gateways

Update

To try what is suggested by dirkt, I removed the default gateway

route del -net 0.0.0.0 gw 172.16.61.1 netmask 0.0.0.0 dev eth4

and I used:

ip rule add from 192.168.1.0/24 pref 15000 table route61

what happens is

the pc that uses 192.168.1.150 as GW, that is the pc object of this post, is able to connect to the internet
I can't connect to GW anymore unless I remove the instruction ip rule add from 192.168.1.0/24 pref 15000 table route61

With postrouting the packets coming from 192.168.1.0/24 shouldn't assume 172.16.61.2 as source address ?

If i use or remove default GW using route add -net 0.0.0.0 gw 172.16.61.1 netmask 0.0.0.0 dev eth4 OR route del -net 0.0.0.0 gw 172.16.61.1 netmask 0.0.0.0 dev eth4 and I do :

ip route get 216.58.205.78 from 192.168.1.5

I have RTNETLINK answers: Invalid argument

if I use ip route get 216.58.205.78 from 172.16.61.2 I have 216.58.205.78 from 172.16.61.2 via 172.16.61.1 dev eth4

After route add -net 0.0.0.0 gw 172.16.61.1 netmask 0.0.0.0 dev eth4 if I do :

nc -v 216.58.205.78 443

I have

Connection to 216.58.205.78 443 port [tcp/https] succeeded!

If I remove default GW route del -net 0.0.0.0 gw 172.16.61.1 netmask 0.0.0.0 dev eth4 nc connects only if I specify source ip:

nc -v 216.58.205.78 443 -s 172.16.61.2

Is this about **routing**, or about **NAT**? You do not need to use NAT for routing. You only need it if this box is connected to something that needs a single IP address. If this is e.g. a home router, then you end up with double NAT. So please edit your question and add information about your **complete** network setup. And there are probably alternative network setups which may work better. — dirkt, Jul 27 '21 at 08:36
Instead of `172.16.61.0/24 via 172.16.61.1 dev eth4 proto static`, you should have `172.16.61.1 dev eth4 proto static` or `172.16.61.0/24 dev eth4 proto static`. (Actually it could even be unnecessary.) Also you should have a single SNAT rule with NEITHER `-p tcp` NOR `-p udp`. — Tom Yan, Jul 27 '21 at 13:32
Besides, is there a proper reason for you to do policy routing? Seems unnecessary to me in your case. — Tom Yan, Jul 27 '21 at 13:33

dirkt · Answer 1 · 2021-07-27T18:59:06.150

If I understand you correctly, you have 192.168.1.0/24 on eth3, and you want to route and NAT that to something behind eth4, were it's unspecified what exactly is behind eth4, if it does DHCP, and if it needs static addresses or not.

(The usual case is that you have some kind of router behind eth4, which runs DHCP, and gives out addresses).

So first, the usual way to achieve that is to enable forwarding and do

iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

where $EXTIF is your external interface (eth4). The MASQUERADE is SNAT with whatever address happens to be on eth4, so this is resilient against address reassignment. And if you don't mention protocols, just everything gets NAT'ed, which is usually what you want.

Second, there isn't any reason to do table based routing (unless there's additional details which you didn't explain). So forget about the tables, and if there's no DHCP behind eth4 which would distribute the gateway information for the default route, just do

ip route default via 172.16.61.1

Assuming that eth4 already got an address in 172.16.61.*/24, this is enough (the 172.16.61.0/24 route gets set when you set the ip address).

But the best things is to just leave this to DHCP, if it's enabled on your router (I guess that would be BOOTPROTO=dhcp, if that is Red Hat).

If the problem is the policy routing: You want to route outgoing packets coming in from 192.168.1.0/24 via a default route in table route61, so you need to do

ip rule add from 192.168.1.0/24 pref 15000 table route61

and not from 172.16.61.0/24.

That said, I am not actually sure how NAT interacts with policy routing. I'd assume the returning packets get deNATed first, and then the rules in the main table will route them correctly, but I've never tried that myself.

To debug, ip route get X.X.X.X from Y.Y.Y.Y may be helpful, as will tcpdump on both interfaces like you already did in the other question you mentioned.

It will also be easier to first try this without tables. Once that works, you know nothing else gets in the way, and you can try with tables.

Thanks for your answer, as you have identified what I proposed is part of a larger problem as you can see in https://superuser.com/questions/1664793/route-based-on-source-ip-address-via-2-gateways-on-same-nic Having not found a solution to the main problem, I am trying to face it by dividing it into sub-parts — famedoro, Jul 27 '21 at 13:44
Behind eth4 there is a vpn that allows me to access the internet — famedoro, Jul 27 '21 at 13:45
What I need is to understand why I can't get it to work table based routing as I will have to use it in a context with multiple gateways — famedoro, Jul 27 '21 at 13:58
What I need is to understand why I can't get "table based routing" to work as I will have to use it in a context with multiple gateways — famedoro, Jul 27 '21 at 14:03
To try what you recommend I removed the default gateway route del -net 0.0.0.0 gw 172.16.61.1 netmask 0.0.0.0 dev eth4 and I used: ip rule add from 192.168.1.0/24 pref 15000 table route61 what happens is 1) the pc that uses 192.168.1.150 as GW, that is the pc object of this post, is able to connect to the internet 2) I can't connect to GW anymore unless I remove the instruction ip rule add from 192.168.1.0/24 pref 15000 table route61 With postrouting the packets coming from 192.168.1.0/24 shouldn't assume 172.16.61.2 as address? — famedoro, Aug 10 '21 at 09:00

score 0 · Answer 2 · answered Aug 15 '21 at 16:14

Using a new routing table, you have to add even connected routes : 172.16.61.0/24 via 172.16.61.1 dev eth4 proto static should be proto scope link instead with the source IP specified, and you miss the 192.168.1.0/24 route Routing decision is performed before output / post routing nat so ip rule add from should use the original source IP.

# Configure tables
echo '2 table61' >> /etc/iproute2/rt_tables 
echo '3 table62' >> /etc/iproute2/rt_tables 

# routing decision is performed before output / post routing nat
ip rule add from 192.168.1.5 lookup table62
ip rule add from 192.168.1.0/24 lookup table61

ip route add 192.168.1.0/24 dev eth3 proto kernel scope link src 192.168.1.150 table table61
ip route add 172.16.61.0/24 dev eth4 proto kernel scope link src 172.16.61.2 table table61
ip route add default via 172.16.61.1 table table61

ip route add 192.168.1.0/24 dev eth3 proto kernel scope link src 192.168.1.150 table table62
ip route add 172.16.62.0/24 dev eth4 proto kernel scope link src 172.16.62.100 table table62
ip route add default via 172.16.62.254 table table62

Also, I might have mixed your questions in my head, if it doesn't work immediately, could you please add the following to your question?

ip a
ip r show table table61
ip r show table table62
ip rule show
iptables -L -n -v
iptables -t nat -L -n -v

Routing using iproute2 after NAT

2 Answers2