3

I have a server with 2 network cards of 4 ports, but only one of them is used. Each port is connected to a L2 switch and configured for a different VLAN with 1 usable IP from a different /24 subnet - each with different default gateway. Only 1 of the ports works properly - eno4. The rest could be pinged, but with losses every other time. I get this error when applying netplan:

Problem encountered while validating default route consistency.Please set up multiple routing tables and use routing-policy instead. Error: Conflicting default route declarations for IPv4 (table: main, metric: default), first declared in eno2 but also in eno4

netplan itself:

network:
  ethernets:
    eno1:
      addresses: ['aa.bb.cc.dd/24', 'a:b:c:x:e:f:g:1/64']
      gateway4: aa.bb.cc.1
      gateway6: a:b:c:x:e:f:g:0001
      nameservers:
          addresses: [1.1.1.1, 1.0.0.1, 8.8.8.8, 8.8.4.4, 2606:4700:4700::1111, 2606:4700:4700::1001, 2001:4860:4860::8888, 2001:4860:4860::8844]
      accept-ra: no
    eno2:
      addresses: ['ee.ff.gg.hh/24', 'a:b:c:y:e:f:g:2/64']
      gateway4: ee.ff.gg.1
      gateway6: a:b:c:y:e:f:g:0001
      nameservers:
          addresses: [1.1.1.1, 1.0.0.1, 8.8.8.8, 8.8.4.4, 2606:4700:4700::1111, 2606:4700:4700::1001, 2001:4860:4860::8888, 2001:4860:4860::8844]
      accept-ra: no
    eno3:
      addresses: ['ii.jj.kk.ll/24', 'a:b:c:z:e:f:g:3/64']
      gateway4: ii.jj.kk.1
      gateway6: a:b:c:z:e:f:g:0001
      nameservers:
          addresses: [1.1.1.1, 1.0.0.1, 8.8.8.8, 8.8.4.4, 2606:4700:4700::1111, 2606:4700:4700::1001, 2001:4860:4860::8888, 2001:4860:4860::8844]
      accept-ra: no
    eno4:
      addresses: ['mm.nn.oo.pp/24', 'a:b:c:v:e:f:g:4/64']
      gateway4: mm.nn.oo.1
      gateway6: a:b:c:v:e:f:g:0001
      nameservers:
          addresses: [1.1.1.1, 1.0.0.1, 8.8.8.8, 8.8.4.4, 2606:4700:4700::1111, 2606:4700:4700::1001, 2001:4860:4860::8888, 2001:4860:4860::8844]
      accept-ra: no
#    ens1f0:
#      dhcp4: true
#    ens1f1:
#      dhcp4: true
#    ens1f2:
#      dhcp4: true
#    ens1f3:
#      dhcp4: true
  version: 2

What am I missing here and how to properly route?

lion
  • 91
  • 1
  • 1
  • 9

1 Answers1

2

There typically can be only one functional default route. The default route is where packets go when there is no obvious better route. If you have multiple default routes, they might be treated round robin style, which would explain why you have packet loss -- some packets are going to different places.

There are multiple possible solutions to this:

  • Use a routing daemon to manage dynamic routes, outside of netplan
  • Assign static routes for specific networks with one default route for everything else.
  • Set up routing tables and priorities so that the selected "default route" depends on the incoming host side IP address, so that responses to packets coming in on a specific ip go out the same interface. Similarly, you would need to prioritize it so that wildcarded outgoing connections select a primary IP for the source address.

It may be necessary to do more than one of these things to make it work.

user10489
  • 3,564
  • 2
  • 5
  • 22
  • OK. Thanks! The last one worked. ip rule add from IP1 table 1 and then ip route add default via GATE1 table 1 Same for the rest. Added it to startup script, so it is applied across reboots. – lion Jan 08 '22 at 23:43
  • There should be a way to do that inside netplan, but I'm not sure. – user10489 Jan 08 '22 at 23:57
  • Same here, so did what I said and seems to have worked. – lion Jan 09 '22 at 17:07
  • So this worked for the outside world, but not for docker containers. How to write static routing for each separate container? – lion Jan 09 '22 at 18:17
  • not enough context to answer, maybe worth a separate question. Ultimately, every packet needs a consistent destination, and every reply must go out on the same ip the corresponding incoming packet came to. – user10489 Jan 09 '22 at 19:38
  • Was a firewall thing. I solved it. :) – lion Jan 09 '22 at 19:45