0

I am setting up a cloud environment on Ubuntu and I am running into an interesting problem. Out of the box Ubuntu (all linux really) doesn't allow you to specify more than one gateway in /etc/network/interfaces

Since I am not using DHCP for any of the NICs and I am using vLANs on my switch and router (reference image below) I need each NIC on this server to not just have it's own IP but have it's own gateway specified. Since vLAN 2 which is 10.0.1.0/24 cannot access 10.0.0.1 which is the default gateway but inaccessible due to netmask 255.255.255.0.

All my research has lead me to understand that I need to configure the routing tables by hand, however, when I try to add routing for more than the first eth0 NIC I get the following error: 

# ip route add default via 10.0.1.1 dev eth1 table eth1
RTNETLINK answers: File exists

enter image description here

At this point I am lost for things to try... I cannot add the routes to the new route tables, and without explicit routes each NIC card tries to use 10.0.0.1 as the gateway since it's the default gateway for 10.0.0.0/24

Ethode
  • 200
  • 10
  • 2
    You already have a default gateway, and you can _only_ have one! What exactly are you trying to route where? – Michael Hampton Apr 08 '15 at 23:09
  • Read my entire post :-) – Ethode Apr 08 '15 at 23:10
  • 2
    I did. It doesn't make sense. Which is why I asked you to clarify. – Michael Hampton Apr 08 '15 at 23:10
  • I need each NIC to operate on separate networks, and vLAN's do not share a gateway, therefore if you want them to make it to the router for things like MAC binding it needs to use the vLAN gateway address which is respectively 10.0.0.1 (management network) 10.0.1.1 (vLAN 2) 10.0.2.1 (vLAN 3) The only way to use 10.0.0.1 as the gateway for all three NIC's and keep my IP scheme is to move the netmask t0 255.255.0.0 which would in effect eliminate the security of the separate networks in the first place – Ethode Apr 08 '15 at 23:13
  • I should also mention it really doesn't matter if I'm dealing with vLAN's or literally 2 physical networks, I can't imagine it's THAT uncommon to have NIC's communicate on different networks lol – Ethode Apr 08 '15 at 23:15
  • https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System – hookenz Apr 08 '15 at 23:22
  • @Matt thanks for the link, that's actually the post I've been using to configure, and the command in my question uses the very same command he asks to run, however, I receive that error message "RTNETLINK answers: File exists" when I do – Ethode Apr 08 '15 at 23:25
  • Playing around with this in the past I recall some time back that it's because the command line was incorrect in all the examples... As usual, people copy and paste stuff all around the net. Check the man pages and manually try it until you find the right combo. I'm sure it's possible... it might also be a change to the ip command somewhere down the line. Can't quite recall but I did encounter this one time. – hookenz Apr 08 '15 at 23:28
  • I suspect that's what I'll wind up doing tonight then hahah..Trial and fire.. I mean trial and error :) – Ethode Apr 08 '15 at 23:29
  • It doesn't look like the route command in your question is copied from that howto, the command in your question didn't include a table specifier. – Zoredache Apr 09 '15 at 00:56
  • @Zoredache, you're correct, I wound up using the same route table just to verify it's indeed my problem. I could and am still debating to create separate route tables, one for each NIC and setting the default GW, but I've lost motivation to keep working on it tonight,and I am just happy my vLANs are talking :-) – Ethode Apr 09 '15 at 02:18

2 Answers2

2

So here's the answer after taking Matt's suggestion and going the trial and error route.

The command I used had the following pattern ip route add via dev

The final command actually was

# ip route add 10.0.1.31 via 10.0.1.1 dev eth1
# ip route add 10.0.2.31 via 10.0.2.1 dev eth2

After adding these I had no issues pinging between the IP's on the vLAN.. Fantastic...

Essentially the difference here is that I added a static route so that interaction next hop is sent to the vLAN gateway ID instead of going to the default gateway.

Also so these are permanent I wound up adding them to the /etc/network/interfaces config file as post-up configs. So I wound up with this line for eth1 and eth2, but I left eth0 alone since it can use the default gateway

post-up route add 10.0.1.31 via 10.0.1.1 dev eth1
Ethode
  • 200
  • 10
  • 1
    If this solved the problem, why did you keep going on about default gateways, which have nothing to do with this? You should have automatically gotten routes for these if you configured the `netmask` correctly in `/etc/network/interfaces`. – Michael Hampton Apr 09 '15 at 01:31
  • The route I added IS the gateway. Just as @zoredash mentioned below, an alternative would have been to add multiple routing tables for each subnet and then adding a default GW to each routing table; thus giving me three default gateways at the end. I just happen to prefer the simple one liner I can add to interfaces instead. Changing the netmask wouldn't fix anything because I need the network separation. Had I opened up the netmask to be 255.255.0.0 instead of 255.255.255.0 sure it would have found the route, but then I would have lost the network separation. I WANT /24 it's not an accident – Ethode Apr 09 '15 at 02:13
0

You can only have one default gateway, that's why it's called default. You can create static routes for the other NICs for particular networks reachable on that NICs VLAN.

Jim G.
  • 2,657
  • 1
  • 19
  • 19
  • That's precisely what I attempted to do, and reference in my question. However, I receive that error message which will not allow me to add the route which I thought was odd... That being said, it's actually incorrect to state that all NIC cards CANNOT have separate gateways. Using route tables it's possible and there are about 1000 posts showing how to do it on Google but most of not applicable to Debian flavors and the ones that are debian based have me run the command I posted in my question (which fails), so I am stuck :-( – Ethode Apr 08 '15 at 23:19
  • 1
    Nonsense, the Linux kernel allows you to specify multiple default gateways and weight them. – hookenz Apr 08 '15 at 23:20
  • 2
    You can only have one useful 'gateway' per route table. Sure you can have backups by setting metrics that will take over if the main gateway fails, and on Linux you can have **a** gateway that is a weighted set of addresses. But really you can only have one effective 'default' gateway per route table. – Zoredache Apr 09 '15 at 00:54
  • @Zoredache you're correct, if I add multiple routing tables I also can verify that they work, as they should provided I have the proper ordering. However, I have to admit, I kind of like the 1 liner I add to the /etc/network/interfaces config file a bit better, just seems cleaner than adding brand new route tables and setting up default gw in each. Do you see a potential downside to using a static route in the post-up vs separate route tables with a gw in each? – Ethode Apr 09 '15 at 01:22