Assume you have a Linux system with more than one network interface card (NIC) — say eth0 and eth1. By default, administrators can define a single, default route (on eth0). However, if you receive traffic (i.e., ICMP pings) on eth1, the return traffic will go out eth0 by default.
This can be a bit of a problem — especially when the two NICs share the same parent network and you’re trying to preserve sane traffic flows. In a nutshell, this post will explain how you can ensure traffic going into eth0 goes out only on eth0, as well as enforce all traffic going into eth1 goes out only on eth1.
You’ve found the one post that actually explains this issue; your googling has paid off. You wouldn’t believe how many advanced Linux routing websites out there explain how to route everything including your kitchen sink — yet fail to clearly explain something as simple as this.
As always, we’ll explain by example. Assume the following:
eth0 - 10.10.70.38 netmask 255.255.255.0eth0's gateway is: 10.10.70.254eth1 - 192.168.7.126 netmask 255.255.255.0eth1's gateway is: 192.168.7.1
First, you’ll need to make sure your Linux kernel has support for “policy routing” enabled. (As a reference, I’m using a v2.6.13-gentoo-r5 kernel.)
During the kernel compilation process, you’ll want to:
cd /usr/src/linux
make menuconfig
Select "Networking --->"
Select "Networking options --->"
[*] TCP/IP networking
[*] IP: advanced router
Choose IP: FIB lookup algorithm (FIB_HASH)
[*] IP: policy routing
[*] IP: use netfilter MARK value as routing key
Next, you’ll want to download, compile, and install the iproute2 [1] utilities. (Most Linux distributions have binary packages for this utility.) Once installed, typing ip route show should bring up your system’s routing table. Type man ip for more information about this utility, in general.
Speaking of which, assume the system’s initial route configuration looks like this:
# netstat -anr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.7.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.10.70.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 192.168.7.1 0.0.0.0 UG 0 0 0 eth1
So, basically, the system is using eth1 as the default route. If anyone pings 192.168.7.126, then the response packets will properly go out eth1 to the upstream gateway of 192.168.7.1. But what about pinging 10.10.70.38? Sure, the incoming ICMP packets will properly arrive on eth0, but the outgoing response packets will be sent out via eth1! That’s bad.
Here’s how to fix this issue. Borrowing the method from a really sketchy website [2], you’ll first need to create a new policy routing table entry within the /etc/iproute2/rt_tables. Let’s call it table #1, named “admin” (for routing administrative traffic onto eth0).
# echo "1 admin" >> /etc/iproute2/rt_tables
Next, we’re going to set a couple of new entries within this “admin” table. Specifically, we’ll provide information about eth0‘s local /24 subnet, along with eth0‘s default gateway.
ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
ip route add default via 10.10.70.254 dev eth0 table admin
At this point, you’ve created a new, isolated routing table named “admin” that really isn’t used by the OS just yet. Why? Because we still need to create a rule referencing how the OS should use this table. For starters, type ip rule show to see your current policy routing ruleset. Here’s what an empty ruleset looks like:
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
Without going into all the boring details, each rule entry is evaluated in ascending order. The main gist is that your normal main routing table appears as entry 32766 in this list. (This would be the normal route table you’d see when you type netstat -anr.)
We’re now going to create two new rule entries, that will be evaluated before the main rule entry.
ip rule add from 10.10.70.38/32 table admin
ip rule add to 10.10.70.38/32 table admin
Typing ip rule show now shows the following policy routing rulesets:
0: from all lookup local
32764: from all to 10.10.70.38 lookup admin
32765: from 10.10.70.38 lookup admin
32766: from all lookup main
32767: from all lookup default
Rule 32764 specifies that for all traffic going to eth0‘s IP, make sure to use the “admin” routing table, instead of the “main” one. Likewise, rule 32765 indicates that for all traffic originating from eth0‘s IP, make sure to use the “admin” routing table as well. For all other packets, use the “main” routing table. In order to commit these changes, it’s a good idea to type ip route flush cache.
Congratulations! You’re system should now properly route traffic to these two different default gateways. For more than 2 NICs, repeat the table/rule creation process as necessary.
Please provide comments, if you find any errors or have corrections to this post. I don’t claim that this method will work for everyone; this information is designed primarily to preserve my sanity, when configuring routing on future multi-NIC Linux systems.
References:
[1] http://www.policyrouting.org
[2] http://www.linuxhorizon.ro/iproute2.html
Update: Here are some additional resources, that I have found useful.
http://lartc.org/howto/lartc.rpdb.multiple-links.html
http://linux-ip.net/html/routing-tables.html
Update: Apparently, OpenBSD also now supports multiple default routes through a new feature called the Virtual Routing Table:
http://www.packetmischief.ca/2011/09/20/virtualizing-the-openbsd-routing-table/
Darien Kindlund's Blog 
379 Comments
What a fantastic article, you’ve saved my butt, I had my 2 nic system configured ok, but every reboot would change it, and voila my listening servers would nose dive. this works perfectly
This is just exactly what I have been looking for, and you are right googling has paid off. I was looking for something that solved this issue, and went straight to the point. Many articles written out there tends to touch on the issue but never really explains what’s happening or maybe it seems to them so simplistic, they overlook the point all together. I don’t know if they ever think about the novices reading their articles…. This one sure did it for me, thank you very much.
Great article indeed… but I’m in a situation where I need to be able to fail over to a different “default” gateway in case of the primary default gateway is unreachable. Lets say I have three upstream routers 10.0.0.1, 10.0.0.2, 10.0.0.3/24. Is there a way for me to inject all 3 IPs into a routing table as my “default” routes on a Linux host?
RE: Failover Default Routes
Here are some articles that describe what you’re looking for:
http://www.uwsg.iu.edu/hypermail/linux/net/0009.1/0027.html
http://www.linux.com/articles/113988
http://mailman.ds9a.nl/pipermail/lartc/2002q1/002667.html
OpenBSD also has a daemon called “hoststated” that provides this type of capability:
http://www.openbsd.org/cgi-bin/man.cgi?query=hoststated&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html
Thank you very much for the links…
gc_timeout did the trick for me as the default is 5 min which is too long for a route to fail over.
just wondering if there is a kernel value to control how long a “” arp entry may stay in the arp cache…
#arp -a
hosta (100.100.100.1) at on eth0
not sure why the word “incomplete” did not show up in my previous post…
RE: ARP Cache Timeout
Try this link, for information about how to control how long entries stay in the ARP cache:
http://www.ussg.indiana.edu/hypermail/linux/net/0012.2/0035.html
great article
thanks to all who rights this article
Great article
Thanks to all who Wright this article
Thanks for this article. It helped me get past a roadblock in a project.
i’m actually kind of curious – the most common, default situation is where eth0 is the interface that packets are routed through — heck, that’s noted in the first sentence — so i’m wondering why you constructed your example with the reverse configuration – with eth1 as the default? i found it kind of confusing, in an otherwise most excellent article….
Sorry about the confusion; in my example, the Linux distribution I used in my example (Gentoo) seemed to always specify eth1 as the default route — when both eth0 and eth1 were present and active (and when automatic configurations were used).
Let me ask you this, in your Linux distribution, when you have both an eth0 and an eth1 on the system with active links for both — does your distribution automatically select eth0 as the default route? I’m guessing it’s pretty much random, as the standard defaults are to use DHCP for both eth0 and eth1, in which case, whichever interface gets the DHCP lease (first? or last?) will ultimately decide which default route actually gets used by the system.
I hope this explains where I was coming from.
oh, okay, that makes sense. i’m always running servers with fixed IP’s, so DHCP never enters into the picture.
for what i’m attempting, i actually found that going with bonded interfaces worked best – using a bond0 device and making both interfaces capable of “believing” that they’re the server’s IP, and the only path out. thus if one interface goes offline, traffic can still pass.
I have a slightly different requirement.
I have internet access both over WLAN and over DSL. I use the DSL for VoIP setup and WLAN for internet. But sometimes the WLAN or the DSL goes out of service. In this caseI want to use both the services – Internet and my VoIP setup over the either WLAN or DSL whichever is active.
How could I do this?
Thanks,
Amit
kahnoie@gmail.com
Hi Amit,
See my earlier comment about “Failover Default Routes”:
— Darien
Hi,
Thank you for this helpfull article.
Your exemples are related to 2 NICs : eth0 and eth1. But, what if instead of physical eth0 and eth1, we have a bonding logical one’s like this :
bond0 : eth0 + eth1
bond1 : eth3 + eth4.
That is we have 4 NICS or a quad card in a server.
In that case, what will be the configation.
Tahnks,
Bras
Hi Bras,
Actually the configuration would be almost identical with the example, but just replace eth0 with ‘bond0’ and eth1 with ‘bond1’.
Linux treats ‘bond0’ and ‘bond1’ as physical NICs, so the rest of the networking subsystem will simply think that these are physical devices.
Hope that helps,
— Darien
Hi,
Many thanks for this wonderful Info !!!
how can I make this permanent, so that
when my server reboots, I will have the same ruling.
Where can table admin and IP ruling
mods be placed …
Thanks in advance…
Each Linux distribution is slightly different. Some distributions offer a way to save these configurations within a configuration file under /etc to be used upon reboot.
Other distributions require you to add the “echo” and “ip rule” commands into a script file that gets called as one of the last commands when the system boots up.
For example, on Ubuntu Linux, there is a script file called “/etc/rc.local” that you can add all the “echo” and “ip rule” commands into, that will apply these configurations upon complete server bootup.
However, please be aware that if you have other network services that require the multiple default routes to be set up BEFORE those services can successfully load (e.g., an automatic VPN/IPSec tunnel), then you’ll have to create an rc.d init script, similar to those found in the “/etc/init.d” directory.
The short answer:
– Try adding the commands to “/etc/rc.local”.
– If other services fail, then make a copy of a simple script in “/etc/init.d”; then edit that script to reflect the commands needed to be run at boot time; then register that script to be started upon bootup (“man update-rc.d”, for Ubuntu/Debian Linux).
Hope this helps,
— Darien
Hello,
This is a good article, but when you say : “However, if you receive traffic (i.e., ICMP pings) on eth1, the return traffic will go out eth0 by default.”, car you please explain or give examples where we will or can receive traffic on eth1 if we have eth0 as a gateway.
Thanks
Lasy
Sure,
Lasy, when you set a single default route, then that route is relevant for *all traffic outside your local subnet*.
Let’s say you have a public class-C subnet of 128.253.1.0/255.255.255.0 and eth0 is 128.253.1.1 while eth1 is 128.253.1.2 and your default route is through eth0.
Then, let’s say someone from a completely different subnet (from 128.5.1.1, for example) pings your ETH1 IP address (128.253.1.2). The ICMP request packets will arrive on eth1; however, when your system sends out the ICMP response packets, *because the destination is outside your local subnet*, these packets will go out on eth0, since that is your default route.
The point of the article is that if you have two or more network interfaces and have only one default *route* set on the OS, then *for all non-local-subnet traffic*, all outgoing packets will go out on the interface marked as the default route — unless you follow the steps listed in this article.
— Darien
I have a backup server that has 2 NIC’s. One NIC is on my Server VLAN 10.1.0.0/16 and the other is on my storage VLAN 10.240.1.0/24. What I would like to do is force the backup traffic coming in on the 10.1.0.0/16 NIC to then be routed out to the backup NAS on the 10.240.1.0/24 NIC. How cna this be done?
Thanks
Thanx 4 that! Everything now seems to work… NICE DONE!
Great article! Thank you very much for writing :o)
At last! Thanks for writing this. While it didn’t save a me a lot of time I *already* wasted, it was the solution to my networking conundrum. You rock.
Thanks for this writeup! It saved me a lot of time!
That’s a very handy piece of writing, Darien ! Thanks for posting it.
I can’t find IP_ROUTE_FWMARK in the new kernel for marking the packets (I’m using 2.6.27) or by googling. Has it been removed ?
Any pointers would be great.
GB
Hi Ghostbuster,
Yes, IP_ROUTE_FWMARK was removed from linux v2.6.20 and later releases. See this link for details:
http://cateee.net/lkddb/web-lkddb/IP_ROUTE_FWMARK.html
Presumably, the feature still exists, but it’s probably named something else (or perhaps merged with some other feature).
Hope this helps,
— Darien
Hi Darien,
Thanks for that pointer.
I’m thinking it’s the MARK command in Netfilter these days.
http://www.linuxtopia.org/Linux_Firewall_iptables/x4368.html
I’ve done some qdisc stuff so hopefully it won’t be too alien.
GB
Okay, scrap that last comment 🙂
I’ll try this instead hoping the support is all handled by iproute2 these days:
http://www.physics.umd.edu/pnce/pcs-docs/Glue/linux-route-hack.html
GB
Kick-ass ! This fixed a problem I was having with an FTP server with two NICS. I could only get it to work with one default gateway at a time. Using what I found here, I created two additional entries, one for each NIC. Works perfectly now. I had to use a slight variation on the ip route statement (using CentOS) – it was: “ip route add 10.1.250.0/24 via 10.1.250.1 dev eth0 table inside” instead of dev eth0 src . Great article – thanks so much !!
This is a really good article that gets me 90% of where I need to go, I really appreciate it. I was wondering if you also know how to configure a system with 10 nics with addresses on the same subnet to send packets directly to the same gateway instead of passing them indirectly through the dfault route on the machine. So, for example, 10.0.0.2->10.0.0.1 and 10.0.0.3->10.0.0.1 instead of
10.0.0.2->10.0.0.1 and 10.0.0.3->10.0.0.2->10.0.0.1. (where 10.0.0.1 is the gateway for that subnet, and 10.0.0.2 is the address of the interface with the default local route.)
I see how your example works for addresses on different subnets, but not for my situation. I would appreciate any ideas!
Hi Howard,
Yes, I’m familiar with that problem. To be honest, this problem is solved at a different layer than routing. What you’re looking for is “Link Aggregation” (aka. making multiple NICs in your server act as one large NIC). There is an entire page dedicated to the concept, available here:
http://en.wikipedia.org/wiki/802.3ad
In order to get it to work, though, you’ll need to make sure your upstream switch/router supports the 802.3ad protocol (most recent managed switches do support it; however, if you have an unmanaged switch, probably not).
Hope this helps,
Darien
Hi,
Am a little lost. On which file do you put all those commands? Secondly, after you have done all the configurations as shown in your solution, what will be the output when you do netstat -rn. Your assistance is highly appreciated.
Hi Anderson,
There is no specific file, as all the commands I’ve listed are performed from the command line. If you’re looking to run these commands every time your OS boots, then you can put them all in /etc/rc.local (for Ubuntu Linux).
Second, ‘netstat -rn’ can’t show you “policy routes” properly. So even after you run the policy routing commands, it’s likely ‘netstat -rn’ will still show you the same thing before and after the policy routes are in place.
If you’re looking to verify that policy routes are working properly, run the “ip rule show” before and after setting up policy routes. If you want more detailed information, see the man pages for the “ip” command, by typing “man ip”.
— Darien
Hi Darien,
Thanks so much for your response. One more question. Should I include the respective gateways in each interface configuration files for my case ifcfcg-eth0 and ifcfcg-eth1? My OS is RedHat El5. My default gateway for eth0 I have defined it in /etc/sysconfig/network file. Where should I define the default gateway for eth1?
Hi Anderson,
You can define your eth1 gateway in the set of commands you enter into /etc/rc.local
The “ip rule” commands Listed in the article will properly configure your eth1 gateway, so you won’t need to define it within /etc/sysconfig/network
— Darien
What do you change if both NICs are on the same network segment, using the same gateway IP? I tried following the steps above but I was left with all traffic going out eth0 while some went in eth1.
Hi nobodyfamous,
If both NICs are on the same network segment, the policy routing isn’t going to help you. The problem has to do with Linux kernel’s TCP/IP stack employing short-circuit routing.
Specifically, this problem is solved at a different layer than routing. What you’re looking for is “Link Aggregation” (aka. making multiple NICs in your server act as one large NIC). There is an entire page dedicated to the concept, available here:
http://en.wikipedia.org/wiki/802.3ad
In order to get it to work, though, you’ll need to make sure your upstream switch/router supports the 802.3ad protocol (most recent managed switches do support it; however, if you have an unmanaged switch, probably not).
— Darien
Yeah, bonding was the initial goal, but we don’t have access to the switches in this case, so it had to be solved at the host level.
We were able to make it work following a modified form of your instructions, FWIW:
In this example, the gateway is 192.168.0.1 and the server’s IPs are 192.168.0.211 and 192.168.0.212 on eth0 and eth1 respectively:
printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
printf “2\tuplink1\n” >> /etc/iproute2/rt_tables
ip route add 192.168.0.211/32 dev eth0 src 192.168.0.211 table uplink0
ip route add default via 192.168.0.1 dev eth0 table uplink0
ip rule add from 192.168.0.211/32 table uplink0
ip rule add to 192.168.0.211/32 table uplink0
ip route add 192.168.0.212/32 dev eth1 src 192.168.0.212 table uplink1
ip route add default via 192.168.0.1 dev eth1 table uplink1
ip rule add from 192.168.0.212/32 table uplink1
ip rule add to 192.168.0.212/32 table uplink1
By the way, in the second paragraph you say that the solution is applicable to cases where the two NICs are on the same network, that’s why I followed your lead on this.
This is extremely helpful. Thank you.
One question about routing in general, is there a way to have subnets share a common default gateway?
Hi Tekknogenius,
If we’re talking about two subnets sharing a common, physical gateway (such as a router), then yes, but that common gateway must have a valid IP address on each subnet. For example, assume you have two subnets:
192.168.0.0/24
192.168.1.0/24
You can then configure your router to have the IPs of:
192.168.0.254 and 192.168.1.254 for use by members of their respective subnets.
Bottom line is that the gateway must have an address in each subnet; otherwise, you would have to figure out some other way to reach the gateway’s IP address (e.g. static ARP entry if on the local LAN)
— Darien
Hi – i have another slightly different goal. My Linux box has an eth0 (connected via a default gw to the internet) and a on-demand ppp0 (gprs) connection. My goal is to programmatically create sockets via both interface at the same time. I think i am stuck setting up the correct routes for this – do you know how to handle this issue?
Hi Johnny,
See my earlier comment as I’ve already addressed this issue:
— Darien
Thanks for your quick reply. I’m not sure failover routing is the correct issue here, as both routes should be up all the time. My app decides on various criteria which connection should be used. I had read som article, whiches in short says: What goes out on one device, should comme in on that again. Is this the right idea? And how can I realize that?
Hi Johnny,
Okay, thanks for the clarification. I had assumed when you mentioned ‘on-demand’, that you somehow wanted to switch from eth0 to ppp0 when eth0 failed. Instead, you want both eth0 and ppp0 alive all the time.
In that case, just replaced ‘eth1’ with ‘ppp0’ in the example and you can still use policy routing to accomplish what you’re looking for.
The real challenge, is if ppp0’s network configuration changes constantly. If the IP address, subnet, and route of ppp0 constantly changes, then you need to somehow have logic to detect when these changes occur and update your policy routes accordingly.
In that case, you’ll probably want to research DHCP notification code. Depending on your Linux distribution, there are scripts which execute when your network changes, which you could use to also update your corresponding policy routes.
— Darien
Great article. This is so simple, yet so hard to find the information. Thanks a lot 🙂
Hey,
Great article, helped me loads compared to the other sites were they just skimm abit on the top.
Thou, having problem adapting it to my setup.
I currently have 2 nic’s on my server that is acting as router/gateway.
1 of the physical nic’s has 5 virtuell interfaces (g1-5), g1 goes to the router itself as it also goes as a webserver. The rest are nated straight to nic2(which has no virtuell interfaces). As i have 4 computer, g2-5 are nated 1:1 towards the inside local ip:s 10.10.1.X.
I have a simple iptables rule to fix the NAT part, and i had it working “sortoff” since everything was working except that i could not reach the servers/pc’s from the outside, they could only talk from the inside->out (torrents etc works flawlessly thou).
Any hints? If you got a mail or something it would be great to get some advice..
Hi DevizioN,
It sounds like you have one-way NAT working and you want bi-directional NAT. From your post, it sounds like g1 goes to the router, as well as nic2 goes directly to the router … is that correct?
Regardless, it sounds like setting up multiple default routes correctly will NOT fix your NAT problem… I would recommend you try and fix your NAT problem first, and then setup the multiple default routes.
To be very specific, with your current NAT configuration, try configuring your server with only ONE upstream gateway and fix your NAT so that it’s working in both directions (insideout).
Once you get a single route working, then add your second upstream gateway. If your bi-directional NAT breaks and no longer works correctly, then let me know and we can troubleshoot further (as configuring multiple default routes may help you).
The general approach is: reduce the complexity of your configuration, verify each piece works first (e.g., NATing, virtual interfaces), and THEN verify the total configuration works. Otherwise, it can be a nightmare to troubleshoot multiple unknown pieces at the same time.
— Darien
Hi, if i have 2 or more subnets
example
10.10.70.0/24
10.10.71.0/24
ip rules add and from are with /32 or /24?
example
ip rule add from 10.10.70.38/32 table admin
ip rule add to 10.10.70.38/32 table admin
ip rule add from 10.10.71.38/32 table admin2
ip rule add to 10.10.71.38/32 table admin2
or
ip rule add from 10.10.70.38/24 table admin
ip rule add to 10.10.70.38/24 table admin
ip rule add from 10.10.71.38/24 table admin2
ip rule add to 10.10.71.38/24 table admin2
I don’t hunderstand why you use all 32 bits.
Thanks
Excellent Post!
I have followed the various bonding solutions available on Linux since I was doing load balancing work in the late 90’s but have never found anything that delivers on bonding two links to maximize bandwidth utilization.
I have a FIOS and Cable link that I’d been trying with bonding modes 5,6 on with various 3com 509’s and netgear 100mbit nics, after following your references here to Astro Shapes pages (also very informative). mii-tool seems to work fine with the nics.. but can’t get them to attach correctly for ifenslave
Given I’m interested in maximizing traffic on the links I wonder if anyone has rules that will monitor and direct traffic based on load on the individual nics?
Thanks
Thanks for writing a very useful article and subsequently answering so many questions. Here’s another 🙂
Supposing you have one or more aliases on the public interface (e.g., eth0:0) would you repeat all of the above steps, substituting eth0:0 for eth0?
Yes. Using eth0:0 should work.
Thanks for a great article. This looks like kinda what I need. However, my situation is a bit different (aren’t they all 🙂 ). Anyway, I am basically, trying to setup openVPN on my Ubuntu 8.10 server. I got 2 NICs in my server. eth0 and eth1. eth0 is the external NIC while eth1 is the internal one (the LAN. eth0 is the default route. VPN connects through eth0 and I can ping my server and I can ping eth1, however I can’t ping anything behind eth1 (which is my local LAN). From the server console itself I can ping my internal LAN. I think the problem lies with traffice coming via eth0 and since the default route is eth0 it only goes through eth0. So if I ping anything which is behind eth1 it is routed via eth0 and not eth1. Is it possible to route all VPN traffic meant for eth1 through eth1 even though it’s not the default route?
Does this make any sense or am I making you guys as confused as myself?
Thanks for your help in advance.
Hi Mazzer,
1) As a sanity check, are you able to ping your eth1 network if you TURN OFF your VPN? I just want to make sure that the problem isn’t more fundamental.
2) On your server, once you have your VPN setup, you actually have THREE different interfaces: eth0, eth1, and (probably something like vpn0 — check your ‘ifconfig -a’ for details). Essentially, you have a third ‘virtual’ interface, representing the effective IP address of your server, once connected on the *inside* of your VPN network.
Here’s what I need to know:
– The IP address and netmask of your eth1 network.
– The IP address and netmask of your *vpn0* network (see above for what I mean).
Specifically, I believe that perhaps your eth1 network and your vpn0 network may have conflicting IP addressing, which is causing you to lose connectivity to your eth1 network once a VPN is established. However, I need more information in order to accurately say that this is the problem.
— Darien
I used the same kinda configuring (but created two tables instead of one) one per interface. This makes sure i get the response to through the right interface.
# Assign IP address on the interfaces eth0 and eth1
ip route add 10.1.0.0 dev eth0 src 10.1.0.250
ip route add 10.0.0.0 dev eth1 src 10.0.0.250
# IP route and default gateway for eth0 in the table 100
ip route add 10.1.0.0 dev eth0 src 10.1.0.250 table 100
ip route add default via 10.1.0.1 table 100
# IP route and default gateway for eth1 in the table 101
ip route add 10.0.0.0 dev eth1 src 10.0.0.250 table 101
ip route add default via 10.0.0.1 table 101
# Assign the rule to corresponding particular table
ip rule add from 10.1.0.250 table 100
ip rule add from 10.0.0.250 table 101
ip route flush cache
Now, i can ping the box to the corresponding box from behind the
gateway. Everything works fine according to my requirement.
Now if i use the ping utility (from the box) with a destination address not in 10.1.X.X or in 10.0.X.X network using both the interfaces (or the
ip-address) it works
ping -I eth0 some-address (works)
ping -I eth1 some-address (works)
ping some-address (not working) with “network is unreachable” as the error
Now this force me to use the default route on the main table which i
thought was not neccessary using below command
ip route add default via 10.1.0.1.
Since with out any interface or ip-address explicity given, ping
should have chosen the source ip-address as either 10.1.0.250 or
10.0.0.250. If so , rule should have hit in routing table 100 or table
101. But just wondering why it is hiting the main routing table. This
forces me to have a two gateway programming option for each interface
and another default gateway to select among two. Not sure why..
Also how do we use the same concept in the DHCP enabled network. DHCP client running on both the interfaces. DHCP clients always deals with the main routing table (instead of the custom created routing table).
Handling DHCP leases is tough. You’ll need to use a DHCP client that supports custom extensions, so that you can call a script (which you create) that will setup the appropriate policy routes.
What does your ‘netstat -anr’ and ‘ip rule show’ look like?
Hi Darien,
Thank you for your quick reply. I was away from my computer for a while so couldn’t reply.
As per your questions, I’ll answer the ones I can right now.
1) I can ping eth1.
2) I know what you mean about the 3rd virtual interface. Sorry for not providing more information. I do have one and it’s a tun interface (tun0). I’m using routing and not bridging.
The IP address of eth1 is 10.10.2.102, netmask is 255.255.255.0 so thats the 10.10.2.0 subnet.
The details about tun0 I’ll get for you when I go home tonight.
I think the problem lies between the virtual Interface and eth1. From the VPN server I can ping both eth1 and eth0 and everything behind those interface. I can ping everything behind the eth1 subnet (internal network) and I can ping everyhting behind the eth0 subnet (the internet). When I’m connect via VPN to the server. I can ping both eth0 and eth1 but can’t ping anything behind eth1. Once connected via VPN, I can SSH into the VPN server (through the VPN connection) and one connected to the server console. I can ping everything behind eth1. I guess thats no surprise as I’m am then doing the pinging from the console. So as I said, I think the problem lies with the virtual interface having the default route of eth0 and so the pings arte going through eth0 even though they meant for eth1.
Phew, I can’t believe I wrote all that down :). Thanks for your help though.
Also Just so you know, I’m using OpenVPN as my VPN server and I’ve setup the redirect gateway option as well.
Got the details for the tun0 interface.
IP Address: 10.30.2.6
Subnet Mask: 255.255.255.252
Default Gateway: 10.30.2.5
DHCP Server: 10.30.2.5
DNS Servers: 10.10.2.11
The 10.10.2.0 subnet is the internal eth1 network so looks like the VPN server is pushing the DNS server to the client correctly.
Good article.
Forgive my ignorance, but what’s the purpose of the “to” rule, ie
ip rule add to 10.10.70.38/32 table admin
that would be used only from incoming packets, and even in that case it doesn’t seem very useful. But I’m sure I’m missing something obvious.
Thank you.
madinc,
The “to” rule is to tell the tcp/ip stack that for all packets where the destination is 10.10.70.38/32, then route those packets to the “admin” interface. This is important for STATEFUL flow processing. When Linux constructs the appropriate reply (e.g., a TCP ACK), it will know which policy route to use, when sending the reply. Otherwise, without this rule, you’ll find that your ACKs will get sent using the default route in the main table.
Sorry for being slow to understand, but for ACK reply packets the “to” (destination IP) will NOT be 10.10.70.38, but rahter the IP of the host the ACK is directed to.
So if you want to use a specific rule to route those packets out, you probably want a rule that matches on *from* 10.10.70.38, rather than “to”.
I still fail to see how that “to” rule can match anything other than incoming packets.
Thank you!
Hi madinc,
The “to” rule is primarily important when BOTH of your ethernet interfaces are on the SAME SUBNET. If both interfaces are on separate subnets with NO possibility of getting overlapping traffic, then the “to” rule is less of an issue.
However, when both interfaces are on the same subnet, then the “to” rule tells Linux which packets to ignore or listen to if each interface is getting duplicate traffic.
Just to clarify, when I mean SAME SUBNET, I mean that both ethernet interfaces are connected to the same upstream hub/switch/router, where both ethernet interfaces receive copies of all network traffic destined to any of the interfaces.
When both ethernet interfaces are on the same subnet, only the one whose IP address is the destination IP address of the incoming packet will get it.
It’s like having many hosts in a LAN: only the destination host will reply to ARP and thus get the packet, despite all other hosts being in the same subnet.
Forgive me, but I still don’t see where the “to” rule comes into play 🙂
Thank you.
Hm, on second thought…with Linux’s weak host model, both interfaces will reply. So yeah, now I see where the “to” can be useful.
Thank you and sorry for the noise.
No problem; glad to help. In general, when getting real answers, I’d recommend actually testing against an actual Linux system, since a lot of these nuances could change, depending on which kernel patches (from which Linux distributions) are loaded on your test machine.
On numerous occasions, I’ve found that “what should happen” is largely different than “what really happens”.
I tried using the steps that you used to configure Multiple Routes and I ran into an error when I ran the command “ip route add default vi 10.141.50.50 dev eth2 table ccsadmin”. The error that I received was “error: either “to” is duplicate, or “10.141.50.50” is a garbage. I’m not sure what steps do I need to take to solve this problem that I’m having. I’m running CentOS 4.7 kernel version 2.6.9-78
Umm, I think you mis-typed the command:
“ip route add default vi 10.141.50.50 dev eth2 table ccsadmin”
Should be:
“ip route add default via 10.141.50.50 dev eth2 table ccsadmin”
Specifically, the “vi” should be “via”.
Hmmm…
I keep getting:
[root@bjwww001 ~]# ip route add 192.168.20.0/24 dev eth1 src 192.168.20.2 table hkvpn
RTNETLINK answers: Invalid argument
All my ifcfgXX files are correct and have broadcast and network IP’s defined too….
Did you enable “policy routing” in the Linux kernel and install the iproute2 utilities? (Both were mentioned above in the article.)
iproute yes… policy routing – will have to check. This is a stock CentOS 5.4 64 bit. WIll I need to recompile the kernel or is there a kernel flag I can set in sysctl?
Policy routing requires a kernel recompilation, if it’s not enabled by default. There’s no way to dynamically enable/disable it via sysctl.
Not sure about CentOS, but many Linux distributions will store a copy of the actual Linux kernel config in /proc/config.gz, so you could try inspecting that kernel config (if it exists) to see if Policy Routing has been defined. Otherwise, you’ll have to do a recompile.
OK. Yes the .config is included. Have’t recompiled a kernel for probably 10 years now. Oh well – like riding a bike….
This works too for CentOS with no kernel recompile!
http://www.sysadminsjourney.com/content/2009/04/15/doing-simple-source-policy-routing-centos
Worked great for ssh, but borked my apache server. I don’t know why but requests arriving on one device to the apache server will be sent out on the other one. And with the ip rule will try to go the wrong way. Maybe its an apache bug (or feature?)
Do you have Apache listening on all interfaces or just one interface? You may need to adjust this setting to get your desired effect.
http://httpd.apache.org/docs/2.2/bind.html
Since you were using Gentoo:
How do you setup the IP address in the “admin” routing table during boot? How did your modify your “/etc/conf.d/net.eth0”-file?
The default gateway can be added with “routes_eth0=( “default via 10.10.70.254 table admin” )
The rules can be added in the “postup” section of the /etc/conf.d/net; but I simply cannot bring Gentoo to add the IP address in the other table. It either adds it in the default table; or throws a syntax error.
Instead of trying to figure out the proper way to add the routing table commands to the network startup scripts, just add them to your /etc/rc.local script (or equivalent). Essentially, you want to use the same absolute commands as listed in the examples (in the article) and write them to a script that will run after *all* automatic networking scripts are ran. On most systems, commands written to /etc/rc.local are literally the last commands executed before the system finished bootup and presents a login prompt to the user.
That would be “/etc/conf.d/local.start” on a Gentoo system.
While it does work, of course, there is a problem if you restart the interface with an “/etc/init.d/net.eth0 restart”, since the routing table will be empty afterwards.
Alas, at least it works now. Thanks for pointing me in the right direction.
Replying to my own comment:
Using OpenRC with baselayout2 the /etc/conf.d/net.eth0 script would look like this:
config_eth0=”192.168.1.100 netmask 255.255.255.0 broadcast 192.168.1.255″
routes_eth0=”default via 192.168.1.1
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.150 table dmz
default via 192.168.1.1 dev eth0 table dmz”
dns_domain_eth0=”localdomain.local”
dns_servers_eth0=”dns.ip.1 dns.ip.2″
metric_eth0=”0″
Modify to your own pleasure.
Thanks so much for this! I had no idea, until I watched it with tcpdump, that a packet coming in on one interface would get an (attempted) response on another.
The fix you give worked like a charm.
–Al Evans
Wow, this worked great! Really cool. Does anybody know if something like this can be done under Windows?
Hi,
Thanks for the tutorial, it helped a lot! One thing, is there a way to make the routing tables persistent? Or will the newly configured tables still be there after a system reboot?
Regards,
mdewet
Hi mdewet,
The policy routing settings should persist across reboots. If, for whatever reason, they do not, you can usually add these settings to /etc/rc.local (or equivalent), so that these settings are applied at the very end of system boot.
— Darien
Great article. Much appreciated!!!
Hi Darien,
First of all great post.
I just have one question. Is it possible to do something similar but with two interfaces connected to the same network? For instance two wireless cards connected to the same access point.
Hi rodmar,
Yes, the same technique can be used on 2 wireless NICs connected to the same access point — Linux treats those interfaces the same when performing policy routing.
— Darien
Thanks for the tutorial. Does this also work with configuring multiple default routes on vlan interfaces?
Hi,
Yes, this technique can be applied to vlan interfaces as well, because Linux treats those interfaces exactly the same as physical interfaces — just substitute eth0/eth1 with the corresponding vlan interfaces you want to perform policy routing on.
— Darien
Bless you, O Darien, for this gem on a subject which has stumped not only me but all who I have hit up for the solution, til now!
Bless you, O Darien, for this gem on a subject which has stumped not only me but many who I have hit up for the solution!
Classful routing hasn’t been used since 1994. The fact that the mask is included means it is a classless network. So stop saying ‘class C’, just because the Class C default mask was 255.255.255.0. It is simply a /24 subnet.
Done.
Darien, Thnak you for this info..
Do i have to remove the gateway in the interface file for this to work???
Hi Will,
No, you don’t need to remove the gateway in the interface file for this to work. Essentially, you want to perform the policy route configuration *after* basic networking (including setting the default gateway) has been done. This is why I recommend setting the policy route in /etc/rc.local, so that it is the *last* thing initialized, which guarantees the settings will not conflict with another daemon upon boot up.
— Darien
Can you look at this and see what I’m doing wrong??? I would gladly pay you for your time.
We have 2 NIC cards and when I try to set the eth1 gatway we lose SHH to the server.
ETH0 is for (SSH<WEB<ECT.)
ETH1 is for SIP traffic only
ifconfig –a
eth0 Link encap:Ethernet HWaddr 00:1c:23:d0:0b:2c
inet addr:192.168.1.69 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::21c:23ff:fed0:b2c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6215 errors:0 dropped:0 overruns:0 frame:0
TX packets:59877 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:908397 (887.1 KiB) TX bytes:13233015 (12.6 MiB)
Interrupt:16 Memory:f8000000-f8012100
eth1 Link encap:Ethernet HWaddr 00:1c:23:d0:0b:2e
inet addr:192.168.3.70 Bcast:192.168.3.255 Mask:255.255.255.0
inet6 addr: fe80::21c:23ff:fed0:b2e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:271 errors:0 dropped:0 overruns:0 frame:0
TX packets:123 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:181558 (177.3 KiB) TX bytes:13875 (13.5 KiB)
Interrupt:16 Memory:f4000000-f4012100
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:197 errors:0 dropped:0 overruns:0 frame:0
TX packets:197 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:68135 (66.5 KiB) TX bytes:68135 (66.5 KiB)
vi interfaces
auto lo
iface lo inet loopback
# The primary network interface[m
auto eth0
iface eth0 inet static
address 192.168.1.69
netmask 255.255.255.0
gateway 192.168.1.10
auto eth1
iface eth1 inet static
address 192.168.3.70
netmask 255.255.255.0
#gateway 192.168.3.11 —-When I try to use this line it locks me out oh SSH
ip rule show
0: from all lookup local
32759: from 192.168.3.70 lookup sip
32760: from all to 192.168.3.70 lookup sip
32763: from all to 192.168.1.69 lookup admin
32764: from 192.168.1.69 lookup admin
32766: from all lookup main
32767: from all lookup default
ip route show
192.168.3.0/24 dev eth1 proto kernel scope link src 192.168.3.70
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.69
gla
Hi Will,
Your last post looks like some information was cut off, but I think I have enough information to help you.
1) In /etc/interfaces, do NOT define the gateway for eth1. You are correct that this declaration will cause problems. So, remove the line:
“gateway 192.168.3.11” completely from /etc/interfaces.
2) It looks like you’re trying to define TWO (2) policy tables: “sip” and “admin”. You do NOT need to do this. Please remove *both* “sip” and “admin” tables, and clear out all the custom “ip rule” and “ip route” commands you have defined.
3) Once you’ve down that, reboot your system and verify that there are NO custom “ip rule show” and “ip route show” commands listed, and that only the eth0 default gateway is showing up in “netstat -anr”.
4) THEN, type the following commands:
echo “1 admin” >> /etc/iproute2/rt_tables
ip route add 192.168.3.0/24 dev eth1 src 192.168.3.70 table admin
ip route add default via 192.168.3.11 dev eth0 table admin
ip rule add from 192.168.3.70/32 table admin
ip rule add to 192.168.3.70/32 table admin
ip route flush cache
Hope this helps,
— Darien
Great article – worked like a charm.
Thanks very much!
Thanks very much for the excellent info!
However there was an error for me for the route setting for the default gateway, which I tried to set to same as in the /etc/network/interfaces file, but from an online IP-calculator I found that the gateway should be the first usable host address from the range of machines on the sub network. After changing to that IP, it worked like a charm.
Many thanks for this I have been trying to consolidate some servers and I was combining our Proxy box with our openvpn box for a different subnet. I decided to use Untangle for the proxy as it was a quick easy set up. However, I knew that I would have to run my own instance of openVPN as Untangle only does routed VPNs and I needed a bridged VPN. All setup fine, but then when I pinged to the openVPN bridged interface through a second router from a PC on my LAN which is the one subnet being proxied I was getting no response, because it was killing them, I looked for a way to route the traffic back out the same interface it came in one for a couple of weeks and even posted questions about this at some forums, but your web page here finally provided me the information I needed to do it. I hope you don’t mind I have resolved the questions I asked on the forums and linked back to your article. Many thanks.
Darien – thanks this is very helpful!
nobodyfamous and howard refer to same subnet and same gatway with 2 (or multiple) NICs. But would anything change in your precription if I want 2 gateways (10.0.0.1 and 10.0.0.2) with 2 NICS (10.0.0.11 and 10.0.0.22) on the same subnet. (This is different from link aggregation).
To stay with your example – what if eth0 has ip 192.168.7.127
and gateway 192.168.7.1
oops I shouls have said:
To stay with your example – what if eth0 has ip 192.168.7.127
and gateway 192.168.7.2
Hi Dave,
Take a look at the comments posted right after nobodyfamous:
So, let’s assume:
eth0 – 192.168.7.127 netmask 255.255.255.0
eth0’s gateway is: 192.168.7.2
eth1 – 192.168.7.126 netmask 255.255.255.0
eth1’s gateway is: 192.168.7.1
Here are the corresponding commands:
printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
printf “2\tuplink1\n” >> /etc/iproute2/rt_tables
ip route add 192.168.7.127/32 dev eth0 src 192.168.7.127 table uplink0
ip route add default via 192.168.7.2 dev eth0 table uplink0
ip rule add from 192.168.7.127/32 table uplink0
ip rule add to 192.168.7.127/32 table uplink0
ip route add 192.168.7.126/32 dev eth1 src 192.168.7.126 table uplink1
ip route add default via 192.168.7.1 dev eth1 table uplink1
ip rule add from 192.168.7.126/32 table uplink1
ip rule add to 192.168.7.126/32 table uplink1
I have set it up as stated. It boots fine but I am unable to ping 10.0.0.211 (eth1) from another machine on the subnet – but I can ping 10.0.0.201 (eth0). From the machine itself I can ping 10.0.0.211. Here is some output:
oft201:~# ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
oft201:~# vi /etc/rc.local
oft201:~# /etc/rc.local
oft201:~# ip rule
0: from all lookup local
32762: from all to 10.0.0.211 lookup uplink1
32763: from 10.0.0.211 lookup uplink1
32764: from all to 10.0.0.201 lookup uplink0
32765: from 10.0.0.201 lookup uplink0
32766: from all lookup main
32767: from all lookup default
oft201:~# netstat -nra
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 eth0
What gives? [I am assuming that policy routing must be enabled in the latest stable debian kernel:
Linux oft201 2.6.26-2-686-bigmem #1 SMP Thu Sep 16 20:29:07 UTC 2010 i686 GNU/Linux]
Okay, can you paste the contents of your /etc/rc.local file?
Also, I assume you entered “ip route flush cache” and still ran into problems?
I am able to get it to work if the line:
ip route add 192.168.7.127/32 dev eth0 src 192.168.7.127 table uplink0
is changed to
ip route add 192.168.7.0/24 dev eth0 src 192.168.7.127 table uplink0
and the line
ip route add 192.168.7.126/32 dev eth1 src 192.168.7.126 table uplink1
is changed to
ip route add 192.168.7.0/24 dev eth1 src 192.168.7.126 table uplink1
Okay, glad it’s working for you.
My rc.local with what I beleive to be the incorrect lines commented out:
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will “exit 0” on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
#ip route add 10.0.0.201/32 dev eth0 src 10.0.0.201 table uplink0
ip route add 10.0.0.0/24 dev eth0 src 10.0.0.201 table uplink0
ip route add default via 10.0.0.1 dev eth0 table uplink0
ip rule add from 10.0.0.201/32 table uplink0
ip rule add to 10.0.0.201/32 table uplink0
#ip route add 10.0.0.211/32 dev eth1 src 10.0.0.211 table uplink1
ip route add 10.0.0.0/24 dev eth1 src 10.0.0.211 table uplink1
ip route add default via 10.0.0.2 dev eth1 table uplink1
ip rule add from 10.0.0.211/32 table uplink1
ip rule add to 10.0.0.211/32 table uplink1
exit 0
Turns out that the above settings work but not as expected.
If I run ‘ifdown eth1’ and then try
ping or ssh 10.0.1.211 (the interface associated with eth1) – I can still get through (and vice versa). This may be seen as a feature rather than a bug – but I am spooked by it because I dont understand it. I have abandoned the quest of dual gateway on same subnet. There is no compelling reason for having this anyway. I can live with the original prescription of Darien – which works as expected. (I used the latest debian lenny kernel and did not need to make any special kernel compile to get Dariens prescription working). Thanks again.
Would have been a great article if it had explained where one wrote the admin routing information:
ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
ip route add default via 10.10.70.254 dev eth0 table admin
Which file is this placed into? I don’t know. The article does not say.
Therefore, I am back to googgeling once more for another site. But at least I know I should Google for policy routing. Thus a partial thank-you.
PS. I am using SLES 11. Maybe its obvious on gentoo.
why: Read the corresponding comments for your answer. Specifically, this one:
Better yet, just search the page for “rc.local”. What you’re asking for is very generic… you just want a way to have this commands auto-executed upon boot up.
If you do a google search for “suse linux rc.local”, you get this as a top hit:
http://linux.derkeiler.com/Mailing-Lists/SuSE/2003-12/1428.html
Which states: /etc/init.d/boot.local is the equivalent file of rc.local in SLES.
Sorry, I have realised that these lines are actual commands.
ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
ip route add default via 10.10.70.254 dev eth0 table admin
Please disregard my initial posts.
Hi,
I have had a hard time following the post, and am unsure its what I have to do. Would you be able to have a look at this scenario and add some helpers for it?
Both interfaces are on different VLANs (Should not make an once of difference). The default route is the one that goes to the commodity Internet, while the 10/8 destinations are for the internal network.
eth0 IP 10.254.60.90/22
default 10.254.60.74 255.255.252.0 eth0
10.0.0.0 10.254.60.51 255.0.0.0 eth0
eth1 IP 10.254.41.13/22
default 10.254.40.74 255.255.252.0 eth1
10.0.0.0 10.254.40.51 255.0.0.0 eth1
I’ll take a look, but why are you defining two different netmasks for each interface? That makes no sense.
If you’re going to define a full Class-A /8, then you don’t need to explicitly define a /22 for the upstream Internet gateway. The /22 serves no purpose in this context.
I’ll generate some rules that show you what I mean…
I think I see what you’re trying to do… each interface has two _virtual_ IPs assigned to it:
eth0: 10.254.60.90, 10.254.60.51
eth1: 10.254.41.13, 10.254.40.51
I’ll adjust accordingly…
Actually, can you confirm each interface has two virtual IPs? The way you wrote the initial IP information is confusing…
These are both IPs plumbs on real NICs, not virtual NIC (not eth0:1 etc), and neither are these bonded in anyway.
Hi Darie,
Apologies for the confusion.
The 10/8 reference was only mean to refer to the second route below for all traffic to the 10.0.0.0 network, and this understandingly caused confusion.
All networks use /22. The default route is for all traffic other than the 10.0.0.0/22 network, (which is really the public Internet and other private nets 192.168.n.n. and 172.16.n.n networks).
The current system has one NIC eth0. Its routing table as defined is :
default 10.254.64.74 255.255.252.0 –
10.0.0.0 10.254.64.51 255.0.0.0 –
I have to plumb a new eth1 onto it on another VLAN and keep the traffic separate. (I had never heard of priority routing until now.) The eth1 has to send the default Internet traffic via the router 10.254.40.74 on eth1, and the rest back to the internal LAN via 10.254.40.51 on eth1.
^Hi Darie,^Hi Darien,
Okay, ignore my previous post… will provide documentation based on this new information, shortly.
FYI
These are all routers somewhere on the network:
10.254.64.74
10.254.64.51
10.254.40.74
10.254.40.51
Okay, the real issue is that you have a single IP address defined for each physical interface (eth0 and eth1), and yet you have multiple _overlapping_ netmasks defined for the routes in each interface.
For example, eth0:
Class-A /8 – respond directly
/22 – out to gateway
This makes the issue not as easy to solve, because normally you would assign the same netmask for each interface — or you could define multiple virtual interfaces with different IP addresses and netmask combinations.
In any event, I believe the following commands will work without having to define any virtual interfaces, but I have never tested this scenario, so you’ll have to try it out and see for yourself:
Here are the corresponding commands:
printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
printf “2\tuplink1\n” >> /etc/iproute2/rt_tables
printf “3\tuplink2\n” >> /etc/iproute2/rt_tables
printf “4\tuplink3\n” >> /etc/iproute2/rt_tables
ip route add 10.254.60.51/22 dev eth0 src 10.254.60.51 table uplink0
ip route add default via 10.254.60.74 dev eth0 table uplink0
ip rule add from 10.254.60.51/22 table uplink0
ip rule add to 10.254.60.51/22 table uplink0
ip route add 10.254.60.51/8 dev eth0 src 10.254.60.51 table uplink1
ip rule add from 10.254.60.51/8 table uplink1
ip rule add to 10.254.60.51/8 table uplink1
ip route add 10.254.40.51/22 dev eth1 src 10.254.40.51 table uplink2
ip route add default via 10.254.40.74 dev eth1 table uplink2
ip rule add from 10.254.40.51/22 table uplink2
ip rule add to 10.254.40.51/22 table uplink2
ip route add 10.254.40.51/8 dev eth1 src 10.254.40.51 table uplink3
ip rule add from 10.254.40.51/8 table uplink3
ip rule add to 10.254.40.51/8 table uplink3
For example, eth0:
Class-A /8 – respond directly
/22 – out to gateway
Yes, this is exactly this. There is a multitude of overlapping netmasks here.
I shall try all this on a test server, and then report back here. This’ll be later tomorrow, as its nearly home time here.
Okay, here’s the set of rules for this scenario. I assume that each interface actually has _two_ virtual IP addresses. In this case, you would be defining “eth0:0 and eth0:1” as the virtual interfaces for eth0 and “eth1:0 and eth1:1” for the virtual interfaces for eth1. If you’re not sure what this means or how to set it up, the SLES documentation about virtual interfaces should help.
Regardless, instead of thinking about two interfaces, think about this problem in terms of _four_ virtual interfaces. If you split the problem based upon each virtual interface, it’s much easier to solve. See below for details.
Assumptions:
eth0:0 – 10.254.60.90 netmask 255.255.252.0
eth0:0’s gateway is: 10.254.60.74
eth0:1 – 10.254.60.51 netmask 255.0.0.0
eth1:0 – 10.254.41.13 netmask 255.255.252.0
eth1:0’s gateway is: 10.254.40.74
eth1:1 – 10.254.40.51 netmask 255.0.0.0
Here are the corresponding commands:
printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
printf “2\tuplink1\n” >> /etc/iproute2/rt_tables
printf “3\tuplink2\n” >> /etc/iproute2/rt_tables
printf “4\tuplink3\n” >> /etc/iproute2/rt_tables
ip route add 10.254.60.90/22 dev eth0:0 src 10.254.60.90 table uplink0
ip route add default via 10.254.60.74 dev eth0:0 table uplink0
ip rule add from 10.254.60.90/22 table uplink0
ip rule add to 10.254.60.90/22 table uplink0
ip route add 10.254.60.51/8 dev eth0:1 src 10.254.60.51 table uplink1
ip rule add from 10.254.60.51/8 table uplink1
ip rule add to 10.254.60.51/8 table uplink1
ip route add 10.254.41.13/22 dev eth1:0 src 10.254.41.13 table uplink2
ip route add default via 10.254.40.74 dev eth1:0 table uplink2
ip rule add from 10.254.41.13/22 table uplink2
ip rule add to 10.254.41.13/22 table uplink2
ip route add 10.254.40.51/8 dev eth1:1 src 10.254.40.51 table uplink3
ip rule add from 10.254.40.51/8 table uplink3
ip rule add to 10.254.40.51/8 table uplink3
Apologies. I;m confused.
Actually, your original thread was so close what I want to do, yet the ip rules you defined only ever mentioned eth0:
# eth0 – 10.10.70.38 netmask 255.255.255.0
# eth0’s gateway is: 10.10.70.254
But never mentioned eth1:
# eth1 – 192.168.7.126 netmask 255.255.255.0
# eth1’s gateway is: 192.168.7.1
Since the eth1 is not mentioned, then I presume that it has to be indirectly decided upon by iproute, but this is where I lost the plot. There are only these entries:
ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
ip route add default via 10.10.70.254 dev eth0 table admin
ip rule add from 10.10.70.38/32 table admin
ip rule add to 10.10.70.38/32 table admin
Nor does it say what to add or remove for the /etc/sysconfig/network/routes table, which currently has some routing information in it
Anyway, cheers for your time.
The default route for anything other than that subnet specified in the table is that which is set for the network. In other words: if those rules don’t apply to the packets, they head on out eth1 towards 192.168.7.1 (unless the packets came from the 7.x network, in which case they would head on over to the machine that made the request.)
This may be an old post, but wonderfully explains how to properly set up the routes. I’m running Ubuntu 10.04. One note, if it’s not clear, I defined a gateway (192.168.1.1) for eth0 as my default in the /etc/network/interfaces file. For eth1, I did not define a gateway, but rather set up the route as noted above. Again, many thanks.
Excellent article. It served most of my problem, specially where multiple vlan interfaces setup for different network.
Splendid post!! Thanks =))
Thanks.
This is really great. Finally got my bridged virtual VBOX server environment connected to the internet through a separate nic, leaving my primary nic for the physical ubuntu server access only.
This makes my virtual business environment finally completely safe and separate from my physical private environment.
Thanks a lot !
(If someone wants the setup details, leave a reply and I will post them.)
http://reddragon-linux.blogspot.com/2011/05/linux-internet-gateway-server-setup.html
Two years later, and this is the best post i found to accomplish the job without need to learn all about iptables, routes, rules, etc of a linux box (but still is necesary ;)).
Thank you very much. You make my day a happy day 😉
Thank you Thank you Thank you,
I’ve been trying to shovel several vlans across as many openvpn tunnels on a virtual server without success for what felt like eternity, now in 2 minutes flat it actually does stuff.
Thanks again
Man, Thank You !!!!!!
I spent days trying to solve this. This worked right away.
Darien,
Thanks to you for great article and thanks to everyone else for some great reading.
Set this up on BackTrack 5 – 32 bit and it worked fine. Now trying the same thing on BT5 R1 – 64 bit and it’s not playing
I have two distinct networks and never the twain shall meet.
My home network (eth0) is a 192.168.2.0/24 with the gateway as 192.168.2.1 and this connects to the internet
My ‘special’ network (eth1) is a 10.0.0.16/28 with the gateway as 10.0.0.17 and this connects nowhere due to some of the software sitting on the VM’s that live on this network (vm’s are fixed to eth1 only using VMWare Virtual Network Editor). Both nics have static ip’s though it did work with DHCP when I had it running pre 64 bit install.
Here is what i’m getting
have added following to rc.local
ip route add 192.168.2.0/24 dev oth0 src 192.168.2.30 table admin
ip route add default via 192.168.2.1 dev eth0 table admin
ip rule add from 192.168.2.30/32 table admin
ip rule add to 192.168.2.30/32 table admin
ip route flush cache
ip rule show
0: from all lookup local
32764: from all to 192.168.2.30 lookup admin
32765: from 192.168.2.30 lookup admin
32766: from all lookup main
32767: from all lookup default
ip route show
default via 192.168.2.1 dev eth0 metric 100
default via 10.0.0.17 dev eth1 metric 100
10.0.0.16/28 dev eth1 proto kernel scope link src 10.0.0.20
192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.30
netstat -anr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 eth0
0.0.0.0 10.0.0.17 0.0.0.0 UG 0 0 0 eth1
10.0.0.16 0.0.0.0 255.255.255.240 U 0 0 0 eth1
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
With eth1 enabled, I can’t access the internet or if I use ‘route del default gw 10.0.0.17’ it will let me connect to the internet until next time I reboot. I had it working before and am now baffled as to why it’s playing silly buggers. Have I missed something obvious above or can you think of something else I can try?
This system is a training rig and I need to be able to access the internet from the main machine whilst using the VM’s on their special network.
Here’s your problem:
“netstat -anr
…
0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 eth0
…”
You’ve _already_ specified a default route in the _admin_ table for eth0; as such, you should NOT specify a default route for eth0 in the _main_ table.
To correct this, remove the following command [FLAGGED] from your list:
“ip route show
default via 192.168.2.1 dev eth0 metric 100 <— [REMOVE THIS ONE]
default via 10.0.0.17 dev eth1 metric 100
10.0.0.16/28 dev eth1 proto kernel scope link src 10.0.0.20
192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.30"
Thanks! This article really helped me.
Thanks man ! you saved my life 🙂 i was looking this information from a long time ago.
cheers!
hi, and thank you for your post! i have a question….
eth0 – ip 10,60,90,45 gw 10,60,90,1
eth1 – ip 10,60,91,45 gw 10,60,91,1
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.60.90.0 * 255.255.255.0 U 0 0 0 eth0
10.60.91.0 * 255.255.255.0 U 0 0 0 eth1
link-local * 255.255.0.0 U 0 0 0 eth0
loopback * 255.0.0.0 U 0 0 0 lo
default 10.60.90.1 0.0.0.0 UG 0 0 0 eth0
if i need that a machine on 69,50,20,xx subnet can connect to 10,60,91,45 i add a new route
69,51,20,0 10,60,91,1 255,255,255,0 UG 0 0 0 eth1
but if i don’t know all the subnets or more easy, how can i do to all traffic that goes through an interface is returned by the same interface.
Hi sun,
Take a look at the comments posted right after nobodyfamous:
So, let’s assume:
eth0 – 10.60.90.45 netmask 255.255.255.0
eth0′s gateway is: 10.60.90.1
eth1 – 10.60.91.45 netmask 255.255.255.0
eth1′s gateway is: 10.60.91.1
Here are the corresponding commands:
printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
printf “2\tuplink1\n” >> /etc/iproute2/rt_tables
ip route add 10.60.90.0/24 dev eth0 src 10.60.90.45 table uplink0
ip route add default via 10.60.90.1 dev eth0 table uplink0
ip rule add from 10.60.90.45/32 table uplink0
ip rule add to 10.60.90.45/32 table uplink0
ip route add 10.60.91.0/24 dev eth1 src 10.60.91.45 table uplink1
ip route add default via 10.60.91.1 dev eth1 table uplink1
ip rule add from 10.60.91.45/32 table uplink1
ip rule add to 10.60.91.45/32 table uplink1
ip route flush
THANK YOU
Thank you! Your page made it so simple to fix my routing dilemma! I had two NICs on two separate dmz networks and needed to keep traffic on the the same NIC it started on. And have a default gateway for each NIC. This helped! Great Work!
Four years on, this is still the clearest guide to this. It just works.
Thanks.
Thanks for the article.
Unfortunately it does not work in my setup.
How can I make sure my kernel has support for policy routing?
I use Proxmox Virtualization Environment (http://pve.proxmox.com) with kernel 2.6.32-6-pve.
My setup shows:
vserver3:~# ip rule show
0: from all lookup local
32764: from all to 192.44.37.62 lookup vmbr1
32765: from 192.44.37.62 lookup vmbr1
32766: from all lookup main
32767: from all lookup default
vserver3:~# ip route show
192.168.20.0/24 dev vmbr101 proto kernel scope link src 192.168.20.1
192.44.37.0/24 dev vmbr1 proto kernel scope link src 192.44.37.62
192.168.19.0/24 dev vmbr100 proto kernel scope link src 192.168.19.1
153.96.8.0/24 dev vmbr0 proto kernel scope link src 153.96.8.76
default via 153.96.8.1 dev vmbr0
vserver3:~# ip route show table vmbr1
192.44.37.0/24 dev vmbr1 scope link src 192.44.37.62
default via 192.44.37.101 dev vmbr1
I have vmbr100 connected via iptables NAT rules to vmbr0
and vmbr101 via iptables NAT rules to vmbr1.
Connectivity of vmbr0 and vmbr100 works.
Connectivity of vmbr1 and vmbr101 is limited to network 192.44.37.0/24.
Any ideas? Thanks.
Hi Birger,
“vmbr*” interfaces are not standard network interfaces (http://pve.proxmox.com/wiki/Vlans) — they are bridging interfaces, which is why policy routing does NOT work properly.
In order to use policy routing, you need to assign one or more NICs (eth*) interfaces to a bridging interface (vmbr*), and then assign the routing to the attached eth* interface, accordingly.
Hi Darien,
With respect to multiple default routes on OpenBSD: the virtual tables are primarily meant to create separate logical networks over top of a common base network. For instance you could create a private network for contractors which would keep them logically separated from your employees.
However within a routing table, OpenBSD actually supports equal-cost multipath route entries. The kernel will choose a path based on the result of a hash and send the packets on their way. It’s really easy to setup and does not require the complexity of virtual tables. Details are in the OpenBSD FAQ (http://www.openbsd.org/faq/faq6.html#Multipath).
Thanks for the feedback.
Special Thanks to Darien for the very valuable blog post.
Everything works as expected. We tested the configuration with the command “ping -I vmbr1 …”, which gave an error “Destination host unreachable”. Now we found that ping is buggy.
“ping -I 192.44.37.62 …” works as expected. “wget -q -O – -bind-address=192.44.37.62 http://whatismyip.org” works also.
It does work with bridging interfaces and it does work in combination with NAT.
Okay, glad to hear it works with bridging interfaces… I wasn’t 100% sure of this, as I had never tested it before.
thx for this. you made my day. if you got 2 nics make sure you add two route tables
# echo “1 uplink0” >> /etc/iproute2/rt_tables
# echo “2 uplink1” >> /etc/iproute2/rt_tables
# ip route add 192.168.0.0/24 dev eth0 src 192.168.0.101 table uplink0
# ip route add default via 192.168.0.1 dev eth0 table uplink0
# ip rule add from 192.168.0.101/32 table uplink0
# ip rule add to 192.168.0.101/32 table uplink0
# ip route add xxx.xxx.xxx.0/24 dev eth3 src xxx.xxx.xxx.219 table uplink1
# ip route add default via xxx.xxx.xxx.1 dev eth3 table uplink1
# ip rule add from xxx.xxx.xxx.219/32 table uplink1
# ip rule add to xxx.xxx.xxx.219/32 table uplink1
(xxx.xxx.xxx. replace with your settings :P)
to make it permanent under gentoo you can use
# nano /etc/local.d/network.start (http://de.gentoo-wiki.com/wiki/Local.d)
(add #!/bin/sh & the ip xxx commandos; check with rc-update show if the local is added to the default level)
# chmod +x /etc/local.d/network.start
have fun and enjoy!
Hi. Great post and the comment stream is going strong…
I used your post to set up my system.
I have a gateway box with 2 NICs.
One nic sits on the internal network (eth1; 192.168.0.7), the other nic (eth0; 110.xxx.xxx.173) connected to a 10 MBit pipe used for incoming VPN.
There is an ADSL router (192.168.0.1) sitting on the network as well…
Everyone on the network uses 192.168.0.1 as their gateway.
I want people to be able to set 192.168.0.7 as their gateway if they have special speed needs.
So what happens is that when the user’s (say user 192.168.0.10) gateway is set to 192.168.0.7 the traffic still goes via 192.168.0.1… whereas I want it to go through 110.xxx.xxx.173…
Host Loss% Snt Last Avg Best Wrst StDev
1. gateway1.e3(192.168.0.7) 0.0% 4 0.1 0.2 0.1 0.2 0.0
2. firewall.e3(192.168.0.1) 0.0% 4 0.4 0.5 0.4 0.5 0.0
3. ras8.nap.net.au 0.0% 3 30.0 146.9 30.0 263.7 165.2
4. g3.cor1.adp.net.au 0.0% 3 280.6 229.0 140.8 280.6 76.8
5. g1.ap.net.au 50.0% 3 155.1 155.1 155.1 155.1 0.0
here are my configs:
/etc/iproute2/rt_tables:
2 gatewayeth1
3 gatewayeth0
/etc/network/interfaces:
# The primary network interface
auto eth1
iface eth1 inet static
address 192.168.0.7
netmask 255.255.255.0
gateway 192.168.0.1
# 10Mbit interface
auto eth0
iface eth0 inet static
address 110.xxx.xxx.174
netmask 255.255.255.252
gateway 110.xxx.xxx.173
/etc/rc.local:
ip route add 110.xxx.xxx.172/30 dev eth0 src 110.xxx.xxx.174 table gatewayeth0
ip route add default via 110.xxx.xxx.173 dev eth0 table gatewayeth0
ip route add 192.168.0.0/24 dev eth1 src 192.168.0.7 table gatewayeth1
ip route add default via 192.168.0.1 dev eth1 table gatewayeth1
ip rule add from 110.xxx.xxx.174/32 table gatewayeth0
ip rule add to 110.xxx.xxx.174/32 table gatewayeth0
ip rule add from 192.168.0.7/32 table gatewayeth1
ip rule add to 192.168.0.7/32 table gatewayeth1
iptables rules:
#! /bin/bash
iptables –flush
#NAT TABLE
iptables -t nat –policy PREROUTING ACCEPT
iptables -t nat –policy POSTROUTING ACCEPT
iptables -t nat –policy OUTPUT ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
#FILTER TABLE
iptables –policy INPUT DROP
iptables –policy OUTPUT DROP
iptables –policy FORWARD DROP
# localhost rules
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
# in chain for eth0 (outward facing telstra port on network 110.xxx.xxx.172/30)
iptables -X IN_RULES_ETH0
iptables -N IN_RULES_ETH0
iptables -A INPUT -i eth0 -j IN_RULES_ETH0
# in chain for eth1 (inward facing internal network 192.168.0.0/24)
iptables -X IN_RULES_ETH1
iptables -N IN_RULES_ETH1
iptables -A INPUT -i eth1 -j IN_RULES_ETH1
# out chain
iptables -X OUT_RULES
iptables -N OUT_RULES
iptables -A OUTPUT -j OUT_RULES
# forward chain for eth1
iptables -X FORWARD_ETH1
iptables -N FORWARD_ETH1
iptables -A FORWARD -j FORWARD_ETH1
# drop stuff for eth0
# IN_RULES_ETH0
iptables -A IN_RULES_ETH0 -s 10.0.0.0/8 -j DROP
iptables -A IN_RULES_ETH0 -s 172.16.0.0/12 -j DROP
iptables -A IN_RULES_ETH0 -s 192.168.0.0/16 -j DROP
iptables -A IN_RULES_ETH0 -s 224.0.0.0/4 -j DROP
iptables -A IN_RULES_ETH0 -s 240.0.0.0/5 -j DROP
iptables -A IN_RULES_ETH0 -s 127.0.0.0/8 -j DROP
iptables -A IN_RULES_ETH0 -s 0.0.0.0/8 -j DROP
iptables -A IN_RULES_ETH0 -d 255.255.255.255 -j DROP
iptables -A IN_RULES_ETH0 -s 169.254.0.0/16 -j DROP
# drop stuff for eth1 allow only a few addresses
# IN_RULES_ETH1
iptables -A IN_RULES_ETH1 -m iprange –src-range 192.168.0.2-192.168.0.9 -j DROP
iptables -A IN_RULES_ETH1 -m iprange –src-range 192.168.0.11-192.168.0.26 -j DROP
iptables -A IN_RULES_ETH1 -m iprange –src-range 192.168.0.27-192.168.0.59 -j DROP
iptables -A IN_RULES_ETH1 -m iprange –src-range 192.168.0.61-192.168.0.254 -j DROP
iptables -A IN_RULES_ETH1 -s 169.254.0.0/16 -j DROP
# drop stuff out rules
iptables -A OUT_RULES -d 224.0.0.0/4 -j DROP
# allow previously established / related connections
iptables -A OUT_RULES -m state –state ESTABLISHED,RELATED -j ACCEPT
iptables -A IN_RULES_ETH0 -m state –state ESTABLISHED,RELATED -j ACCEPT
iptables -A IN_RULES_ETH1 -m state –state ESTABLISHED,RELATED -j ACCEPT
# forward all on eth1
iptables -A FORWARD_ETH1 -i eth1 -j ACCEPT
iptables -A FORWARD_ETH1 -o eth1 -j ACCEPT
####################################################################
# out services / ports
# ICMP
iptables -A OUT_RULES -p icmp -j ACCEPT
# Port 22 (SSH)
iptables -A OUT_RULES -p tcp –dport 22 -j ACCEPT
# Port 1723 (VPN)
iptables -A OUT_RULES -p tcp –dport 1723 -j ACCEPT
# Port 123 (NTP)
iptables -A OUT_RULES -p udp –dport 123 -j ACCEPT
iptables -A OUT_RULES -p tcp –dport 123 -j ACCEPT
iptables -A OUT_RULES -j LOG –log-prefix “BLOCKED OUTPUT ”
iptables -A OUT_RULES -j REJECT
####################################################################
# input services / ports
# ICMP
iptables -A IN_RULES_ETH0 -p icmp -j ACCEPT
iptables -A IN_RULES_ETH1 -p icmp -j ACCEPT
# Port 22 (SSH)
iptables -A IN_RULES_ETH0 -p tcp –dport 22 -m state –state NEW -j ACCEPT
iptables -A IN_RULES_ETH1 -p tcp –dport 22 -m state –state NEW -j ACCEPT
# Port 1723 (VPN)
iptables -A IN_RULES_ETH0 -p tcp -m tcp –dport 1723 -j ACCEPT
iptables -A IN_RULES_ETH0 -p gre -j ACCEPT
######################################################################
# Drop MS NetBIOS scanning early (it spams the log)
iptables -A IN_RULES_ETH0 -p tcp –dport 137 -j DROP
iptables -A IN_RULES_ETH1 -p tcp –dport 137 -j DROP
iptables -A IN_RULES_ETH0 -p udp –dport 137 -j DROP
iptables -A IN_RULES_ETH1 -p udp –dport 137 -j DROP
iptables -A IN_RULES_ETH0 -p tcp –dport 138 -j DROP
iptables -A IN_RULES_ETH1 -p udp –dport 138 -j DROP
# Log and reject
iptables -A IN_RULES_ETH0 -j LOG –log-prefix “BLOCKED INPUT ”
iptables -A IN_RULES_ETH1 -j LOG –log-prefix “BLOCKED INPUT ”
iptables -A IN_RULES_ETH0 -j REJECT
iptables -A IN_RULES_ETH1 -j REJECT
Thanks.
Hopefully you can help me…
Best
Luke.
> Hopefully you can help me…
Luke, I need some additional clarification… I assume you posted this information because multiple default routing for you isn’t working correctly — is that correct? If so, can you clarify as to what is and is not working correctly? (e.g. which route works properly?)
Also, remove _all_ the firewall rules and just test to make sure multiple default routing works from the 192.168.0.7 system, first (without any further NAT’ing). If multiple default routes for that host _fail_ to work, then the problem isn’t with your iptables rules — however, if multiple default routing _works_ from (just) the host without the iptable rules, then you’ve isolated the problem to your iptables rules.
The overall idea here, is to reduce the number variables in your configuration, in order to identify the root cause.
Regards,
— Darien
Hi Darien. Thanks for your reply.
The problem is that users on the 192.168.0.0 network who set their gateway to 192.168.0.7 should have their traffic go straight out through the 110.xxx.xxx.173 interface, but it goes first to the 192.168.0.1 interface…
I will try what you suggest and get back here…
Ive removed the gateway statement from the local NIC for 192.168.0.7 and it seems to have fixed the problem.
I thought that having routes added later (as in /etc/rc.local) would have overridden the default gateway set in /etc/network/interfaces but apparently not…
Hi Darien. Lastly is there a way I can edit my posts … I seem to have mentioned my ip address in my original post, although I thought I had edited them all, one still remains, could you replace with xxx… thanks
Okay, this is fixed..
Thanks. You’re the man!
I nee a help, I m delete device from ‘network configuration’ and again add it. It may be wrong so I unable to ping eventhough default gateway of my router. so plz guide me to configure properly to access internet via netgare 150 adsl router. plz replay in mail id : paradise_437@yahoo.com
anybady online
Thanks Darien, this has moved things forward for me. I hope that someone can help with the 1 (!) remainng problem.
I have eth0 on the 10.10.170 subnet and eth1 on the 10.10.177 subnet. There is more than 1 gateway on each subnet, for reasons I won’t go into. The suggestion in the initial post, modified for my circumstances, allowed me to ssh and ping into either interface. So far, so good. I wasn’t able to do that before. Now however, I’m having trouble transitting from one subnet to the other. I wanted to use the 10.10.177.1 gateway, and that is how I set it up. However, a tracert tries to send me through the 10.10.177.2 gateway. I’ve searched the system and there is no mention of that ip address anywhere to be found.
Here’s the commands I used.
ip route add 10.10.177.0/24 dev eth1 src 10.10.177.20 table voice
ip route add default via 10.10.177.1 dev eth1 table voice
ip rule add from 10.10.177.20/32 table voice
ip rule add to 10.10.177.20/32 table voice
This is under Centos 6
Cheers,
Chris
Hi Chris,
I’ll need more information about your network setup, including the configuration used to setup eth0.
Regards,
— Darien
Hi, i’ve follow your step, but when i run /etc/init.d/networking restart , the ip table admin is gone, how to save it?
thanks
sandm3n: See this earlier comment:
Thanks for the ansewer, but i still confuse about this (i’m new in ubuntu) , i’ve problem about connection, you can see here (please) http://i41.tinypic.com/nl2as6.jpg
in that case, i want computer on lan, can connect through squid proxy to the internet, when i setting up without client connect, it can’t ping to LAN because of 10.10.7.0 default route (that connect to internet)
when i disable that default route, i can successfully ping to lan, But link to internet is off.
How could i make two connection, so i can connect to LAN and the internet, i refer to your site but still not work.
Many thanks
Hi sandm3n,
The problem is that your Squid proxy is running inside VMware. I assume you are running VMware Workstation. The problem you are running into is specific to VMware’s network configuration.
1) You need to change eth0 to use a vmnet that is _bridged_ with a physical network interface card (NIC).
2) You need to change eth1 to use a vmnet that is _bridged_ with a _different_ physical network interface card (NIC).
Because you are using (host-only) and (NAT) for these interfaces, VMware is not letting the policy routing settings work correctly.
i will let you know my whole simulated network, see here http://i40.tinypic.com/2f061hd.jpg , using GNS3 virtual box edition with cisco router and inter vlan on it, i use Virtual Box as client, you can see cloud on the right that is connect to vmware (ubuntu server), client to client between vlan can connect .
first question is my topology correct? on the cloud (server) i run Samba file sharing and Squid (not running)
Do i must running samba and squid separately ?
many thanks!
Hi sandm3n,
You have enough complex components that I’d recommend testing the multiple default route settings on the Ubuntu server in _isolation_ from the rest of your network. The samba and squid services running on the Ubuntu server do not matter.
Specifically, if you have two separate clients connected directly to the two interfaces on the Ubuntu server, and you can ping each interface separately from each client, then your multiple default route settings should be working properly.
Once you have established that this works, you can slowly increase the complexity of your network architecture and re-test these basic checks at each step (e.g., pinging each interface). That way, when the test fails, you will be able to isolate which component is responsible for the problem.
Thanks for your nice advice, want to test your suggest first, i’ve to learn more about this, sorry if i bothering you Darien.
Great Article…
Thank you
I bookmarked this page
regards
Samy
Wow, this absolutely saved my bacon; thanks for posting this. So easy to do and works like a champ (once you add everything into rc.local that is!).
I apologize if the question is stupid, I’m new with linux/ubuntu.
I have a pc (ubuntu-server) with two network cards (eth0 and eth1).
eth1 (ip 192.168.1.40) is connected to the internet (ADSL) via a gateway-router (ip 192.168.1.1).
eth0 (ip 192.168.0.5) is connected to the Internet (another provider DSL) via another gateway-router (ip 192.168.0.1).
I use eth1 for all internet and local connections.
I just want to do a reverse ssh tunnel “ssh -R 9999: localhost: 22 xxxxx@123.432.xxx.xxx” that go out to 123.432.xxx.xxx through eth0 so that if I’m connecting (from another PC on the internet) to port localhost:9999 of 123.432.xxx.xxx I am redirected to my pc/ubuntu-server at it’s port 22 using the internet line of eth0 which has an upload bandwidth much greater than ADSL to which eth1 is connected.
How can I do this?
I must do what we are talking about in this article? Or just a rule of static route? Can you help? thanks infinitely
Hi Jack,
If you want all traffic destined for “123.132.x.x” to go out eth0, then you would need to create a static route that forces all “123.132.0.0/16” out through eth0, accordingly.
Specifically, the command is:
route add -net 123.132.0.0 netmask 255.255.0.0 dev eth0
Thanks, it works fine.
I used:
route add -host xxx.xxx.xxx.xxx gw 192.168.0.1 dev eth0
and to make it permanent I added to /etc/network/interfaces:
up route add -host xxx.xxx.xxx.xxx gw 192.168.0.1 dev eth0
where “xxx.xxx.xxx.xxx” is the ip of a ssh server.
Very Nice info, neatly explained.
Good day Darien,
I nearly have the same situation you described.
I have a strato vserver (2.6.18-028stab091.2) OpenSuSE 11.1:
>ifconfig
venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:127.0.0.1 P-t-P:127.0.0.1 Bcast:0.0.0.0 Mask:255.255.255.255
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
venet0:0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:85.214.156.xxx P-t-P:85.214.156.xxx Bcast:0.0.0.0 Mask:255.255.255.255
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
venet0:1 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:81.169.166.xxx P-t-P:81.169.166.xxx Bcast:0.0.0.0 Mask:255.255.255.255
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
>uname -a
Linux hxxxxxxx 2.6.18-028stab091.2 #1 SMP Fri Jun 3 00:02:40 MSD 2011 i686 athlon i386 GNU/Linux
>netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 0.0.0.0 0.0.0.0 U 0 0 0 venet0
>route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
loopback * 255.0.0.0 U 0 0 0 lo
default * 0.0.0.0 U 0 0 0 venet0
>cat /etc/sysconfig/network/ifcfg-venet0
STARTMODE=onboot
BOOTPROTO=static
BROADCAST=0.0.0.0
NETMASK=255.255.255.255
IPADDR=127.0.0.1
IPADDR_0=85.214.156.xxx
LABEL_0=0
IPADDR_1=81.169.166.xxx
LABEL_1=1
>cat /etc/sysconfig/network/ifroute-venet0
default – – venet0
default :: – venet0
venet0:0 – IP 85.214.156.xxx netmask 255.255.255.255
venet0:0 – gateway is: 0.0.0.0
venet0:1 – IP 81.169.166.xxx netmask 255.255.255.0
venet0:1 – gateway is: 0.0.0.0
My goal is to do exactly like that:
http://www.linuxquestions.org/questions/linux-networking-3/ssh-tunneling-with-multiple-ips-720679/
Case 1) User (A) connects with putty sock5 tunnel to venet0:0 (outgoing traffic has to go back venet0:0)
“C:\Program Files\putty\plink.exe” -ssh -T -D 7777 -load
I configured my firefox for case 1)
->Options->Advanced->Settings..->Manual proxy configuation->SOCKS Host: localhost, Port: 7777
Now I can browse through that tunnel with the IP of linux server venet0:0
Test with http://whatismyipaddress.com shows 85.214.156.xxx -> OK.
Case 2) User (B) connects with putty sock5 tunnel to 2nd IP (outgoing traffic has to go back venet0:1)
“C:\Program Files\putty\plink.exe” -ssh -T -D 8888 -load
I configured my firefox for case 2)
->Options->Advanced->Settings..->Manual proxy configuation->SOCKS Host: localhost, Port: 8888
Now I want to browse through that tunnel with the IP of linux server venet0:1
I tried to setup you wrote above, but the case 2) always show the IP of venet0:0
Hope you and the other can help me.
Thanks and regards,
Tien.
Hi Tien,
Some of the information you provided in your example conflicts.
1) Earlier, you had said:
> venet0:1 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
> inet addr:81.169.166.xxx P-t-P:81.169.166.xxx Bcast:0.0.0.0 Mask:255.255.255.255
> UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
Notice that the netmask for this interface is 255.255.255.255
Then, you said:
> venet0:1 – IP 81.169.166.xxx netmask 255.255.255.0
> venet0:1 – gateway is: 0.0.0.0
In this case, the netmask is 255.255.255.0.
Which is it? The netmask is either 255.255.255.255 or 255.255.255.0, but not both.
2) Also, both venet0:0 and venet0:1 need valid gateways:
venet0:0 – gateway is: 0.0.0.0
venet0:1 – gateway is: 0.0.0.0
The gateways you specified “0.0.0.0” are not valid, in this case. You need specific IP addresses for each gateway on each interface.
3) Therefore, let’s assume you had the following information:
venet0:0 – 85.214.156.2 netmask 255.255.255.0
venet0:0’s gateway is: 85.214.156.1
venet0:1 – 81.169.166.2 netmask 255.255.255.0
venet0:1’s gateway is: 81.169.166.1
Here are the corresponding commands you would run:
printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
printf “2\tuplink1\n” >> /etc/iproute2/rt_tables
ip route add 85.214.156.0/24 dev eth0 src 85.214.156.2 table uplink0
ip route add default via 85.214.156.1 dev venet0:0 table uplink0
ip rule add from 85.214.156.2/32 table uplink0
ip rule add to 85.214.156.2/32 table uplink0
ip route add 81.169.166.0/24 dev eth1 src 81.169.166.2 table uplink1
ip route add default via 81.169.166.1 dev venet0:1 table uplink1
ip rule add from 81.169.166.2/32 table uplink1
ip rule add to 81.169.166.2/32 table uplink1
ip route flush
Hi Darien,
Many thanks for the feedback.
Yes you’re right, I pasted wrong regarding the netmask.
Here is the right one:
venet0:0 – 85.214.156.2 netmask 255.255.255.255
venet0:0′s gateway is: 0.0.0.0
venet0:1 – 81.169.166.2 netmask 255.255.255.255
venet0:1′s gateway is: 0.0.0.0
I asked the guys from Strato provider (http://www.strato.de/server/virtual-linux-server/) to give me the gateways, they said this is a v-server and has gateway = 0.0.0.0
With the route command I got that:
>route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
loopback * 255.0.0.0 U 0 0 0 lo
default * 0.0.0.0 U 0 0 0 venet0
As I can see in your example with eth0 & eth1, I do only have venet0, venet0:0 & venet0:1
Hi Tien,
I’m skeptical that the real gateway is 0.0.0.0… The best way to know for sure, would be to run tcpdump on your two interfaces (venet0:0 and venet0:1), and see what the destination IP address is, when you issue a traceroute to a server on the internet (e.g., “traceroute yahoo.com”).
So, to be clear, the steps would be:
1) tcpdump -ni venet0:0
2) traceroute yahoo.com
3) Look at the destination IP addresses listed in the traceroute output and packet dump
You can repeat these steps for venet0:1, also.
Hi,Darien
Very thanks for your article,i have same situation and i need your help.my vps have 2ips,and they all use a NIC venet0,vent0:0 vent0:1
DEVICE = venet0
BOOTPROTO = static
ONBOOT = yes
IPADDR = 127.0.0.1
NETMASK = 255.255.255.255
BROADCAST = 0.0.0.0
IPV6INIT = “yes”
DEVICE = venet0:0
BOOT = yes
IPADDR=119.XX.XX.206
NETMASK = 255.255.255.255
DEVICE = venet0:1
ONBOOT = yes
IPADDR = 119.XX.XX.245
NETMASK = 255.255.255.255
how to achive accessing ip1 get ip1,acessing ip2 get ip2? I think i need a correct code similar the following for my case.
—-quote—
printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
printf “2\tuplink1\n” >> /etc/iproute2/rt_tables
ip route add 85.214.156.0/24 dev eth0 src 85.214.156.2 table uplink0
ip route add default via 85.214.156.1 dev venet0:0 table uplink0
ip rule add from 85.214.156.2/32 table uplink0
ip rule add to 85.214.156.2/32 table uplink0
ip route add 81.169.166.0/24 dev eth1 src 81.169.166.2 table uplink1
ip route add default via 81.169.166.1 dev venet0:1 table uplink1
ip rule add from 81.169.166.2/32 table uplink1
ip rule add to 81.169.166.2/32 table uplink1
ip route flush
—–quote—-
pls write a correct codes smilar to the above,thanks very much. i am not good at programming and linux so i expect to get a very correct code and input in my vps.pls attention to if the “eth0″,”eth1” is correct,because i pass these codes to my providers to help me they told me the eth0 cause error.eth0 don’t exist and replace it with venet0:0 cause error :RTNETLINK answers:invalid argument,so looking forwards to your help.
Best Regards
Thomas
Hi Thomas, what is the default gateway for each interface in your setup?
vservers are “thin” virtual machines which hand off to the real machine to do actual routing, so the “gateway” is meaningless and left as 0.0.0.0. If you want policy routing it would have to be done in the real machine.
Thanks for the clarification, Martin!
I am connected to 2 interfaces simultaneously like eth0 and wlan0. And eth0 is the default interface. Now I start downloading something on the eth0 interface. In midst of the download if I change the default interface to wlan0, the download breaks. I don’t understand why this happens. Can you please tell me ??
Abhay, that is because you only have one default route setup. During your download, the application probably got temporarily disconnected and tried to resume the download after the default gateway switched.
I love you man!
Hi Darien,
Awesome post. I have followed your post, and definitely learnt more about routing in linux. Thanks.
Now to my problem. I have a web server with two NIC’s:
eth0 is used as the local network nic: 192.168.250.43/24 (no gateway)
eth1 is used as the public network nic: 41.222.8.146/29 (gateway 41.222.8.145)
eth1 is the default route, and the only device with a specified gateway.
If I ping from an external address, I get replies on the external/public IP for roughly 5 min, and then the time-outs start, which continue indefinitely. If, however, I ping the public IP’s primary DNS, on the webserver itself, I start getting replies again on the external PC I am polling connectivity on.
Is this a routing issue on my webserver, or a routing issue on my router, or is it a non-routing issue.
Thanks again for the great post.
While this isn’t a multi-default route issue, It sounds like you have some sort stateful packet inspection (SPI) firewall between your 2 NIC system and the external system you are pinging from. The firewall could be built into the 2 NIC system — not 100% sure. Regardless, this issue is quite common in those types of setups. Specifically the outbound traffic from the 2 NIC system through the SPI firewall triggers the firewall to allow reply traffic for a set period of time (a max timeout). Therefore, you’ll need to check the systems for corresponding firewalls. Disabling them altogether should alleviate this problem.
Regards,
Thank you very much. I was having this issue since yesterday. Your email just save my day and my job too. You are a star.
Many thanks to You !!!!!!
I spent days trying to solve this. I even gave up on it but had to start again several times. This worked right away and finally I can move up with my network upgrade.
Exactly what I was looking for, this worked perfectly.
Is not that possible that we can add a static route on one ethernet and add a default gateway on one ethernet ?
Yes, that is possible, but I’m not sure that is what you mean.
Regards,
is not that possible that we can add a static route on one ethernet and keep another ethernet on default gateway
Wow 200 posts (this is 201 for those keeping count) and 5 1/2 years later, amazing article…
2 things.. first let me quote you… “this information is designed primarily to preserve my sanity”, hows that working out for you?
And second, Im using Ubuntu 12.04.
> netstat -anr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 control
0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 wlan
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 control
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 wlan
the was after adding “route add default gw 192.168.1.1” to /etc/rc.local
the second line “0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 wlan” should not be there, ever. But when ubuntu boots up, the wlan device is the once to get the lease first. How can I set the default gateway in this multi-nic system… At the same time, the machine needs to talk to everything on the 192.168.2.X network as well.
Thanks, and again… wow.. really impressive article, I read every post, you are a trove of information.
Never mind. I remembered something you said in one of the comments, tried it, and it works…
I’m not saying what or what comment, because people should really read all of it before posting, all the information is there.. really amazing.
i did as per ur article but the problem is not solved
eth0 = 10.0.0.2
eth1= 192.168.5.102
default gateway is 192.168.5.1
gateway for eth0 is 10.0.0.138
when i am pinging it tries to go via 192. which i dont want it should go thru 10…
so i did
ip route add 10.0.0.0/24 dev eth0 src 10.0.0.2 table admin
ip route add default via 10.0.0.138 dev eth0 table admin
============================
ip rule add from 10.0.0.2/32 table admin
ip rule add to 10.0.0.2/32 table admin
==================================
ip rule show
0: from all lookup 255
32764: from all to 10.0.0.2 lookup admin
32765: from 10.0.0.2 lookup admin
32766: from all lookup main
32767: from all lookup default
============================
but still
ping smtp.bizmail.yahoo.com
PING smtp.bizmail.yahoo.com (98.139.221.126) 56(84) bytes of data.
From 192.168.5.102 icmp_seq=1 Destination Host Unreachable
From 192.168.5.102 icmp_seq=2 Destination Host Unreachable
From 192.168.5.102 icmp_seq=3 Destination Host Unreachable
it is still going thru 192…
Hi Ahmed,
Okay, then switch your configuration where eth0 is set by default and use table “admin” to define your eth1 settings.
Regards,
i did the opposite
ip rule show
0: from all lookup 255
32764: from all to 192.168.5.102 lookup admin
32765: from 192.168.5.102 lookup admin
32766: from all lookup main
32767: from all lookup default
==============================================
still the problem exist
when i am pinging to outside it is try to go via 192.168.. , i want it to go via eth0 10.0.0. … bcause
dont know y its not working for me
Okay, I see another issue. Your first rule has:
0: from all lookup 255
And that should be instead:
0: from all lookup local
Perhaps “255” = “local” on your system, but I don’t know that for sure.
Regards, Darien
Hi, This is a great post. thanks again,.
One question – How do you print the routing table the one you created as “admin” in the above example? how do I see my connection to the particular IP uses the correct gateway?
“ip route show table main” and “ip route show table default”
Regards,
Hi all,
I am trying to create the following set up
I have following linux networking set up.
Machine A Ip-10.8.11.100 (Source machine)
Machine B ip-10.8.11.1 (acting as a gateway for 10.5.0.0 network )
Machine C ip-10.8.11.2 (acting as a gateway for 192.68.11.0 network )
Machine D ip-10.5.0.1 (Dest machine)
Machine E ip-192.168.11.1 (Dest machine)
Machine F ip-10.8.11.5 (default agteway)
Now I want to add both the destination networks into Machine A so that I should able to ping ip address of both the networks from it.
The rules and routes that I ave added on machine A a below :-
$ip rule add from 10.8.11.100 table 2
$ip route add 10.5.0.0/24 via 10.8.11.1 dev eth0 src 10.8.11.100 table 2
$ip route add 192.168.11.1 via 10.8.11.2 dev eth0 src 10.8.11.100 table 2
$ip route add default via 10.8.11.5 table 2
$ip route flush cache
Here the rules on the Machine A
$ip rule show
0: from all lookup local
32763: from 10.8.11.100 lookup 2
32766: from all lookup main
32767: from all lookup default
Here are the routes in table 2 of machine A
$ip route show table 2
192.168.11.0/24 via 10.8.11.2 dev eth0 src 10.8.11.100
10.5.0.0/24 via 10.8.11.1 dev eth0 src 10.8.11.100
default via 10.8.11.5 dev eth0
Now what is happening is when I trying to ping any destination from Machine A, I am not getting any ping reply from any of the network and I think the packet is also not reaching to destination machine as I am not seeing anything in the tcpdump traces at the destination.(Machine D & Machine E).
Now when I am adding the same rules to the main table of linux everything is working fine . I can able to ping all the static as well as default networks from machine A.
I don’t know whts was happening when I am adding the same routes to table 2.
A part from that I am directly using the table index 2 without adding its entry in /etc/iproute2/rt_tables file.
Is it necessary to add the table name into that file.?
I did that as well but looks like it is not working out.
How can we know which routing table currently linux is using.?
Please help me here
Thanks alot, worked for me!
Finally found the webpage that sorted my multi nic networking issues. Excellent post. Followed it, and it worked a treat.
I am having trouble getting broadcast message to their respective networks/interface. All broadcast messages to 255.255.255.255 go out the default route in the main table even if I set up default routes in other tables and create the rules to use those routes.
Should the advanced routing features handle broadcasting or is the kernel/library somehow changing the broadcast message to the default route address before getting to the routing library?
Wow! Such a great post! It still comes up at the top although it was written so long ago!
I don’t know if you keep up the comments but I am really hoping that you do.
I have a question for you… I am trying to use your guide with multiple wifi connections. Is that any different than wired ones? I’m thinking that it shouldn’t right?
For example I want all peer 2 peer traffic to go through wlan1. I’ve selected the appropriate option in ktorrent for limiting all traffic to wlan1 and created a p2p rule table following your guide. Before reading your post (and realizing that connections still go out from the default route) when I was limiting ktorrent to wlan1, I would see no traffic…0.
Now I created a new table rule like you are doing with admin, however when I do ‘netstat -n | grep LISTENING’ established connections are still bound to the primary ip address (wlan0) and not wlan1. Is there a reason why those new connections are not listening to the additional ip address? The weird thing is that now I do see traffic in ktorrent, albeit slow, but I do see some. But even regardless of peer to peer traffic, not even 1 connection is listed as LISTENING for the secondary wlan1 ip? All connecitons are listening on the primary (wlan0) ip. What do you make of that?
Hi,
Yes, policy routing works on any type of network interface — wired or wireless.
It sounds like some sort of ‘ktorrent’ problem, then. Specifically, if you configured ktorrent to only use wlan1, but it _still_ uses wlan0, then it sounds like a but with ktorrent, regardless of policy routing settings.
I’d recommend reaching out to the ktorrent support mailing lists and find out if this is a known or unknown issue. If you completely turn off policy routing _and_ ktorrent _still_ uses wlan0 (even when you explicitly set wlan1), then completely disable wlan0 and see if ktorrent still works properly. If not, then it sounds like ktorrent has a bug where it always uses the first registered interface of the system.
Regards,
Thank you for the reply,
so you were right, it was a bug with ktorrent. I managed to make it work with Vuse though.
Also I made it work with the following two commands:
ip route add table 2 default via 192.168.1.1
ip rule add from 192.168.1.0/24 table 2
here’s my rules table:
0: from all lookup local
32765: from 192.168.1.0/24 lookup 2
32766: from all lookup main
32767: from all lookup default
going over your post again I noticed that 32764 is missing. I don’t mean to sound stupid but would you mind explaining a bit more why do we need 32764 *and* 32765?
thank you!
PS: It’s such a shame that you stopped blogging! Got too busy? You have some really nice posts, and judging by the number of comments pretty good audience.
> so you were right, it was a bug with ktorrent. I managed to make it work with Vuse though.
I figured. 😉
> Also I made it work with the following two commands: > > ip route add table 2 default via 192.168.1.1 > ip rule add from 192.168.1.0/24 table 2 > > here’s my rules table: > 0: from all lookup local > 32765: from 192.168.1.0/24 lookup 2 > 32766: from all lookup main > 32767: from all lookup default > > going over your post again I noticed that 32764 is missing. I don’t mean to sound stupid but would you mind explaining a bit more why do we need 32764 *and* 32765?
Sure. Without 32764, any traffic from the internet going to 192.168.1.0/24would get directly routed your default interface, instead of the secondary interface (e.g., 192.168.1.1). This only matters if your secondary interface is connected to some sort of upstream network translation device (NAT) that is able to route your RFC 1819 addressable interface out to the internet. If that’s the case (most likely), then you’ll find that any internet-based traffic going to IP 192.168.1.1 will likely go through your primary interface, instead of your secondary interface.
You can run tcpdump on all interfaces to confirm this, by issuing a simple ping out to the internet, where you specify that it should use the secondary interface for all outbound traffic.
> PS: It’s such a shame that you stopped blogging! Got too busy? You have some really nice posts, and judging by the number of comments pretty good audience.
Thanks. I don’t have much time to blog publicly, currently. As such, I tend to try and stick to topics that aren’t already discussed on the internet, elsewhere. Therefore, if I come across an issue that isn’t easily found via standard Google searches, I’ll likely create a future blog post about it. 😉
Regards,
Ok so, when I do traceroute -i wlan0 and tracertoute -i wlan1 I do see traffic going out from the appropriate routes
So rule 32764 forwards any traffic FROM the internet (meaning coming in from other interfaces?) TO the secondary interface gateway? Because rule 32765 reads “from 192.168.1.0/24 lookup 2” so I am assuming it is taking care of traffic coming in the secondary interface making sure that it goes out from 192.168.1.1 right?
And if so, Vuse is working fine without rule 32764, because all traffic is generated directly from the secondary interface and also arriving into the interface? Is that why so far everything seems to be working just with rule 32765 ?
THAT’s what confuses me!
thank you,
>So rule 32764 forwards any traffic FROM the internet (meaning coming in from other interfaces?) TO the secondary interface gateway?
Correct.
> Because rule 32765 reads “from 192.168.1.0/24 lookup 2” so I am assuming it is taking care of traffic coming in the secondary interface making sure that it goes out from 192.168.1.1 right?
Correct.
> And if so, Vuse is working fine without rule 32764, because all traffic is generated directly from the secondary interface and also arriving into the interface? Is that why so far everything seems to be working just with rule 32765 ?
Pretty much, yes.
> THAT’s what confuses me!
Yup, policy routing is not straight forward, at all.
Regards, Darien
The conversation reached the max limit depth so I’ll reply here..
Thanks again for the last clarification, I got it now!
Oh and please continue blogging good stuff!
Hi again Darien,
sorry to bother you again, I was just going through the comments section in this page checking to see how others might be using policy routing for any smart scenarios.
In the last comment made by user “anestis” where he used the -I parameter to bind to ping to an interface, I tried it too. I’m having the same issue as well. Binding ping works only if I bind it to the IP of the secondary interface (i.e. -I 192.168.1.12). If I bind it directly to the interface (i.e. -I wlan1) it’s giving me destination host unreachable. With the primary interface, it works both ways (either with -I ip or -I wlan0). Maybe -I is not a reliable way to test?
Thanks again,
Yeah, I think you’re right. It seems that option is not a reliable way to test policy routing.
Regards, Darien
Regards,
When I get the change I’ll test it with centos or RHEL too. Maybe it’s a debian based distro issue. Thanks again for the reply back.
Hi Daren, awesome post!! Googling paid off indeed!
Let me ask you a question though,
I’m having an issue and I don’t know if it’s my configuration
# traceroute -i eth0 8.8.8.8 (goes out eth0 gw)
# traceroute -i eth1 8.8.8.8 (goes out eth1 gw)
# ping -I eth0 8.8.8.8 (works ok)
# ping -I eth1 8.8.8.8 Destination Host Unreachable)
# ping -I 192.168.2.2 (eth1 ip) (works ok)
So for some reason if I use the direct ip of eth1 ping works but if I use the interface, I get Destination Host Unreachable.
Any ideas as to what might be the problem here?
last line I meant # ping -I 192.168.2.2 8.8.8.8
oh and sorry for butchering your name… I meant Darien 🙂
Please provide full network and routing information for each interface, as I don’t have enough information in order to identify the core problem.
Regards, Darien
On Nov 8, 2012, at 3:16 AM, “Darien Kindlund’s Blog”
Oh I’m sorry I should have added some more details.
So here’s my setup:
eth0: 192.168.1.2 gw0:192.168.1.1
eth1: 192.168.2.2 gw1:192.168.2.1
/etc/iproute2/rt_tables
#added:
1 table1
2 table2
##route1
ip route add default via 192.168.1.1 dev eth0 table table1
ip rule add from 192.168.1.0/24 table1
##route2
ip route add default via 192.168.2.1 dev eth1 table table2
ip rule add from 192.168.2.0/24 table2
so, that’s what I did pretty much and then tried to test with traceroute -i and ping -I
Traceroute seems to work in both instances (going out from the correct gateways).
Ping worked for my default interface (eth0), however for eth1 it only works if I put the actual IP but not with the eth1
ie.
ping -I 192.168.2.2 8.8.8.8 (works)
ping -I eth1 8.8.8.8 destination host unreachable
Isn’t that weird? Do you thing I misconfigured something? Let me know if you need any more details.
Thank you,
After issuing the “ip rule” commands, did you issue “ip route flush cache” at the end, so that your commands will take immediate effect?
Regards, Darien
On Nov 9, 2012, at 12:19 AM, “Darien Kindlund’s Blog”
Sorry for the long delay but I had to wait to get access to that system.
So, I did the simple setup from scratch with the two routes and I tried flushing the cache like you suggested but same thing. Any thoughts?
It looks like using the -I option with ping is not a reliable way to test policy routing; see sxty1’s earlier comment for details.
Regards, Darien
Regards,
For the 2 local networks 192.168.7.0 and 10.10.70.0 in your example, you defined 2 default gateways for them nice enough to access the Internet in 2 distinct paths. That is very good but I need more than that. I also desire the 2 local networks be able to ping each other meanwhile. What’s the trick? Please teach me.
I assume you want _all_ the systems on both networks to be able to ping each other — not just the two interfaces on your Linux system. In that case, you need to look at standard routing guides, which clearly explain this. You don’t need policy routing, specifically.
Here are some useful references on general routing for Linux: http://www.google.com/url?sa=t&source=web&cd=3&ved=0CCAQFjAC&url=http%3A%2F%2Flinux-ip.net%2Fhtml%2Frouting-intro.html&ei=UziqUOTPFqnB0QGm7YG4DQ&usg=AFQjCNG8qeJPOA12XiW69xbSt0WKg7uArQ
http://www.google.com/url?sa=t&source=web&cd=2&ved=0CBUQFjAB&url=http%3A%2F%2Flartc.org%2Fhowto%2F&ei=UziqUOTPFqnB0QGm7YG4DQ&usg=AFQjCNGcAW7fcd573mwGwbp6fk3ZjXZ5kg
Regards,
Actually I have setup a Ubuntu linux gateway with 2 interfaces. For instance
auto eth1
iface eth1 inet static
address 192.168.1.20
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
auto eth2
iface eth2 inet static
address 10.10.10.20
netmask 255.255.255.0
network 10.10.10.0
broadcast 10.10.10.255
I have enabled the IP forwarding such that PC in 192.168.1.0/24 and PC in 10.10.10.0/24 could ping each other though this machine as their default gateway.
TWO other routers, A and B, have independing Internet connections respectively. Each one has an LAN interface directly connected to the 2 LANs. The LAN port IP of router A is 192.168.1.19. The LAN port IP for router B is 10.10.10.19. I desire each network go through a specified router for Internet access. For instance; PC in 192.168.1.0/24 go through router A. PC in 10.10.10.0/24 go through router B. To do this, I have added 2 tables to the file: /etc/iproute2/rt_tables
100 pma
200 ma
Then I issued the following commands:
ip rule add from 192.168.1.0/24 table pma
ip rule add from 10.10.10.0/24 table ma
ip route add default via 192.168.1.19 table pma
ip route add default via 10.10.10.19 table ma
ip route flush cache
Now that the 2 networks could have the Internet access independently. But they cannot ping each other anymore. If I reboot this gateway without issuing the previous 5 commands. PC from both networks could each other again. And none of the PC could get Internet access anymore. Of course I could add a gateway statement to the network configuration file, for instance, the last line of the following stance.
auto eth1
iface eth1 inet static
address 192.168.1.20
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.19
But that is not I want since it would become a default gateway to all connected networks. I want each connected network go through different path for Internet access, namely router A and router B. In fact, I wish I could dynamically assigned some selected PC from 10.10.10.0/24 go through route A in additional to the whole 192.168.1.0/24 network.
This is what I mean I want “more” than that. Sorry for my fault of ambiguous illustration previously. Again I appreciate your document. And I am expecting further teaching from you. Thank you in advance.
Hi,
I see what you mean. I appreciate your detailed description, as that really helps me better understand what the possible issues are. First, I agree with you that you do _not_ want to include the gateway directive in your individual network configuration files.
Basically, you want: 1) policy routing from this “router” system out to the internet, so that each upstream link is used independently -AND- 2) proper routing between other systems on networks A and B through this router, so that they don’t have to go out to the internet to reach each other
Assuming your initial IP configuration for eth1 and eth2, as well as the default gateways defined for each interface, you indicated your tables are setup as such:
/etc/iproute2/rt_tables 100 pma 200 ma
Issue the following commands:
ip rule add from 192.168.1.0/24 table pma ip rule add to 192.168.1.0/24 table pma
ip rule add from 10.10.10.0/24 table ma ip rule add to 10.10.10.0/24 table ma
ip route add 192.168.1.0/24 dev eth0 src 192.168.1.20 table pma ip route add default via 192.168.1.19 dev eth0 table pma
ip route add 10.10.10.0/24 dev eth1 src 10.10.10.20 table ma ip route add default via 10.10.10.19 dev eth1 table ma
ip route flush cache
Regards,
Thanks for your teaching Darien. For the following suggested statements:
ip rule add from 192.168.1.0/24 table pma ip rule add to 192.168.1.0/24 table pma
ip rule add from 10.10.10.0/24 table ma ip rule add to 10.10.10.0/24 table ma
Could you explain the functions? Again I would like PC in these 2 networks 192.168.1.0/24 and 10.10.10.0/24 could ping each other through this Linux gateway. How the 2 statements could fullfill this requirement?
Suppose I would like some selected nodes (e.g. 10.10.10.100 is the PC of the General Manager and requires a preference connection to the Internet) to reach Internet by going through eth0 with table pma instead of the deault path of the subnet the node belongs to. What statement shoud be added? I understand that there is a PRIORITY paramenter setting for ip rule statements. How could I make use of it?
How to use statement: “ip route add” and “ip route append” What’s the difference? Could you provide some examples to illustrate the difference?
That’s a lot of questions but I’ve found your teaching the most comprehensive to me and will help in a more effiective way to complete my study on policy routing. Thank you very much Darien. Good teaching.
Hi Shawn,
Sorry about the confusion. It looks like wordpress combined multiple lines into a single statement. Here was my original recommendation, but this time, I’ve added multiple lines between each command, so that it’s clearer to see when one command ends and another begins:
Assuming your initial IP configuration for eth1 and eth2, as well as the default gateways defined for each interface, you indicated your tables are setup as such:
/etc/iproute2/rt_tables
100 pma
200 ma
Issue the following commands:
ip rule add from 192.168.1.0/24 table pma
ip rule add to 192.168.1.0/24 table pma
ip rule add from 10.10.10.0/24 table ma
ip rule add to 10.10.10.0/24 table ma
ip route add 192.168.1.0/24 dev eth0 src 192.168.1.20 table pma
ip route add default via 192.168.1.19 dev eth0 table pma
ip route add 10.10.10.0/24 dev eth1 src 10.10.10.20 table ma
ip route add default via 10.10.10.19 dev eth1 table ma
ip route flush cache
Hopefully, the extra spaces will help. I will reply to your other questions separately.
Regards,
> Could you explain the functions?
“ip rule add from …” and “ip rule add to …” directives indicate when Linux should use the corresponding routing table (e.g., “pma” or “ma”) when processing incoming packets. Only packets matching these “rules” will get processed by the corresponding table.
“ip route add …” directives indicate _what_ Linux should do with the packets that matched on the corresponding “ip rule …” directives for a given routing table.
> Again I would like PC in these 2 networks 192.168.1.0/24 and 10.10.10.0/24 could ping each other through this Linux gateway. How the 2 statements could fullfill this requirement?
The “ip rule …” and “ip route …” statements help the Linux gateway route packets through the right interfaces. I think your confusion was due to the fact that wordpress combined the “ip rule” and “ip route” statements onto a single line, which is incorrect. Let me know of this is still unclear.
> How to use statement: “ip route add” and “ip route append” What’s the difference?
Use “ip route add” when you want to add a new routing entry to the TOP of the corresponding routing table.
Use “ip route append” when you want to add a new routing entry to the BOTTOM of the corresponding routing table.
I believe, Linux processes entries in a routing table from TOP to BOTTOM; first match wins.
Regards,
How about this: Suppose I would like some selected nodes (e.g. 10.10.10.100 is the PC of the General Manager and requires a preference connection to the Internet) to reach Internet by going through eth0 with table pma instead of the deault path of the subnet the node belongs to. What statement shoud be added? I understand that there is a PRIORITY paramenter setting for ip rule statements. How could I make use of it?
Defining route priorities is documented here: http://linux.die.net/man/8/ip
Regards,
This is the closest thing to an answer I have! Thank you
I have a question. What if i Don’t always know the network IP of the second interface?
EG
192.168.0.10 Eth0 Gateway 192.168.0.1 This will always be like this.
but then i have a VPN connection where the network IP might change, but i ALWAYS know it will be on link PPP0
Is there a way to route based soley on the interface or link ?
You would have to setup some sort of script to run the policy route commands _after_ the VPN is established. The details of this depend on which VPN software you’re using and also which Linux distribution you’re running.
Regards,
2 snaps up for this thread
Hello, Darien
I’m working on a selective VPN routing on tomato router which is really useful here in China.
Using routing table, getting custom list of host and net to route through VPN is working perfectly with one big problem, I will have to manually add every single IPs for all the site that I found is block to the script and it is very time consuming. I am in the searching for some kind of script or technique to make this process somewhat automated.
(I am not routing all connections through VPN since that it would make everything slow.)
>>Here is the part of the script I’m using
VPNGW=$(nvram get pptp_client_gateway)
##### begin batch route #####
# DNS
route add -host 8.8.8.8 gw $VPNGW
route add -host 8.8.4.4 gw $VPNGW
route add -host 209.244.0.3 gw $VPNGW
route add -host 74.207.242.213 gw $VPNGW
# twitter.com
route add -net 37.61.54.0/24 gw $VPNGW
route add -net 120.88.53.0/24 gw $VPNGW
route add -net 184.169.84.0/24 gw $VPNGW
route add -net 199.16.156.0/22 gw $VPNGW
route add -net 199.59.148.0/22 gw $VPNGW
route add -net 168.143.161.0/24 gw $VPNGW
# fbcdn
route add -net 125.252.224.0/24 gw $VPNGW
route add -net 60.254.185.0/24 gw $VPNGW
route add -net 96.17.69.0/24 gw $VPNGW
route add -net 60.254.175.0/24 gw $VPNGW
…
>>This is part of my #ip route show
50.112.0.0/16 via 10.9.0.1 dev ppp40
174.129.0.0/16 via 10.9.0.1 dev ppp40
54.242.0.0/15 via 10.9.0.1 dev ppp40
23.22.0.0/15 via 10.9.0.1 dev ppp40
15.240.0.0/15 via 10.9.0.1 dev ppp40
54.234.0.0/15 via 10.9.0.1 dev ppp40
54.236.0.0/15 via 10.9.0.1 dev ppp40
50.16.0.0/14 via 10.9.0.1 dev ppp40
10.9.0.1 dev ppp40 proto kernel scope link src 10.9.28.24
XXX.XXX.XXX.XXX via 192.168.1.1 dev vlan3 #XXX is my vpn server
10.0.0.0/24 dev br0 proto kernel scope link src 10.0.0.1
192.168.1.0/24 dev vlan3 proto kernel scope link src 192.168.1.198
127.0.0.0/8 dev lo scope link
default via 192.168.1.1 dev vlan3
I have spent about half a month on this already, trying to find some way to automate this.
Please give some advice or help if you have time,
Noch
I’m thinking about something close to Automatic Failover, if default route is not responsive then if a request is repeated more than 2 times, try the next alternate gateway in order. (not switching completely from default gateway, but only use alternate gateway with some hosts that is blocked on default route)
## Sorry for my English language, I am Thai 🙂
Hi Noch,
What you’re looking for is called “Failover Default Routing”. This article describes the concept in more detail: http://www.linuxquestions.org/questions/linux-networking-3/gateway-router-failover-798109/
It looks like there is a “metric” parameter, that may work for you, in that example.
Also, you can check out this resource, as well: http://lartc.org/howto/lartc.rpdb.multiple-links.html#AEN298
Regards,
Thank you. I’m curious about a mix of IPv4 and IPv6. With an IPv4 address on eth0 and an IPv6 address on eth0, plus an IPv4 address on eth1, what should one setup? Our eth1 interface is generally to a private network, but eth0 goes to both IPv4 and IPv6 default gateways. Thanks! Alan
Hi Alan,
To be honest, I’m not sure, as I haven’t used Linux policy routing with IPv6 interfaces. It looks like Linux does support policy routing over IPv6: http://www.policyrouting.org/PolicyRoutingBook/ONLINE/CH09.web.html http://note-to-self.baker.com/2013/01/03/routing-traffic-control-linux-ipv6-and-iproute2/
That said, it looks like there’s a couple of issues with this: http://itkia.com/ipv6-policy-routing-linux-gotchas/ http://lists.ucc.gu.uwa.edu.au/pipermail/ucc/2010-November/016028.html http://serverfault.com/questions/203387/ipv6-policy-routing-on-linux
Hope that helps.
Regards,
This is a great tutorial! But how do I get it to work for 8 NIC’s?
Simply define unique routes for each interface. As long as there is no overlap, you should be all set.
Regards,
Thanks for the amazingly fast reply! I’ve got another quick question, I’ve noticed that you use static IP’s a lot, is there any way to get this working with DHCP?
It’s possible, but you would have to customize your DHCP scripts to issue the appropriate policy routes once successful.
Also, you would have to make sure your DHCP servers issue leases on non-overlapping subnets.
Lastly, your DHCP servers need to be on separate, logical networks, so that way they don’t see overlapping requests from clients.
Regards,
I’m sorry if my question seems silly (I’m a novice) but would this work if both interfaces were on the same gateway?
As long as the DHCP server assigns a fixed IP address for each interface, then yes.
Regards,
Hello Darien,
Well, I’ve tried your setup and it works great when I want to send data from the client to the server. However, the server (with 8 NIC’s) can’t seem to ping the client…would you happen to know whats wrong?
I would need a lot more information about your setup, in order to troubleshoot.
Regards,
After struggling most of the day, this post came to the rescue. I use Ubuntu 12.04 LTS. Thanks Darien!!!
5 years later and people still sending thanking you notes. And the list is soo long…!
Here’s yet one more: THANK YOU! It saved my day – or rather night.
I have s set-up with a couple of multi-homed hosts belonging to two networks. Each network is connected to the Internet via a separate NAT router. For various reasons I want outbound traffic using one of the routers, but inbound (port forwarded ssh) traffic from the other. After much grief I realised that for inbound traffic to work, it has to go back to the same router as it came from (i.e. exactly the case described here). Not sure why, as I always thought both source and destination were translated: in which case the inside sshd should believe stream originate from the router and send responses back to it, but obviously this is not so. This really is not my area and a few lines of enlightenment would be appreciated (your band-width permitting of course).
Yeah, sshd doesn’t perform any packet routing by itself — that’s the kernel’s job. Therefore, all packets going in and out of your system are handled through the standard routing tables, which is why you saw this type of activity. The persistence you’re describing is typical of any sort of “stateful packet inspection” (SPI) feature on a firewall or NAT device. Specifically, once an initial TCP connection is established and SPI is enabled, the firewall/NAT will maintain a record of the mapping and route all packets pertaining to that flow through to the mapped destination, by default. It’s the same reason why you may need to send initial packets out from the trusted side through a firewall to the untrusted side — in order to receive equivalent packets back through the firewall (from untrusted to trusted side).
Regarding your “stateful route” issue with sshd… yeah, even though you think it would make sense to route outbound SSH packets through a completely different NAT’d route out, the problem with that approach, is that the two NAT devices have _no_ complete view of the TCP flow state, which can cause problems when managing the full TCP flow. In order for NAT to work correctly, a single NAT device needs to see both sides of the flow; otherwise, you will run into the issues you’ve described. The other aspect to this, is that if you tried to configure asymmetric routing through multiple NAT devices for a single ssh session, the original SSH client would have no idea why they’re receiving TCP packets from a completely different IP address than the original IP address that the client sent TCP packets to.
In order to discover these issues, it’s best recommended to run tcpdump or wireshark on both the server and client sides, in order to witness this asymmetric route issue while you try running TCP connection tests — the problem will become much clearer when you have this type of visual as a reference point.
Regards,
Dear Sir,
I need the information regarding multiple interface card.
My Scenario is :
Eth0: 172.X.X.106 / 255.255.255.252 GW 172.X.X.105
Eth1: 192.168.1.222 / 255.255.255.0 GW 192.168.1.1
My Requirement is that:
1) Eth0 traffic out/in gateway 172.X.X.105
2) Eth1 traffic in/out gateway 192.168.1.1
Mean that traffic generate from eth0 forward to its own gate way, same as eth1.
I am using the elastix server (centos 5.7) with two sip account, one on eth0 and other eth1. Port number 5060.
I am confused how to do this.
Waiting for your quick response.
I try , which is mention on above, no success.
Regards
Shahid Iqbal
Hi Shahid,
So, let’s assume this is your network:
eth0 – 172.16.0.106 netmask 255.255.255.252 (/30) eth0’s gateway is: 172.16.0.105
eth1 – 192.168.1.222 netmask 255.255.255.0 (/24) eth1’s gateway is: 192.168.1.1
Here are the corresponding commands:
printf 1\tuplink0\n >> /etc/iproute2/rt_tables printf 2\tuplink1\n >> /etc/iproute2/rt_tables
ip route add 172.16.0.106/30 dev eth0 src 172.16.0.106 table uplink0 ip route add default via 172.16.0.105 dev eth0 table uplink0 ip rule add from 172.16.0.106/30 table uplink0 ip rule add to 172.16.0.106/30 table uplink0
ip route add 192.168.1.0/24 dev eth1 src 192.168.1.222 table uplink1 ip route add default via 192.168.1.1 dev eth1 table uplink1 ip rule add from 192.168.1.0/24 table uplink1 ip rule add to 192.168.1.0/24 table uplink1
ip route flush
Regards,
I am using elastix server, i put all above mention, which was sent to me, but nothing happen.
Basically using 5060 port to forward traffic to sip ip. but sip registration is failed.(eth0)
I dont know how to manage.
I have a Linux box wtih 4 interfaces: eth0 and eth1 serve as default gateways of network A and network B respectively. eth2 and eth3 serve as gateways to the Internet through 2 different ADSL. I want the incoming traffic from network A to go through eth2 to reach Internet, the incoming traffic of network B to go through eth3 to reach Internet. Meanwhile network A and network B could communicate with each other through their default gateways, namely eth0 and eth1 respectively. Forwarding is enabled in this box. I have added 2 entries to the rt_tables. I know that I have to add some ip rules and ip routes statements. But how could I manage to have 2 different default gateways to route Internet traffic for 2 different networks while enabling the inter-network communication between this 2 networks. Please advise.
Please provide example network subnets for: 1) network A 2) network B 3) eth2 ADSL network 4) eth3 ADSL network
Regards,
Configuration of the Linux Box…
eth0: 192.168.1.20
eth1: 10.10.10.20
eth2: 192.168.6.20 (reach Internet through 192.168.6.1)
eth3: 192.168.0.20 (reach Internet through 192.168.0.1)
LAN A: 192.168.1.0/24 Default Gateway: 192.168.1.20
LAN B: 10.10.10.0/24 Default Gateway: 10.10.10.20
If you copy the following ASCII text and paste it to the Notepad, you could see a better picture.
________
+—————+ /
| ISP 1 | /
+——–+ (MA) +——+
+—————-+ | | gw 192.168.0.1| /
| LAN A | +———–+–+ +—————+ /
| 192.168.1.0/24 +—-+ eth0 eth3 | /
+—————-+ | | |
| Linux Box | |
+—————-+ | | | Internet cloud
| LAN B +—-+ eth1 eth2 | |
| 10.10.10.0/24 | +———–+–+ +—————+ |
+—————-+ | | ISP 2 | \
+——–+ (PMA) +—-+
|gw 192.168.6.1 | \
+—————+ \
\
\________
If I issue the following commands, it seems that traffic between LAN A and LAN B is not possible as they all go their own gateway as listed in the routing tables, ma and pma respectively. The traffic could eventually go to the Internet. I desire LAN A and LAN B could communicate with each other. Meanwhile LAN A and LAN B could go to the Internet through the assigned gateways.
ip rule add from 192.168.1.0/24 table ma
ip rule add to 192.168.1.0/24 table ma
ip rule add from 10.10.10.0/24 table pma
ip rule add to 10.10.10.0/24 table pma
I understand that I may need to additionally issue the following comands:
ip route add 192.168.1.0/24 dev eth0 table ma
ip route add default via 192.168.0.1 dev eth0 table ma
ip route add 10.10.10.0/24 dev eth1 table pma
ip route add default via 192.168.6.1 dev eth1 table pma
Still don’t work. Is there a precedence order for “ip rule” and “ip route”? If so, which one goes first? Getting confused…
Appreciate you advise in advance.
Oops! It seems that WordPress doesn’t like the graphic in ASCII text. Sorry. I hope you understand anyway.
Hi Darien,
Once again, great post! I always come back to this post about linux pbr 🙂
I have a question for you.. when I connect to a vpn my default route is overwritten and all my internet traffic goes through the vpn. So first question, is this normal behaviour? Why does the vpn client feels the need to route ALL traffic through the vpn and not just the traffic destined to the vpn’s subnet.
Second question, what is the best way to prevent this behaviour and route only the vpn’s subnet traffic through the vpn?
Here’s what I did but had no success:
–before vpn:
$ ip r
default via 192.168.1.1 dev wlan0 proto static
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.3 metric 2
–after vpn:
$ ip r
default dev ppp0 proto static
10.72.27.161 dev ppp0 proto kernel scope link src 10.72.27.166 <—-(*)
so my obvious thought was to delete the default route for the ppp0 and recreate the old defaut for the wlan0. This worked for the internet traffic however, traffic destined for the 10.72.27.0/24 (vpn subnet) is still routed via the default route (wlan0) EVEN though the line that I have pointed out with the (*) above STILL exists in the routing table. Shouldn't that line be routing traffic through the vpn even though the default route points to something else?
What am I missing?
thank you!
Hmm it seems like your website ate my first comment (it was extremely long) so I guess I’ll just sum it up what I wrote and say, I’m
thoroughly enjoying your blog. I too am an aspiring blog writer but I’m still new to everything. Do you have any tips for rookie blog writers? I’d certainly appreciate it.
You are so cool! I don’t think I have read something like this before. So good to discover another person with some original thoughts on this subject matter. Seriously.. thank you for starting this up. This web site is one thing that is required on the internet, someone with a little originality!
Hi Darien. Your article is great but didn’t help in my problem.
I have 2 machines directly connected to each other (im doing this because i have to test one protocol in a controlled environment). Both machines (A and B) have two NICS [etho (A) eth0 (B) and eth1(A) eth1(B)].
A:
eth0 address: 10.1.1.1/24 gateway: 10.1.1.2
eth1 address: 10.1.2.1/24 gateway: 10.1.2.2
B:
eth0 address: 10.1.1.2/24 gateway: 10.1.1.1
eth1 address: 10.1.2.2/24 gateway: 10.1.2.1
I have configured two routing tables (T1 and T2) like i saw here and in other websites. But the problem is that ping command shows “Host Unreachable” in both ways and command ARP -a shows the message “Incomplete”.
Here is my configs in machine A (in B are the same with, of course, corrects IPs):
IP Rule
32762: from all to 10.1.2.1 lookup T2
32763: from all to 10.1.1.1 lookup T1
32764: from 10.1.2.1 lookup T2
32765: from 10.1.1.1 lookup T1
IP route show table T1
default via 10.1.1.2 dev eth0 src 10.1.1.1
10.1.1.0/24 dev eth0 scope link src 10.1.1.1
IP route show table T2
default via 10.1.2.2 dev eth1 src 10.1.2.1
10.1.2.0/24 dev eth1 scope link src 10.1.2.1
I have been following this website http://lartc.org/howto/lartc.rpdb.multiple-links.html#AEN2970 and im doing something wrong. Can u help me in my problem?
This webpage finally explained how to make policy routing accessible. Thank you so much. In about 30 minutes I fixed all of my routing woes.
Hello Darien.
Thank you very much for this article very clear.
Can you add to your article the corresponding configuration for / etc / network / interfaces file?
because I do not know what to put too.
Take care of yourself
Hi,
I’ve unexpectedly found myself needing to create multiple default routes, I’m using haproxy with keepalived on two servers, so I have:
server1 server2 floating ip required gateway
eth0: 192.168.52.251 192.168.52.252 192.168.52.250
eth1: 192.168.51.251 192.168.51.252 192.168.51.250
eth2.11 192.168.100.3 192.168.100.4 192.168.100.5 192.168.100.250
eth2.12 192.168.101.3 192.168.101.4 192.168.101.5 192.168.101.250
eth2.13 192.168.102.3 192.168.102.4 192.168.102.5 192.168.102.250
eth3.5 192.168.50.3 192.168.50.4 192.168.50.5 192.168.50.250
keepalived also includes the rest of the 192.168.100.0/24, 192.168.101.0/24 and 192.168.102.0/24 as virtual_ipadress_excluded.
192.168.51.0/24 and 192.168.52.0/24 outbound traffic is SNAT’d to these virtual_ipaddress_excluded addresses
(this is just test at the moment, the addresses on eth2.11, eth2.12, and eth2.13 will be changed to public ip’s for production)
your instructions show ip rule add being used with the physical ip addresses, what would I do in this case? do I create rules using the physical addresses, the virtual addresses, or do I need to create rules for both physical and virtual addresses?
thanks for any advice you can give.
damn, formatting makes that hard to read,
server1:
eth0: 192.168.52.251
eth1: 192.168.51.251
eth2.11 192.168.100.3
eth2.12 192.168.101.3
eth2.13 192.168.102.3
eth3.5 192.168.50.3
server2:
eth0: 192.168.52.252
eth1: 192.168.51.252
eth2.11: 192.168.100.4
eth2.12: 192.168.101.4
eth2.13: 192.168.102.4
eth3.5: 192.168.50.4
floating ips:
192.168.52.250
192.168.51.250
192.168.100.5
192.168.101.5
192.168.102.5
192.168.50.5
required gateways:
192.168.100.250
192.168.101.250
192.168.102.250
192.168.50.250
hopefully this makes it clearer.
I believe you will need to treat each interface (physical and virtual) that has a unique IP as a separate interface for policy routing. The commands should work on both interface types, I believe. You will have to likely experiment to see what works for you.
Regards,
— Darien
Hi,
after a bit of experimentation, I’ve found I can get it done in just a few rules.
192.168.50.0/24, 192.168.51.0/24, and 192.168.52.0/24 don’t require anything beyond their default settings
192.168.100.0/24 has the default route.
so it’s just rules for all the VIP’s on 192.168.101.0 and 192.168.102.0
create two tables in /etc/iproute2/rt_tables : 101 table_101 and 102 table_102
ip route add default via 192.168.101.250 dev eth2.12 table table_101
ip rule add from 192.168.101.0/24 table table_101
ip route add default via 192.168.102.250 dev eth2.13 table table_102
ip rule add from 192.168.102.0/24 table table_102
outbound from any of these addresses now use the correct interface. not created inbound rules yet, still working on that, but with what i have to test with, from haproxy to firewall to adsl router, internet bound traffic is getting NATted 3 times, so currently working on inbound traffic problems upstream from this machine first.
hopefully this helps anyone else who finds themselves needing to create rules for large ranges of ip’s.
I certainly wasn’t looking forward to the possibility of creating tables and rules for every individual ip.
Hey Darien,
I just wanted to thank you for the post and for keeping up with replies to everyone asking questions, even after 5 years! This post was exactly what I needed. I used this with my Raspberry Pi to simulate hosts for a Cisco lab network (and linked back): http://toddjames.io/post/64816553890/using-a-raspberry-pi-to-simulate-hosts-on-a-network
Todd
Nice little project but for what you need it to do gns3 or virtualbox would be much better and more convenient
Thanks! Found your blog while watching inbound http requests have responses sent out of the wrong interface.
You’re the man.
Great article. I was wondering if you can help me with a peculiar problem I am having.
I have a Tomato Router with internet connection and public dynamic IP, x.x.x.x and 192.168.1.254 private interface.
x.x.x.x (Tomato) 192.168.1.254
I have port forwarded from (5000) that router using DNAT to a machine called Frodo 192.168.1.2. Frodo has a OpenVPN client that connects to VPN service so that my torrenting some other activities remain anonymous from Frodo. This has VPN address of 10.4.x.x (also dynamic).
192.168.1.2 (Fordo) 10.4.x.x
So prior to the VPN tunnel being brought up. When I get a request into Tomato on port 5000 that is forwarded to Frodo and all is well, because the default route on Frodo without the VPN running is 192.168.1.254 (default GW)
Now when the OpenVPN system starts up on Frodo the default route changes to 10.4.1.1 since it eats up all trafffic. So now when a forwarded request reaches Frodo via the DNAT, it cannot make it back since its going back to internet IP:port because the default GW is via the VPN and so the forwaded port doesn’t respond to external traffic.
I suspect I having a similar issue to what you suggested. I have tried whats on your post but it didn’t do the trick.. any suggestions?
I have two interfaces eth0 and tun0 (VPN)
Thanks!
PS: All ports and IPs appearing in this post are fictitious. Any resemblance to real ports and ips, is purely coincidental.
Hi Manojav,
Yes, this is a known issue with OpenVPN. You’re essentially looking to setup what’s known as a “split-tunnel VPN”. If you search Google for “openvpn split tunnel”, you will like find additional resources on how to accomplish this, such as: http://swimminginthought.com/routing-traffic-vpn/
If those search results don’t yield anything useful, then my next recommendation would be to contact the OpenVPN support forums to look for a possible answer — mention or search for the “split tunnel” keywords, as well.
Regards,
Darien,
agreed this is due to default routes the vpn sets up. but the issue is that when i send a request that originates in the 192.168.1.x network the response is sent back over that interface, this only occurs for DNATed connections that need to traverse my NAT firewall and reach the internet through the default network GW 192.168.1.254 rather than the VPN default GW.
so OpenVPN folks send me back to ip route folks and ip route folks bounce me back to openvpn.. suspect this is something I cannot do..
what i need is to have a default gw as the vpn for all requests except for those to or from a certain port.. i tried setting fwmarks as well. but i dont know how to debug those packets..
Thanks
Got it. Yeah, the bouncing back and forth between the forums can get quite old. Even though the 192.168.1.x network is internal, you could try treating that as an external network within the OpenVPN configuration, so that it thinks there is a split tunnel route altogether. Part of the problem, is I think OpenVPN may shim the kernel in a way that makes packet debugging a real pain…
Regards,
Big D!
Thanks to you I have worked it out.
ip route del table LAN
ip route add 192.168.1.0/24 dev eth0 src 192.168.1.2 table LAN
ip route add default via 192.168.1.252 dev eth0 table LAN
ip rule add from 192.168.1.2/32 table LAN
ip rule add to 192.168.1.2/32 table LAN
ip route add default table LAN via 192.168.1.252
ip rule add fwmark 1 table LAN
ip route flush cache
iptables -t mangle -I PREROUTING -p tcp –sport SRC_PORT_OF_SERVICE -j MARK –set-mark 1
I stuck this in a shell script and called it on route-up on the opevpn connection.
I am not sure what I am doing here.. but seems to work, new connections head out to the VPN GW. port forwarded connections also work.. do i have some hole open that I am not seeing?
thanks a bunch! (for nothing :-))
Une solution permettant d’avoir plusieurs réseaux différents. Donc avec plusieurs passerelles. Et ceci sur une seule machine ayant pour chaque réseau une carte réseau dédié.
Cette solution fonctionne pour des réseaux public ou comme ici pour un réseau de classe C.
Cette solution peut fonctionner également derrière une architecture VLAN.
# The loopback network interface
auto lo
iface lo inet loopback
# The network 0 interface
auto eth0
iface eth0 inet static
address 192.168.0.1
netmask 255.255.255.0
broadcast 192.168.0.255
network 192.168.0.0
gateway 192.168.0.254
up /sbin/ip route add default via 192.168.0.254 dev eth0 table network0
up /sbin/ip rule add from 192.168.0.0/24 table network0
post-down /sbin/ip rule del from 192.168.0.0/24 table network0
post-down /sbin/ip route del default via 192.168.0.254 dev eth0 table network0
# dns-* options are implemented by the resolvconf package, if installed
### Z1.Z2.Z3.Z4 U1.U2.U3.U4 : Les IPs de vos serveurs DNS
dns-nameservers Z1.Z2.Z3.Z4 U1.U2.U3.U4
dns-domain yourdomain.tld
dns-search yourdomain.tld
auto eth0:0
iface eth0:0 inet static
address 192.168.0.2
netmask 255.255.255.0
auto eth0:1
iface eth0:1 inet static
address 192.168.0.3
netmask 255.255.255.0
# The network 1 interface
auto eth1
iface eth1 inet static
address 192.168.1.1
netmask 255.255.255.0
broadcast 192.168.1.255
network 192.168.1.0
### gateway 192.168.1.254
up /sbin/ip route add default via 192.168.1.254 dev eth1 table network1
up /sbin/ip rule add from 192.168.1.0/24 table network1
post-down /sbin/ip rule del from 192.168.1.0/24 table network1
post-down /sbin/ip route del default via 192.168.1.254 dev eth1 table network1
# dns-* options are implemented by the resolvconf package, if installed
### Z1.Z2.Z3.Z4 U1.U2.U3.U4 : Les IPs de vos serveurs DNS
dns-nameservers Z1.Z2.Z3.Z4 U1.U2.U3.U4
dns-domain yourdomain.tld
dns-search yourdomain.tld
auto eth1:0
iface eth1:0 inet static
address 192.168.1.2
netmask 255.255.255.0
auto eth1:1
iface eth1:1 inet static
address 192.168.1.3
netmask 255.255.255.0
# The network 2 interface
auto eth2
iface eth1 inet static
address 192.168.2.1
netmask 255.255.255.0
broadcast 192.168.2.255
network 192.168.2.0
### gateway 192.168.2.254
up /sbin/ip route add default via 192.168.2.254 dev eth2 table network2
up /sbin/ip rule add from 192.168.2.0/24 table network2
post-down /sbin/ip rule del from 192.168.2.0/24 table network2
post-down /sbin/ip route del default via 192.168.2.254 dev eth2 table network2
# dns-* options are implemented by the resolvconf package, if installed
### Z1.Z2.Z3.Z4 U1.U2.U3.U4 : Les IPs de vos serveurs DNS
dns-nameservers Z1.Z2.Z3.Z4 U1.U2.U3.U4
dns-domain yourdomain.tld
dns-search yourdomain.tld
auto eth2:0
iface eth2:0 inet static
address 192.168.2.2
netmask 255.255.255.0
auto eth2:1
iface eth2:1 inet static
address 192.168.2.3
netmask 255.255.255.0
Enjoy ! 😉
Awesome article! This was the only way my vpn would even start with two nics a while back. Thanks!
Here is my goal: Seperate network traffic. I want use my paid vpn service provider to encrypt the connection for a single application. The rest of my network applications I want to operate outside of my vpn. It doesn’t matter to me how I manage this goal: I have two network cards available – the default one in my motherboard and I have a spare pci NIC. I will install that extra nic if I need to but I’d prefer to somehow use virtual interfaces. I want to be able to use port forwarding for the rest of my network applications so they can hit the internet so i access them remotely I’m using Ubuntu 12.04. I’ve been playing with the bind and local options but still haven’t been able to figure this out. i’m sure this is similiar to what others have been saying on this thread, but I think this is slightly different and more basic. Do virtual interfaces work the same way as physical nics?
Thanks Darien
The virtual NICs should operate the exact same way as the physical NICs. The major challenge, will be understanding all the custom IPs that your application uses, which will require custom routes for each netmask.
Regards,
— Darien
Thanks for the advice Darien and for getting back to me so quickly. I will answer sooner next time, I was just away for the week. So it seems like I shouldn’t use the second NIC and just use virtual interfaces. Let’s say for educational sake that
ETH0 = 4.4.4.4
and ETH0:1 = 4.4.4.5
I want to run rtorrent + rutorrent in 4.4.4.5. Everything else is fine at 4.4.4.4. Can you help me configure the routes?
It really depends on the applications. You need to figure out all the TCP/UDP ports that the torrent clients use, in order to develop a same routing policy. My best recommendation, would be to run your torrent clients while capturing full PCAPs in order to see what the common ports/IPs are, which would be the basis for your defined routes.
I am not intimately familiar with all the protocols used by standard torrent clients.
Regards,
— Darien
Hey Darien,
I have another follow up question about my above post. I think I have everything working properly.
I have two interfaces-
So right now everything is working fine with the below routing command in webmin custom commands.
sudo echo “1 admin” >> /etc/iproute2/rt_tables; sudo ip route add 4.4.4.0/24 dev eth0 src 4.4.4.4 table admin; sudo ip route add default via 4.4.4.1 dev eth0 table admin; sudo ip rule add from 4.4.4.4/32 table admin; sudo ip rule add to 4.4.4.4/32 table admin; sudo ip route flush cache; sudo ip rule show
Would there be any benefit to creating a table with the commands below for the vpn?
sudo echo “1 vpn” >> /etc/iproute2/rt_tables; sudo ip route add 4.4.4.0/24 dev eth0:1 src 4.4.5.4 table vpn; sudo ip route add default via 4.4.4.1 dev eth0:1 table vpn; sudo ip rule add from 4.4.5.4/32 table vpn; sudo ip rule add to 4.4.5.4/32 table vpn; sudo ip route flush cache; sudo ip rule show
Thanks!
I’d recommend changing the “1 vpn” string to “2 vpn”, because you already have “1 admin” defined, and your follow on commands will overwrite, rather than append to the routing policy.
Aside from that, the VPN commands you have seem like it would work. My recommendation would be to try and see, as I know others were able to get it to work on virtual interfaces successfully.
Regards,
— Darien
Thanks Darien I updated my code for the vpn table- sudo echo “2 vpn2″ >> /etc/iproute2/rt_tables; and called the table vpn2
0: from all lookup local
32762: from all to 4.4.5.4 lookup vpn2
32763: from 4.4.5.4 lookup vpn2
32764: from all to 4.4.4.4 lookup admin
32765: from 4.4.4.4 lookup admin
32766: from all lookup main
32767: from all lookup default ”
This is my output. So these two connections should be totally isolated at this point.
No my question is which becomes the default route for any internet connection? I want 4.4.4.4 admin table to be the default route, but I always thought lower numbers in linux get the priority.
Thanks Darien this page is awesome!
Yes, you’d want to swap the ordering, so that the admin table gets lower priority numbering (which means it gets evaluated first).
Aside from that, I’d recommend testing it out, to make sure everything is working, by running tcpdump, accordingly.
Regards,
— Darien
Darien. I use two interfaces in parallel, basically setup as your example. Now the problem I am having is that when the primary interface (The one with the default gateway in the main routing table) is gone (And there is no default gateway route in the main table anymore), I am not able to do any dns lookups anymore. (Traceroute still finds the dns server though) Are there perhaps other ip rules to add for this case or should you fall back to adding the secondary gateway to the main table?
Basically, policy routing is not going to help you achieve failover routing, which is what you’re looking to do. I would recommend doing a Google search (terms: “linux failover routing”) to see what options are available for Linux fail over routing. It is likely you will need to install a custom tool to accomplish this.
Regards,
— Darien
Darien, this is a fantastic article! I came back to read it again, as I am having problems configuring my interfaces with Debian Wheezy. I made it work on my previous server, but sadly the motherboard died and when trying to replicate I am failing. One thing I noticed for a start, the routing table with one network card looks like this:
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
239.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0
The default route appears first unlike previous time. Does that matter? From there on, the ip route add and ip rule add commands do get executed, but no change on routing table occurs. Any pointers?
Thanks a lot,
Ivo
Hi Ivo,
It sounds like the kernel doesn’t have policy routing compiled in, which means you may need to custom compile your kernel with that support, accordingly.
Regards,
Well I checked my /boot/config and policy routing is enabled. I do thing is something to do with the way interfaces are configured at boot time, but seem unable to find exactly what is wrong. I have two interfaces and use same gateway. My /etc/network/interfaces:
auto eth0
allow-hotplug eth0
iface eth0 inet static
address 192.168.0.11
netmask 255.255.255.0
network 192.168.0.0
broadcast 192.168.0.255
gateway 192.168.0.1
post-up ip route add 192.168.0.0/24 dev eth0 src 192.168.0.11 table uplink0
post-up ip route add default via 192.168.0.1 dev eth0 table uplink0
post-up ip rule add from 192.168.0.11/32 table uplink0
post-up ip rule add to 192.168.0.11/32 table uplink0
auto eth1
allow-hotplug eth1
iface eth1 inet static
address 192.168.0.21
netmask 255.255.255.0
network 192.168.0.0
broadcast 192.168.0.255
# gateway 192.168.0.1
post-up ip route add 192.168.0.0/24 dev eth1 src 192.168.0.21 table uplink1
post-up ip route add default via 192.168.0.1 dev eth1 table uplink1
post-up ip rule add from 192.168.0.21/32 table uplink1
post-up ip rule add to 192.168.0.21/32 table uplink1
# #dns-* options implemented by resolver
dns-nameservers 8.8.8.8 8.8.4.4
dns-search mydomain.com
If I enable gateway on eth 1, interface does not get configured. If I disable eth0 gateway, the routing table gets configured with routing table configured for both interfaces correctly, but no gateway therefore unable to reach internet at all. If however I boot with the above configuration, the routing table is:
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
239.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0
and I am back to the original problem, all output goes through eth0. Any hints?
Thanks a lot for taking your time.
WOW!!! This is brilliant!! This helped me a lot with an engineering project that I’ve been on involving a Linux server. You are a life saver!!! Thank you so much!
thanks for your help. you save my job. you are boss.
Wyjątkowo interesujący tekst, polecam wszystkim
Let me add more info to this excellent article:
https://access.redhat.com/site/solutions/53031
That is a alternative to the problem: it allows the traffic to pass through the default route dev.
Thanks for the info and it helped. This info is nicely putup.
One of the most rewarding nuggets from Google mining I’ve struck in a long time
Than you for sharing you knowledge! this helped me a lot:)
Thanks for the writeup! I have some additional information that may be of use to some people (and if nothing else maybe you). I didn’t read all comments so if this is here I am sorry for the repeat information. But here goes. Basic idea is this: I have a /29 block and I also have a dynamic IP (well, dynamic in that it isn’t considered static – it only changes when the modem/router the ISP provides is different [e.g., replaced] or change at the DSLAM). But while I have two NICs on the server, I designate eth1 to my internal network (which includes 10.0.0.0/8 and also an IPv6 /125 [which really isn’t internal but I filter it to make it act as such]). For my own domains I use one of my static IPs. However, I also host a friend’s website (only website, not any other service) and I use the dynamic IP for that. This means that eth0 has my static IP, its default gateway and also my dynamic IP. But until now it used my static IP default gateway. Well I’m here to report that indeed this method of yours works on the same NIC. So instead of using eth1 and eth0 (or however you have it named) you can use eth0 for both, eth1 for both or more generall ethN for both (or if you use systemd then it might be something else entirely …).
The other information is for systemd and specifically Red Hat based distributions (not sure of others as I only maintain servers that are Red Hat based). It may be of value even if with modifications. In order to maintain the rules/routes across reboots (rare as they may be) here is how to do it. You have:
directory: /etc/sysconfig/network-scripts
files: /etc/sysconfig/network-scripts/ifcfg-ethN (where N is each NIC).
Add the files:
/etc/sysconfig/network-scripts/rule-ethN (e.g. rule-eth0) and /etc/sysconfig/network-scripts/route-ethN (e.g., route-eth0). Here is an important bit though. IF you have in the ifcfg-ethN file a route for the IP in question you must remove it from the ifcfg-file (after adding it to route-ethN). If you don’t it won’t work properly. So what is the content of the two files?
For route-ethN you have two ways to do it. I use the older way as it is as if you pass each line to the ip command. I’ll use dynroute for the default route, dynnet for the network and dynip for the IP. Replace as necessary:
dynnet dev eth0 src dynip table dynamic
default via dynroute dev eth0 table dynamic
For rule-ethN you have:
from all to dynip table dynamic
from dynip lookup dynamic
Note: I had a problem initially as ‘from dynip table’ and ‘to dynip table’ in the rule-ethN file. This might be however that I (stupidly) used reload on network rather than restarting it. This seems to not have deleted the rules (as I tend to test command line first) and so did not work. Then I forgot I had the route for the dynamic IP in the ifcfg file. This means when I restarted network (after deleting the rules!) the route was not there at all (only static IP route). So whether from all to is necessary or not I don’t know. What I do know is ip rule showed the above which is why I put that in the file.
Thanks again for the writeup! Hope the above is helpful.
Cheers.
I simply want to say thanks as this helped me solve a problem I had.
I’m probably overlooking something but I can’t get the routing to work the way I want it. What I have is a Linux server with 3 nic’s. Eth0 is the local network, eth1 is ISP1 and eth2 is ISP2.
I case you are wondering why there are two ISP’s, well, Provider 1 does allow that you run your own webserver, mailserver and mysql database that can be accessed from the internet. And I can add as many port forwarding rules in the modem/router they delivered. But the internet speed is much slower then from Provider 2.
Provider 2 however doesn’t allow me to have a webserver and so on and did password protect the modem/router. Therefore only outgoing traffic is possible, but very fast.
ifconfig on the Linux box shows:
eth0 192.168.1.150 netmask 255.255.255.0
eth1 10.0.0.150 netmask 255.255.255.0
eth2 10.0.1.150 netmask 255.255.255.0
and route -n shows:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.1.254 0.0.0.0 UG 0 0 0 eth2
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.0.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
Finally, iptables -t nat -n –list -v shows (only showing the postrouting):
Chain POSTROUTING (policy ACCEPT 7 packets, 490 bytes)
pkts bytes target prot opt in out source destination
1 84 MASQUERADE all — * eth1 0.0.0.0/0 0.0.0.0/0
1 84 MASQUERADE all — * eth2 0.0.0.0/0 0.0.0.0/0
Then I did setup the routing and the rules:
ip route add 10.0.0.0/24 dev eth1 src 10.0.0.150 table RT1
ip route add default via 10.0.0.138 table RT1
ip rule add from 10.0.0.150 table RT1
ip route add 10.0.1.0/24 dev eth2 src 10.0.1.150 table RT2
ip route add default via 10.0.1.254 table RT2
ip rule add from 10.0.1.150 table RT2
ip route show:
default via 10.0.1.254 dev eth2
10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.150
10.0.1.0/24 dev eth2 proto kernel scope link src 10.0.1.150
127.0.0.0/8 dev lo scope link
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.240
ip route show table RT1:
default via 10.0.0.138 dev eth1
10.0.0.0/24 dev eth1 scope link src 10.0.0.150
ip route show table RT2:
default via 10.0.1.254 dev eth2
10.0.1.0/24 dev eth2 scope link src 10.0.1.150
ip rule show:
0: from all lookup local
32764: from 10.0.1.150 lookup RT2
32765: from 10.0.0.150 lookup RT1
32766: from all lookup main
32767: from all lookup default
So far everything looks to go the right way. That is, if the forwarding in the modem/router from provider 1 goes to 10.0.0.150 the answer goes back over eth1.
The webserver that runs on the Linux box can now be found from the internet and still from the local network.
But now the problem. If I want to access a Windows machine with remote desktop that Windows machine has ip-address 192.168.1.x. So I added a rule in the modem/router from provider 1 to forward port 3389 to 192.168.1.x and added a static routing. It now shows:
Network: 192.168.1.0 Mask: 255.255.255.0 GW 10.0.0.150
With tcpdump I can see that there is traffic going from an external address to the Windows machine but there isn’t going any answer back to the external address. Not on eth1 and not on eth2.
What is needed to be able to access a machine on the internal network from extern over the connection from provider 1??
As I have turned on masquarading I thought that any answer that goes over eth1 would look like it came from 10.0.0.150 but tcpdump doesn’t show that.
I’m stuck!! Please somebody help!!
Hi Collette,
> So I added a rule in the modem/router from provider 1
> to forward port 3389 to 192.168.1.x and added a static routing.
> It now shows:
>
> Network: 192.168.1.0 Mask: 255.255.255.0 GW 10.0.0.150
I think that’s where the problem is. With this route, you’re essentially telling your Linux system to route all traffic destined to 192.168.1.0/24 *through* the upstream RT1 gateway, which isn’t what you want.
Instead, I think you’ll have to write a custom iptables rule (ROUTING) to essentially perform the port-forwarding activity that you’re trying to accomplish. Look for examples for port-forwarding in iptables on how to accomplish this.
Regards,
— Darien
Hi Darien,
I tried to do forwarding on the server like so:
iptables -A PREROUTING -t nat -i eth1 -p tcp –dport 3389 -j DNAT to 192.168.1.x
and removed the forwarding in the modem/router from provider 1. Then I added a forwarding for port 3389 to forward it to 10.0.0.150 (eth1 on the server). Without that forwarinding nothing comes in on the server.
But the problem seems to be that the answer isn’t coming from 10.0.0.150 but from 192.168.1.x and therefore it goes over the default gw and that is the other provider.
incoming: xxxx -> 192.168.1.x on eth1
outgoing: 192.168.1.x -> xxxx on eht2
greetings,
Collette
Hi Collette,
I think there’s a way to specify in the PREROUTING directive *which* interface the reply traffic should go back out through. If that’s not possible, you may need another iptables directive specified, in order to fix this.
Regards,
Here’s how matching of interface works in iptables. In this case I’m explaining outgoing packets.
I no longer use this for the fact that my service changed (and I now have a static IP block whereas before I had a single static IP), but when I did nat for my local network (specifically: my server was my router – so 10.0.0.0/8 was the localnet, my IP was at the time 64.174.104.17 – and so all internal machines were in the local subnet, all out going packets went out as if 64.174.104.17 and incoming would go to the machine that is associated.. the server itself had the 64. ip bound to its external NIC and sent and received that way… you get the idea). I only had two NICs on the server (still only have two NICs but point is I don’t have 3 like the one asking for help.. only one ISP and so less complicated setup). Regardless, if my masquerade nic was eth0 (was), then I had the following rule:
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
The -o eth0 matches packets going out eth0 (and -i is not allowed with POSTROUTING as -i is for incoming, to explain that more generally). I tried following the setup (as yesterday) but I was quite exhausted (yesterday and today too) … I’ve not really mapped out Collete’s setup and I don’t intend to try (see previous point). But hopefully the iptables rule explains the interface part. The layout of my network was basically this, at least for the above:
server eth0 -> isp router/modem -> Internet
server eth1 -> 10.0.0.0/8 net attached to switch (that was part of the router/modem that eth0 is also connected to)
client machine eth0 -> attached to switch on isp router/modem. no external ip, only one in the 10.0.0.0/8 block. default gw was eth1 of server (let’s say 10.0.0.1). Client could not connect (as I filtered) to 64.174.104.17 (the server public IP) but could the internal NIC. Similar, packets claiming to be from 64.174.104.17 were filtered as the server shouldn’t be sending packets to itself with its IP. Therefore the only way to connect to (maybe better put: external packets could only reach) the internal machines is through the server (obviously there was connections and obviously the server is allowed to communicate with the internal machines so in a sense they aren’t internal, but you get the idea, hopefully). The only requirement was that the client had a way to be on the server’s internal network, so that the server could NAT for it (as it had no public IP.. I had the modem/router in bridged mode, to be specific). The other client was set up the same way.
Hopefully I explained that all correctly and well enough. If not, feel free to respond and I’ll try to elaborate (or fix).
I found a solution (or maybe, the solution).
I gave the machines on the local lan that must be accessible from the internet an extra ip-address (192.178.1.x).
On Linux boxes this is easy (eth 0:0 192….) and even dhcp can still be used for the primary local lan address. But on Windows it only works if you use fixed addresses.
The server must have this secondary address as well on eth0. So ifconfig eht0:0 192.178.1.150
Only 1 Windows machine, the Linux server and 1 other Linux machine have addresses on this secondary network. In the router/modem from internet provider 1 I can just forward port 3389 to 192.178.1.2 (the Windows machine). The answers now come back from 192.178.1.2 but still go out over the default gw (provider 2).
To get them out the right way, I added a rule to like so: ip rule add from 192.178.1.0/24 lookup RT1
Now the answer goes out over the connection from provider 1. All the machines on the local lan can still find each other over the primary network addresses (192.168.1.x). And of course, that Windows machine with the 2 ip-addresses will still use the default gw if I want to access the internet from it. Meaning that it will go over the connection from provider 2.
Based on this article I wrote a python script to automate running these commands. This script determines the ip address information needed and then runs the iproute2 utilities to add the routing rules. I hope this helps someone else out. It is available at https://github.com/r0bst3r/Sourceroute/
Thank you very much for your great how-to. It really helped me, without facing any problem!
Thanks for this post!
Hi,
I have a different requirement. The article clearly tells that the both NICs should be tagged with IP addresses. But if i want to change the IP address of any of interface using ifconfig command, that change wont be reflected in the IP rules. So i will be again need to set new rules for that changed interface address. Is there any way to automatically update the rules with ifconfig command? Or is there a way to set the rules based on device MAC address rather than device IP address?
Thank you
Hi Ron,
Yes, what you are looking for, is to define a “trigger script” that fires everytime one of the IPs change on any of the interfaces. This is usually the case if one of the interfaces is getting IP assignments from a DHCP server. As such, I’d recommend reading the man page of dhclient, as I believe there is a section that talks about how to define such scripts.
Then, whenever the IP address changes, dhclient would call your script, which would redefine the rule sets to allow the correct policy route.
Regards,
— Darien
Darien, thanks for this writeup. I used your blog a few years ago to configure three web hosting dual-nic ubuntu servers during an IP transition. It worked like a charm as they responded to both public ip addresses.
I am now thoroughly confused about when a solution like this would be necessary otherwise. Any clarification would be appreciated!
Lets say I have an NFS server on a local network (default gw, eth0) and decide to give it a second nic (eth1) on another subnet. I then throw a second nic (eth1) into my pc with the same subnet on the same vlan. If I were to try to ping the NFS via my eth1 to it’s eth1, would it try to respond via eth0? If this erroneous behavior occured, NFS would still work because I am directly connecting to it, correct (unlike an ICMP which wouldn’t be the same connection, but a generated response?) Or does all of this only matter when there is a third variable – access came in from an ip address not in the network (from the default gw) ? What if both eth1 were on the same subnet but with different IP addresses, would the NFS server still attempt to respond from eth0? Ugh, thanks in advance!
Hi Adrian,
There is some detail missing from your description, so I’m going to try to filling the blanks by stating some assumptions. You mentioned that you have a PC with two different Ethernet cards, but you don’t explicitly state how eth0 is configured on the PC. On the PC, I’m going to assume that eth0 is connected to the same exact local network that eth1 is connected to.
If this assumption is correct, then you would need to have proper policy routing enabled on both the PC and the NFS server, in order to avoid the type of overlapping problems that you’re encountering. If you have proper policy routing just on the NFS server, then you will still run into the issues you’ve described.
Let me know if this makes sense.
Regards,
Not sure if this helps, but I wrote a python script that helps automate the pbr setup in linux. https://github.com/r0bst3r/Sourceroute
Thanks, Robbie! This script should be very useful to new readers.
Regards,
Thank you Robbie. This is excellent!
I am not the worlds best programmer, so if you find any bugs please let me know or reply on the Github page
Darien, my apologies for not providing enough detail. The setup described is only hypothetical. My confusion lies in understanding *when* I would need to use policy based routing.
server:
eth0: 10.1.1.1 255.255.255.0 gw 10.1.1.254
eth1: 192.168.1.1 255.255.255.0
client:
eth0: 10.1.1.2 – 255.255.255.0
eth1: 192.168.1.2 – 255.255.255.0
server and client eth0 share switch1
server and client eth1 share switch2
Without policy routing, if I ping client(eth1) -> server(eth1) would server attempt to respond via eth0 because eth0 has the default gw? Or.. Does the server know to respond via eth1 because the request came from an ip on its eth1 local network (my client’s eth1) ? Is all of this only really a problem if it’s coming from ip addresses *not* in either of their subnets, ie. that need to be reached via a GW. hence routing configuration telling them “which” gateway ?
Thanks for taking the time to reply to my nonsense. 🙂
> Without policy routing, if I ping client(eth1) -> server(eth1) > would server attempt to respond via eth0 because eth0 > has the default gw?
No, it shouldn’t. This is because you’re trying to reach a resource on your local subnet directly, in which case, the gateway never gets involved. That said, to be absolutely sure, you’d need to provide a copy of your routing table (e.g., ‘netstat -anr’ output). This is because I need to see how far *down* in the routing rules you have your local routes set, compared to the default route going to your gateway. Usually, your default gateway route entry is set as the *last* entry within your routing table (which is the basis of my original assumption).
> Is all of this only really a problem if it’s coming from ip > addresses *not* in either of their subnets, ie. that need > to be reached via a GW. hence routing configuration telling > them “which” gateway ?
Yes, precisely. It’s only a problem when you have two distinct Ethernet interfaces that both need to be publicly, Internet routable *beyond* the local subnet (which requires working with at least one upstream gateway).
Regards,
Thank you! That is exactly what I was after. I owe you a beer.
> This is because I need to see how far *down* in the routing rules you have your local routes set, compared to the default route going to your gateway. Usually, your default gateway route entry is set as the *last* entry within your routing table (which is the basis of my original assumption).
This seems to be my problem, my default route always appears on top, gets evaluated first and traffic goes out the wrong interface. How does default gateway get added to the bottom of the ruleset rather than the top?
> How does default gateway get added to the > bottom of the ruleset rather than the top?
You can delete your default route using the ‘route’ command, and then re-add it using the ‘route’ command. That operation will move your new route rule to the bottom of the ruleset, accordingly.
Regards, Darien
On Tue, Aug 18, 2015 at 2:35 PM, Darien Kindlund's Blog wrote:
>
Hi Darien,
I have two nics, eth0 and eth1. eth0 is only ever used locally, I run an IDS and captures all traffic hitting my DSL port. eth1 is used for everything else. One common gateway and following configuration:
# ip addr
4: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 60:a4:4c:65:cb:c8 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.88/24 brd 192.168.0.255 scope global eth0
valid_lft forever preferred_lft forever
5: eth1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 68:05:ca:1b:60:ae brd ff:ff:ff:ff:ff:ff
inet 192.168.0.11/24 brd 192.168.0.255 scope global eth1
valid_lft forever preferred_lft forever
# netstat -anr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
With this configuration, locally accessing ssh using eth1 (configured as listener) I occasionally get traffic coming out of eth0 and sessions hangs. I used tcpdump and checked ssh sessions going out of wrong interface sometimes. This is local traffic, and the above Gateway configuration should not affect local traffic. Nonetheless I did try configuring policy routing, assuming I will instruct the kernel to send eth1 traffic through eth1 interface and eth0 to eth0 interface. I entered the following commands, after creating two tables, bro and services:
sudo ip route add 192.168.0.0/24 dev eth1 src 192.168.0.11 table services
sudo ip route add default via 192.168.0.11 dev eth1 table services
sudo ip rule add from 192.168.0.11/32 table services
sudo ip rule add to 192.168.0.11/32 table services
sudo ip route add 192.168.0.0/24 dev eth0 src 192.168.0.88 table bro
sudo ip route add default via 192.168.0.88 dev eth0 table bro
sudo ip rule add from 192.168.0.88/32 table bro
sudo ip rule add to 192.168.0.88/32 table bro
I flush the routing cache, but netstat reports no change:
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
# ip rule show
0: from all lookup local
32762: from all to 192.168.0.88 lookup bro
32763: from 192.168.0.88 lookup bro
32764: from all to 192.168.0.11 lookup services
32765: from 192.168.0.11 lookup services
32766: from all lookup main
32767: from all lookup default
Policy routing is enabled in the kernel. Why I am unable to modify routing table? Assuming I will enable port forwarding on my router I will not be able to access ssh from internet if this behaviour continues.
Thanks.
> Why I am unable to modify routing table?
Policy routing is an *overlay* on top of the normal routing table that Linux provides. Just because you enabled policy routing, does not mean that the output of ‘netstat -anr’ will change, at all.
Try looking at it another way. Think of the ‘ip rule show’ output as the *first layer* of routing logic that the kernel uses (this is a rough approximation). If you re-read my original article, I state: “The gist is that your normal main routing table appears as entry 32766 in this list. (This would be the normal route table you’d see when you type netstat -anr.)”
So in your example:
# ip rule show
0: from all lookup local 32762: from all to 192.168.0.88 lookup bro 32763: from 192.168.0.88 lookup bro 32764: from all to 192.168.0.11 lookup services 32765: from 192.168.0.11 lookup services 32766: from all lookup main [<– this is where your 'netstat -anr' table is defined] 32767: from all lookup default That is why your changes to policy routing don't affect 'netstat -anr' output; it's because 'netstat -anr' is a small subset of the total routing table that Linux actually uses.
Lastly, as to why policy routing isn't working in your scenario, it's because both interfaces are sharing the same Class-C subnet. Essentially, all packets are getting evaluated (and matched) in the same order, which means that because both rules are trying to match on 192.168.0.0/24, they'll always match and route to eth1. In order to fix this, I think you'll need to specify the /32 address directly for each, so that each declaration is distinct.
So use: sudo ip route add 192.168.0.11/ 32 dev eth1 src 192.168.0.11 table services
And: sudo ip route add 192.168.0.88/ 32 dev eth0 src 192.168.0.88 table bro
… accordingly. Let me know if that works.
Regards, Darien
On Tue, Aug 18, 2015 at 4:40 PM, Darien Kindlund's Blog wrote:
>
Thanks for the reply Darien,
You are correct about the netstat command, I completely forgot it actually reads the main table, not the additional ones I added. I discovered were my issues come from as I also lost external connectivity.
I made a silly mistake, the line
sudo ip route add default via 192.168.0.11 dev eth1 table services
should actually be
sudo ip route add default via 192.168.0.1 dev eth1 table services
as gateway is located at 192.168.0.1 not .11 which is the interface on eth1!
I also changed the /24 prefix as you suggested to /32 and I could see with tcpdump that traffic is fine, no more ssh traffic going to wrong interface.
A s a side note, suppose eth0 has an ip 192.168.0.88. Then ping -I eth0 google .com command gives destination host unreachable answer. Whereas ping -I 192.168.0.88 google.com works just fine.
Thanks again for the great article,
I owe you a beer.
Hi,
I want to connect to network through two interfaces using ethernet and usb. but only either of one interface is ping is succesful i.e., the interface which is set as default gateway.
on Secondary interface ping is responding with destination host not reachable.
$ ping -I eth0 http://www.google.co.in
(192.168.200.100) icmp_seq=1 Destination Host Unreachable
(192.168.200.100) icmp_seq=2 Destination Host Unreachable
(192.168.200.100) icmp_seq=3 Destination Host Unreachable
Please help
Hi,
Thanks to Anonymous (posted on August 20, 2015 at 2:07 pm) I tested ot based on his reply. its woking fine.
Thanks for the wonder post and answer.
thanks a lot!! you really saved my life!!!
I just wanted to let you know that this article really helped me out. I was trying to set up a network monitoring system based on a Linux OS for my capstone class in the networking program I am in, and was having absolutely no luck getting my server to communicate on two separate VLANs via two NICs. The steps in your article here helped me understand the problem and actually route data correctly to the correct NIC based on the traffic type. Thank you so much for this extremely helpful, extremely well-written article!
Great article, I found it really helpful! Anyway I still have a doubt, related to some issues I’m having while setting up a small cluster (four nodes running Ubuntu 14): let me explain the scenario.
Every node has 2 NICs, a built-in one that allows internet browsing and connects to enterprise network. The second NIC is used only for data moving among cluster nodes, all connected by mean of a switch.
The configuration looks like this (node 1):
# Built-In interface – Internet (192.168.10.0/24)
auto eth0
allow-hotplug eth1
iface eth0 inet dhcp
dns-nameservers 192.168.100.1
# Cluster LAN
auto eth1
iface eth1 inet static
address 192.168.30.20
netmask 255.255.255.0
network 192.168.30.0
up ip route add 192.168.30.0/24 dev eth1 src 192.168.30.20 table mgmt
up ip route add default via 0.0.0.0 dev eth1 table mgmt
up ip rule add from 192.168.30.20/32 table mgmt
up ip rule add to 192.168.30.20/32 table mgmt
Notice that eth1 does not need a gateway, since it is only used for the cluster. Similar configuration in each node.
$ ip rule show
0: from all lookup local
32764: from all to 192.168.30.20 lookup mgmt
32765: from 192.168.30.20 lookup mgmt
32766: from all lookup main
32767: from all lookup default
$ netstat -anr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.10.254 0.0.0.0 UG 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
192.168.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.30.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
Well, I can ping and ssh nodes on network 192.168.10.0/24, but I can not on network 192.168.30.0/24. I suspect this has to do with the default gateway, though I couldn’t find a solution.
Do you have any suggestion?
You need to fix the following line; change this line:
> up ip route add default via 0.0.0.0 dev eth1 table mgmt
To be this:
> up ip route add default via 192.168.30.20 dev eth1 table mgmt
That’s the core problem.
I tried with that fixing, and configured every node by setting the rule above: if I’m not wrong, it defines a way out to network 192.168.30.0/24. still, I can’t ping PCs in that network:
$ ping -I eth1 192.168.30.19
PING 192.168.30.19 (192.168.30.19) from 192.168.30.20 eth1: 56(84) bytes of data.
From 192.168.30.20 icmp_seq=1 Destination Host Unreachable
From 192.168.30.20 icmp_seq=2 Destination Host Unreachable
note that PCs .19 and .20 are linked through a switch, might that be the cause?
this is really puzzling me: I thought it was easier to set up such a simple network!
thanks for your time, anyway.
You don’t configure every node with that exact same rule. You need to specify eth1 IP address in that rule, like:
> up ip route add default via dev eth1 table mgmt
So for .19 it’s:
up ip route add default via 192.168.30.19 dev eth1 table mgmt
For .20 it’s:
up ip route add default via 192.168.30.20 dev eth1 table mgmt
Hope this helps!
yep! that’s exactly what I’ve done, but I still have the same problem.
Ok, try getting rid of these lines:
up ip route add 192.168.30.0/24 dev eth1 src 192.168.30.20 table mgmt
up ip route add default via 0.0.0.0 dev eth1 table mgmt
^ see if that helps during troubleshooting
Hi Darian!
I found your post while searching for a solution for my (possible) routing problem.
I’ve got a Debian system with 2 NICs installed:
eth0 Link encap:Ethernet Hardware Adresse 00:50:56:bf:01:1c
inet Adresse:192.168.160.81 Bcast:192.168.160.255 Maske:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metrik:1
RX packets:683 errors:0 dropped:0 overruns:0 frame:0
TX packets:52 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:54805 (53.5 KiB) TX bytes:3636 (3.5 KiB)
eth1 Link encap:Ethernet Hardware Adresse 00:50:56:bf:81:20
inet Adresse:192.168.167.81 Bcast:192.168.167.255 Maske:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metrik:1
RX packets:10713 errors:0 dropped:0 overruns:0 frame:0
TX packets:1021 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:2124878 (2.0 MiB) TX bytes:150596 (147.0 KiB)
My interfaces config looks like this:
[…]
# The primary network interface
allow-hotplug eth0
iface eth0 inet static
address 192.168.160.81
netmask 255.255.255.0
gateway 192.168.160.250
dns-nameservers 192.168.160.30 192.168.160.32
allow-hotplug eth1
iface eth1 inet static
address 192.168.167.81
netmask 255.255.255.0
dns-nameservers 192.168.167.30 192.168.167.32
post-up ip route add 192.168.167.0/24 dev eth1 src 192.168.167.81 table internal
post-up ip route add default via 192.168.167.254 dev eth1 table internal
post-up ip rule add from 192.168.167.81/32 table internal
post-up ip rule add to 192.168.167.81/32 table internal
I can ping both interfaces from my client which resides in the 192.168.130.0/24 internal network
and I can also ping my client from the server.
192.168.167.0/24 is an internal server network, its gw is a router for internal networks.
The 192.168.160.0/24 is a DMZ network, the gw is the firewall.
It allows outgoing traffic on port 80,443,53 and so on.
My problem is: if I establish an outgoing connection to e.g. google.com it resolves the name
to the correct ip address but holds on status SYN_SENT like this:
tcp 0 1 192.168.160.81:40859 xx.xx.xx.xx:80 SYN_SENT
The firewall log tells me: connection allowed (nothing blocked)
Maybe you have an idea what’s going wrong.
Thank you very much!
Hi There,
In terms of next steps, I’d recommend running a capture on both eth0 and eth1 *while* you try to establish an outgoing connection to google.com. It seems like the source IP is listed correctly, but you really haven’t verified whether that initial SYN packet is leaving eth0 or eth1 correctly — which is what a network packet capture would confirm.
Regards,
— Darien
Hi Darien,
it uses the eth0 interface (tcpdump):
12:03:52.956581 IP 192.168.160.81.35572 > 192.168.160.30.domain: 14485+ A? http://www.google.com. (32)
12:03:52.956603 IP 192.168.160.81.35572 > 192.168.160.30.domain: 48948+ AAAA? http://www.google.com. (32)
12:03:52.993079 IP 192.168.160.30.domain > 192.168.160.81.35572: 14485*- 1/0/0 A 216.58.211.4 (48)
12:03:52.997230 IP 192.168.160.30.domain > 192.168.160.81.35572: 48948*- 1/0/0 AAAA 2a00:1450:4016:804::2004 (60)
12:03:52.997424 IP 192.168.160.81.38036 > muc03s13-in-f4.1e100.net.http: Flags [S], seq 796689412, win 29200, options [mss 1460,sackOK,TS val 41979 ecr 0,nop,wscale 7], length 0
12:03:53.957807 IP 192.168.160.81.33921 > 192.168.160.30.domain: 22381+ PTR? 4.211.58.216.in-addr.arpa. (43)
12:03:53.995072 IP 192.168.160.81.38036 > muc03s13-in-f4.1e100.net.http: Flags [S], seq 796689412, win 29200, options [mss 1460,sackOK,TS val 42229 ecr 0,nop,wscale 7], length 0
12:03:54.358723 IP 192.168.160.30.domain > 192.168.160.81.33921: 22381*- 4/0/0 PTR muc03s13-in-f4.1e100.net., PTR muc03s13-in-f4.1e100.net., PTR muc03s13-in-f4.1e100.net., PTR muc03s13-in-f4.1e100.net. (123)
12:03:55.999070 IP 192.168.160.81.38036 > muc03s13-in-f4.1e100.net.http: Flags [S], seq 796689412, win 29200, options [mss 1460,sackOK,TS val 42730 ecr 0,nop,wscale 7], length 0
12:04:00.007061 IP 192.168.160.81.38036 > muc03s13-in-f4.1e100.net.http: Flags [S], seq 796689412, win 29200, options [mss 1460,sackOK,TS val 43732 ecr 0,nop,wscale 7], length 0
12:04:08.023065 IP 192.168.160.81.38036 > muc03s13-in-f4.1e100.net.http: Flags [S], seq 796689412, win 29200, options [mss 1460,sackOK,TS val 45736 ecr 0,nop,wscale 7], length 0
12:04:24.055072 IP 192.168.160.81.38036 > muc03s13-in-f4.1e100.net.http: Flags [S], seq 796689412, win 29200, options [mss 1460,sackOK,TS val 49744 ecr 0,nop,wscale 7], length 0
But there is no ack to the syn request.
The firewall log tells me:
Time: Today 12:03:53
Description: Accepted on rule 5
Interface Name: eth4
Interface Direction: inbound
To: muc03s13-in-f4.1e100.net
From: 192.168.160.81
Service: http
Action: Accept
Destination: muc03s13-in-f4.1e100.net
Inzone: Internal
Out-Zone: External
Protocol: TCP
Rule: 5
Source Port: 38036
Destination Port: 80
Service ID: http
Service Name: http
Source: 192.168.160.81
Type: Log
XlateDPort: 0
XlateDst:
XlateSport: 0
XlateSrc: WAN2
Regards,
Volker
Thanks, Volker.
Next, can you provide output from the following commands:
# netstat -anr
# ip rule show
Regards,
— Darien
Thank you for your help!!
This is the output:
Kernel-IP-Routentabelle
Ziel Router Genmask Flags MSS Fenster irtt Iface
0.0.0.0 192.168.160.250 0.0.0.0 UG 0 0 0 eth0
192.168.160.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.167.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
0: from all lookup local
32764: from all to 192.168.167.81 lookup internal
32765: from 192.168.167.81 lookup internal
32766: from all lookup main
32767: from all lookup default
Hi Volker,
It looks like you have policy routing set up and configured correctly. At this point, it sounds like your upstream gateway (192.168.160.250) might be instituting some sort of block. You may want to check the gateway to see if there is any additional blocking policy implemented — perhaps run tcpdump on the gateway to see if packets are making it out beyond the gateway.
Regards,
— Darien
Thank you very much Darien for a maintaining this article all these years. It still continues to serve so many people, and I am one of them.
Darien,
Thanks for this page, some really good info here. I have a problem and was wondering if you could advice please?.
I have 2 IP blocks, each with their own gateways. One block (the old one that was in place for years) works fine, the new one however does’t. Ignoring the old block because it works, This is what I’m seeing
* New block can’t ping anywhere
* I can SSH out on the new IPs by using -I to ssh, e.g ssh -b NEw.IP.HERE root@somehost
* traceroute -I bond2 shows first hop to be the gateway for the “OLD” range.
I am using routing tables as per your example
Useful info here
[root@host~]# netstat -anr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 XXX.146.130.129 0.0.0.0 UG 0 0 0 bond1 <—old/default gateway
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 p1p1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 p1p3
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 p1p4
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 bond0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 bond1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 bond2
192.168.128.0 0.0.0.0 255.255.255.0 U 0 0 0 bond0
192.168.128.0 192.168.128.250 255.255.128.0 UG 0 0 0 bond0
XXX.146.130.128 0.0.0.0 255.255.255.240 U 0 0 0 bond1 <—Old IP block
XXX.146.154.0 0.0.0.0 255.255.255.0 U 0 0 0 bond2 <—New IP block
Using a specific routing table "called rt2" should I expect to see the gateway to be listed here?. I do see it with the following commands (as expected)
[root@host~]# ip rule show
0: from all lookup local
32764: from all to XXX.146.154.0/24 lookup rt2
32765: from XXX.146.154.0/24 lookup rt2
32766: from all lookup main
32767: from all lookup default
[root@fw0 ~]# ip route
default via 213.146.130.129 dev bond1
169.254.0.0/16 dev p1p1 scope link metric 1006
169.254.0.0/16 dev p1p3 scope link metric 1008
169.254.0.0/16 dev p1p4 scope link metric 1009
169.254.0.0/16 dev bond0 scope link metric 1010
169.254.0.0/16 dev bond1 scope link metric 1011
169.254.0.0/16 dev bond2 scope link metric 1012
192.168.128.0/24 dev bond0 proto kernel scope link src 192.168.128.250
192.168.128.0/17 via 192.168.128.250 dev bond0
XXX.146.130.128/28 dev bond1 proto kernel scope link src XXX.146.130.132
XXX.146.154.0/24 dev bond2 proto kernel scope link src XXX.146.154.128
btw I’ve tried a windows box with hardcoded values for IP/gateway/netmask using ONE IP only on the nic e.g. a standalone windows 10 laptop and it all worked fine, ping, browsing etc. Therefore I know the upstream gateway is fine.
Again on the actual router I cant ping using that interface, however I can SSH/wget etc no issues on the router but its not working from another box thats configured to use this Linux router with the NEW iprange.
Hi Centos7,
You’ve neglected to provide the output from any “ip route add” commands you issues, in order to create the “rt2” table.
I need to see all “ip route add” commands corresponding to that table. If you don’t remember issuing these commands, then that explains why policy routing isn’t working for you. Re-read my blog post and look at where the “ip route add” commands are issued for further detail.
Lastly, the “netstat -anr” command only shows you the routing table of the “main” table. By default, it will *not* show you the routing table of the “rt2” table (which is why I’m asking for a dump of all your “ip route add” commands).
Regards,
— Darien
Hi Darien,
I didnt add the commands because I dont type them. The way they are setup is via the config files for the interface.
So in my example I have
/etc/sysconfig/network-scripts/route-bond2
The contents of the file are (again the first octal is masked here but its a real number in the config)
default via XXX.146.154.2 dev bond2 table rt2
I’ve checked its right via the following command
[root@host~]# ip route show table rt2
default via XXX.146.154.2 dev bond2
[root@host~]#
so it looks like for that table rt2, its configured. Assuming I’ve understood it all right
Hi Centos7,
Negative. There are some missing commands here. In my original blog post, I indicated that you need to issue something like:
ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
ip route add default via 10.10.70.254 dev eth0 table admin
In your example “route-bond2” file, you just have the 2nd entry listed.
You need to replace the contents of “route-bond2” file with the following:
XXX.146.154.0/24 dev bond2 src XXX.146.154.128 table rt2
default via XXX.146.154.2 dev bond2 table rt2
^ If this doesn’t work, then please explicitly list the following for me:
bond1:
– ip address
– netmask
– gateway
bond2:
– ip address
– netmask
– gateway
Regards,
— Darien
Right i’ll give your suggestion a whirl. I’ve assumed this file in my config was doing what you wrote but I guess its not
[root@hostnetwork-scripts]# cat /etc/sysconfig/network-scripts/rule-bond2
from XXX.146.154.0/24 table rt2
to XXX.146.154.0/24 table rt2
Sorry i’ve been at this so long its all got very confusing. I’ll try your suggestion and report back
Hi Centos7,
No problem! Keep in mind, you may want to issue the commands directly using “ip route add”, rather than relying on the configuration files to work correctly — for just the first time.
At least, the idea is: 1) get policy routing working first using CLI commands and then 2) figure out how to make it persist upon reboots (via the conf files). Trying to combine everything together usually makes it more confusing.
Good luck!
— Darien
Unfortunately it didn’t work (or i’m doing it wrong, which is more likely).
I removed the extra config files leaving JUST the ifcfg-bond1 and ifcfg-bond2 files in place.
Before any changes I see the following after a reboot, the ONLY thing i have in place is the setup of “rt2″ in /etc/iproute2/rt_tables
[root@host ~]# ip route show
default via XXX.146.130.129 dev bond1
169.254.0.0/16 dev p1p1 scope link metric 1006
169.254.0.0/16 dev p1p3 scope link metric 1008
169.254.0.0/16 dev p1p4 scope link metric 1009
169.254.0.0/16 dev bond0 scope link metric 1010
169.254.0.0/16 dev bond1 scope link metric 1011
169.254.0.0/16 dev bond2 scope link metric 1012
192.168.128.0/24 dev bond0 proto kernel scope link src 192.168.128.250
192.168.128.0/17 via 192.168.128.250 dev bond0
XXX.146.130.128/28 dev bond1 proto kernel scope link src XXX.146.130.132
XXX.146.154.0/24 dev bond2 proto kernel scope link src XXX.146.154.128
[root@host ~]# ip rule show
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
OK first things first, the config you wanted
bond1: XXX.146.130.132 netmask 255.255.255.240 gateway XXX.146.130.129
bond2: XXX.146.154.128 netmask 255.255.255.0 gateway XXX.146.154.2
The config looks like this for each interface
[root@host ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond2|grep -v ^#
DEVICE=bond2
IPADDR=XXX.146.154.128
NETMASK=255.255.255.0
ONBOOT=yes
BOOTPROTO=static
TYPE=BOND
BONDING_OPTS=”mode=active-backup miimon=100 downdelay=200 updelay=200″
NM_CONTROLLED=”no”
IPV6INIT=no
DEFROUTE=NO
[root@fw0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond1|grep -v ^#
DEVICE=bond1
IPADDR=XXX.146.130.132
NETMASK=255.255.255.240
NETWORK=XXX.146.130.128
BROADCAST=XXX.146.130.143
ONBOOT=yes
BOOTPROTO=static
TYPE=unknown
GATEWAY=XXX.146.130.129
BONDING_OPTS=”mode=active-backup miimon=100 downdelay=200 updelay=200″
IPV6INIT=no
NM_CONTROLLED=”no”
Now I’ll test each interface to ensure they can ping their respective gateways
[root@host ~]# ping -c5 -I bond2 XXX.146.154.2
PING XXX.146.154.2 (XXX.146.154.2) from XXX.146.154.128 bond2: 56(84) bytes of data.
64 bytes from XXX.146.154.2: icmp_seq=1 ttl=255 time=0.715 ms
64 bytes from XXX.146.154.2: icmp_seq=2 ttl=255 time=0.753 ms
64 bytes from XXX.146.154.2: icmp_seq=3 ttl=255 time=0.649 ms
64 bytes from XXX.146.154.2: icmp_seq=4 ttl=255 time=0.835 ms
64 bytes from XXX.146.154.2: icmp_seq=5 ttl=255 time=0.711 ms
— XXX.146.154.2 ping statistics —
5 packets transmitted, 5 received, 0% packet loss, time 4002ms
rtt min/avg/max/mdev = 0.649/0.732/0.835/0.067 ms
[root@fw0 ~]# ping -c5 -I bond1 XXX.146.130.129
PING XXX.146.130.129 (XXX.146.130.129) from XXX.146.130.132 bond1: 56(84) bytes of data.
64 bytes from XXX.146.130.129: icmp_seq=1 ttl=255 time=0.649 ms
64 bytes from XXX.146.130.129: icmp_seq=2 ttl=255 time=0.726 ms
64 bytes from XXX.146.130.129: icmp_seq=3 ttl=255 time=0.758 ms
64 bytes from XXX.146.130.129: icmp_seq=4 ttl=255 time=0.731 ms
64 bytes from XXX.146.130.129: icmp_seq=5 ttl=255 time=0.699 ms
— XXX.146.130.129 ping statistics —
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.649/0.712/0.758/0.047 ms
These are the commands I’ve typed in
ip route add XXX.146.154.0/24 dev bond2 src XXX.146.154.128 table rt2
ip route add default via XXX.146.154.2 dev bond2 table rt2
ip rule add from XXX.146.154.0/24 table rt2
ip rule add to XXX.146.154.0/24 table rt2
This is how things look after I’ve done that
[root@host ~]# ip rule show
0: from all lookup local
32764: from all to XXX.146.154.0/24 lookup rt2
32765: from XXX.146.154.0/24 lookup rt2
32766: from all lookup main
32767: from all lookup default
[root@host ~]# ip route show
default via XXX.146.130.129 dev bond1
169.254.0.0/16 dev p1p1 scope link metric 1006
169.254.0.0/16 dev p1p3 scope link metric 1008
169.254.0.0/16 dev p1p4 scope link metric 1009
169.254.0.0/16 dev bond0 scope link metric 1010
169.254.0.0/16 dev bond1 scope link metric 1011
169.254.0.0/16 dev bond2 scope link metric 1012
192.168.128.0/24 dev bond0 proto kernel scope link src 192.168.128.250
192.168.128.0/17 via 192.168.128.250 dev bond0
XXX.146.130.128/28 dev bond1 proto kernel scope link src XXX.146.130.132
XXX.146.154.0/24 dev bond2 proto kernel scope link src XXX.146.154.128
[root@host ~]# ping -I bond2 XXX.146.154.2
PING XXX.146.154.2 (XXX.146.154.2) from XXX.146.154.128 bond2: 56(84) bytes of data.
64 bytes from XXX.146.154.2: icmp_seq=1 ttl=255 time=0.742 ms
64 bytes from XXX.146.154.2: icmp_seq=2 ttl=255 time=0.733 ms
[root@fw0 ~]# ping -I bond2 XXX.146.130.129
PING XXX.146.130.129 (XXX.146.130.129) from XXX.146.154.128 bond2: 56(84) bytes of data.
64 bytes from XXX.146.130.129: icmp_seq=1 ttl=255 time=2.66 ms
64 bytes from XXX.146.130.129: icmp_seq=2 ttl=255 time=0.679 ms
64 bytes from XXX.146.130.129: icmp_seq=3 ttl=255 time=0.799 ms
64 bytes from XXX.146.130.129: icmp_seq=4 ttl=255 time=0.839 ms
So I can still ping the new gateway on bond2, as well as the “default” gateway but I cant ping anywhere else like before.
Sorry forgot to provide this from final config dump
[root@host ~]# ip route show table rt2
default via xxx.146.154.2 dev bond2
xxx.146.154.0/24 dev bond2 scope link src xxx.146.154.128
Hi Centos7,
Okay, from your output, here’s what I think your network settings look like:
bond1
ip: XXX.146.130.132
mask: 255.255.255.240
gw: XXX.146.130.129
bond2
ip: XXX.146.154.128
mask: 255.255.255.0
gw: XXX.146.154.2
So one problem is that your “ip route show table rt2” command output looks incorrect.
It should say the following:
XXX.146.154.0/24 dev bond2 scope link src XXX.146.154.128
default via XXX.146.154.2 dev bond2
^ Notice that the order of these statements matters. Currently, these two statements are reversed.
Moreover, your “ip route show” output is also incorrect.
It should say the following:
32764: from all to XXX.146.154.128/32 lookup rt2
32765: from XXX.146.154.128/32 lookup rt2
^ Notice how your original .0/24 should be .128/32 here.
I’d recommend just commenting out all of the statements in your “rule-bond2” and “route-bond2” files, REBOOTING, and then once base networking is initialized, issue the following commands MANUALLY:
ip route add XXX.146.154.0/24 dev bond2 src XXX.146.154.128 table rt2
ip route add default via XXX.146.154.2 dev bond2 table rt2
ip rule add from XXX.146.154.128/32 table rt2
ip rule add to XXX.146.154.128/32 table rt2
^ Again, the ORDER of how you issue these commands MATTERS. Please issue these four commands in this order, accordingly.
Once you have confirmed that policy routing is working for the *manual* commands, THEN you can start to enable them in the start-up configuration files.
Hope this helps.
— Darien
FYI, I had updated my original reply slightly, as there was an unintentional error in my original recommendation; see the corrected reply above.
OK so I removed (renamed) the rule and route config files and rebooted, they are no longer being used and only what I type in is being used.
First test, add line gateway= into bond2 config, this BREAKS bond1 BUT full internet access is possible on bond2 so gateway DOES work if its being routed correctly, the issue is my box 100%.
So now I removed that line as it was only a test.
Task 1, fix “ip route show rt2” order, but it is IMPOSSIBLE to control the order, I know you said above its important but I dont appear to be able to control that. I cleared rt2 and tried adding each line first
[root@host ~]# ip route show table rt2
***nothing returned because no entries in rt2, start by adding first line (not the default one)
[root@host ~]# ip route add XXX.146.154.0/24 dev bond2 src XXX.146.154.128 table rt2
[root@host ~]# ip route show table rt2
XXX.146.154.0/24 dev bond2 scope link src XXX.146.154.128
*** 1 entry (so far so good)
[root@host ~]# ip route add default via XXX.146.154.2 dev bond2 table rt2
[root@host ~]# ip route show table rt2
default via XXX.146.154.2 dev bond2
XXX.146.154.0/24 dev bond2 scope link src XXX.146.154.128
*** 2 entry but wrong order, clear routing table and try adding in other order
[root@host ~]# ip route del default via XXX.146.154.2 dev bond2 table rt2
[root@host ~]# ip route show table rt2
XXX.146.154.0/24 dev bond2 scope link src XXX.146.154.128
[root@host ~]# ip route del table rt2 XXX.146.154.0/24 dev bond2 scope link src XXX.146.154.128
[root@host ~]# ip route show table rt2
blank now again. so lets add “default” line first
[root@host ~]# ip route add default via XXX.146.154.2 dev bond2 table rt2
[root@host ~]# ip route add XXX.146.154.0/24 dev bond2 src XXX.146.154.128 table rt2
[root@host ~]# ip route show table rt2
default via XXX.146.154.2 dev bond2
XXX.146.154.0/24 dev bond2 scope link src XXX.146.154.128
now 2 enties but order is exactly same, doesnt matter which order i input.
Distro btw is Centos 7 (almost same as RHEL7).
Task2, fix up “ip rule show” output
** Start no mention of my rules, good time to add.
[root@host ~]# ip rule show
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
Add both rules verbatim as you had them, i think distro removes /32 part when printing rule but its same i think
[root@host ~]# ip rule add from all to XXX.146.154.128/32 lookup rt2
[root@host ~]# ip rule add from XXX.146.154.128/32 lookup rt2
[root@host ~]# ip rule show
0: from all lookup local
32764: from XXX.146.154.128 lookup rt2
32765: from all to XXX.146.154.128 lookup rt2
32766: from all lookup main
32767: from all lookup default
Test with new setup, same not working. I cant make any outbond connection on new interface, but I can ssh INTO new IP as before
Hi Centos7,
Hrm, I see what you mean. It looks like there are some odd artifacts in your kernel’s implementation of policy routing then.
It’s either that, or perhaps you inadvertently have firewall rules set which are blocking all outbound connections on bond2?
The reason I’m asking, is that I know some stock Linux distributions attempt to “auto-build” firewall rules, based upon what’s set in the conf files. As such, it’s possible you have some sort of script that pulls whatever is set in the “gateway=” lines and pre-builds firewall rules to only allow that sort of outbound traffic.
Can you check iptables to see if there’s anything set there?
Issue the following commands as root to see if there’s any firewall rules set:
# sudo /sbin/iptables -L
# sudo /sbin/iptables -L -t nat
Regards,
— Darien
nothing in the firewall. I’ve got complete control in it and added lots of rules that I’ve written.
I used to have rules in the nat table for port forwarding some other IPs in this range, but when I could get that working because the network isn’t working I removed them.
This IP that ends in .130 is the static IP for this machine, the others in the range would be for me to use for forwarding traffic but this one isnt natt’d to anywhere else. Its reserved for this machine only. I dont believe I need any rules in the nat table as I’m not forwarding traffic. On the filter table in the INPUT and OUTPUT chains there are rules to allow use of the interface.
INPUT chain, relevant rule..
656K 449M ACCEPT all — * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
Nothing in output chain.
Actually I’ve tested without netfilter running at all, to eliminate firewall from picture, no dice. So I’m pretty sure firewall is not interfering.
Hi Centos7,
Okay, to be crystal clear, once you got policy routing set up on “bond2”, you’re saying that:
1) External systems can correctly reach this server via “bond2” and all response traffic is correctly flowing out through the “bond2” interface.
2) But, when you try to initiate any outbound connections from the server, all traffic automatically goes out the “bond1” interface (unless you explicitly specify the “bond2” interface).
^ Is this a fair assessment?
The policy routes you’ve defined will only send outbound traffic out “bond2” if and only if the destination IP is within the XXX.146.154.0/24 network block.
By default, all other outbound traffic should be going through “bond1” anyway.
Therefore, if you issue a “ping XXX.146.154.2”, it should (by default) go out via “bond2” and traffic should return back via “bond2”. Is that not happening?
As you perform these connectivity checks, I’d recommend running “tcpdump -ni bond1” and “tcpdump -ni bond2” to actually observe the packets leave and return via the different interfaces, in order to confirm the issue here.
Lastly, are “bond1” and “bond2” interfaces directly mapped to Ethernet ports? Or are these virtual interfaces that map to possibly a set of Ethernet ports?
Regards,
— Darien
Darien,
OK Let me explain right now how things are looking because I guess with the messages things have gotten a little confused 🙂
Right now
bond1 has has an ip that is on a different subnet to bond2, it also has a seperate gateway. This gateway is the default gateway for the system. I can ping any host on the internet on this interface and everything works fine.
bond2 has 14 IPs (a block). One IP is for the interface and the others are supposed to be used for other machines on the network.
I can as of this moment ssh OUT on bond2 without issue. I can SSH in via an IP on bond2 without issue. I cant ping ANYWHERE on bond2. bond2 has ONLY the IP
I think the policy based routing is working because there is not routing/rule configuration files in place, and when the system is fresh booted I can’t ssh out OR ssh in, until I run the routing commands by hand. Once I run those commands I can ssh in and out without issue.
I dont think this is a firewall rule issue because I turn off iptables and I still cant ping on that interface
e.g. (notice firewall off and bond1 works but bond2 fails)
[root@host ~]# service iptables stop
Redirecting to /bin/systemctl stop iptables.service
[root@host ~]# ping -c5 -I bond1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from XXX.146.130.132 bond1: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=60 time=7.80 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=60 time=7.87 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=60 time=7.87 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=60 time=7.87 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=60 time=7.83 ms
— 8.8.8.8 ping statistics —
5 packets transmitted, 5 received, 0% packet loss, time 4008ms
rtt min/avg/max/mdev = 7.805/7.851/7.879/0.115 ms
[root@host ~]# ping -c5 -I bond2 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from XXX.146.154.128 bond2: 56(84) bytes of data.
— 8.8.8.8 ping statistics —
5 packets transmitted, 0 received, 100% packet loss, time 3999ms
I cant forward packets using iptables to another host. I see packets come in on the “nat table” matching my forwarding rule, but there is never matching packets in the “FORWARD” chain in the filter table.
I cant ping anything from other machine on the network that is trying to use this server as the gateway. I cant connect to any other host on the internet from that host either via HTTP/SSH etc. Using conntrack to try and SSH from the other machine on the network I can see its using this machine as its gateway but neither ssh or ping works..
[root@fw0 ~]# conntrack -E 2>&1|grep -E “(192.168.128.52|62.30.72.233)”
SSH attempt from another machine on the network… (Fails)
[NEW] tcp 6 120 SYN_SENT src=192.168.128.52 dst=XXX.30.72.233 sport=59834 dport=22 [UNREPLIED] src=XXX.30.72.233 dst=XXX.146.154.131 sport=22 dport=59834
[UPDATE] tcp 6 59 SYN_RECV src=192.168.128.52 dst=XXX.30.72.233 sport=59834 dport=22 src=XXX.30.72.233 dst=XXX.146.154.131 sport=22 dport=59834
[UPDATE] tcp 6 58 SYN_RECV src=192.168.128.52 dst=XXX.30.72.233 sport=59834 dport=22 src=XXX.30.72.233 dst=XXX.146.154.131 sport=22 dport=59834
[UPDATE] tcp 6 56 SYN_RECV src=192.168.128.52 dst=XXX.30.72.233 sport=59834 dport=22 src=XXX.30.72.233 dst=XXX.146.154.131 sport=22 dport=59834
[UPDATE] tcp 6 52 SYN_RECV src=192.168.128.52 dst=XXX.30.72.233 sport=59834 dport=22 src=XXX.30.72.233 dst=XXX.146.154.131 sport=22 dport=59834
[DESTROY] tcp 6 src=192.168.128.52 dst=XXX.30.72.233 sport=59830 dport=22 src=XXX.30.72.233 dst=XXX.146.154.131 sport=22 dport=59830
[UPDATE] tcp 6 43 SYN_RECV src=192.168.128.52 dst=XXX.30.72.233 sport=59834 dport=22 src=XXX.30.72.233 dst=XXX.146.154.131 sport=22 dport=59834
[DESTROY] tcp 6 src=192.168.128.52 dst=XXX.30.72.233 sport=59832 dport=22 src=XXX.30.72.233 dst=XXX.146.154.131 sport=22 dport=59832
trying ping (also fails)
[NEW] icmp 1 30 src=192.168.128.52 dst=XXX.30.72.233 type=8 code=0 id=13152 [UNREPLIED] src=XXX.30.72.233 dst=XXX.146.154.131 type=0 code=0 id=13152
[UPDATE] icmp 1 29 src=192.168.128.52 dst=XXX.30.72.233 type=8 code=0 id=13152 src=XXX.30.72.233 dst=XXX.146.154.131 type=0 code=0 id=13152
[DESTROY] tcp 6 src=192.168.128.52 dst=XXX.30.72.233 sport=59834 dport=22 src=XXX.30.72.233 dst=XXX.146.154.131 sport=22 dport=59834
So from this internet connected host I can use the new IP block without issue EXCEPT for icmp. For any other machone on the network I cant use this host at all (thought I can see the data hit this sever). All smells like connection tracking isn’t working OR something.
The other machine is going out on that IP because I’ve got a rule in my firewall that binds it to that.e.g.
iptables -t nat -I POSTROUTING -s 192.168.128.52/32 -j SNAT –to-source XXX.146.154.131
Its obviously working
So really my problem appears to be ICMP not working and ip forwarding not working, i _THINK_ the policy based routing is working.
Fantastic article. Worked perfectly!
Hi Darien,
Thank you for the great article!
I have two questions:
1. Can this be done without the need of iproute2 and/or multiple routing tables?
2. I’ve made a small experiment on my side in 2 vms, one of which has 2 NIC’s, one bridged to my LAN and the other a vbox internal network and things seem to be working without having other routing tables. What could be wrong in this routing table for having the traffic going properly foreach interface? http://imgur.com/a/ncGpc
Admiring the commitment you put into your site and in depth information you
offer. It’s awesome to come across a blog every once in a while that isn’t the
same old rehashed information. Excellent read! I’ve bookmarked your site and I’m including your
RSS feeds to my Google account.
Hi Darien,
Pretty amazing comment and reply section you have built-up here. Do you have any thoughts on the situation described here:
https://serverfault.com/questions/892317/linux-kernel-not-passing-through-multicast-udp-packets-from-secondary-interface
My assumption is that I should not need to define a second routing table as you suggest in this post since modern versions of Linux allow multiple default routes to be defined in the main table? Thanks for your time!
– Duick
Definitely imagine that that you stated. Your favorite reason appeared to be
at the internet the simplest thing to be awae of.
I say to you, I definitely get annoyed while other people consider issues that they plainly do not realize about.
You managed to hit the nail upon the highest and also defined out the entire thing with no need side-effects , people can take a signal.
Will probably be again to gget more. Thank you
Thank you so much for this! Fantastic!
You have a very nice layout for your blog” i want it to use on my site too “
Superb Job, well explained . you saved my time a lot.
Awesome site you have here but I was curious about
if you knew of any discussion boards that cover the same topics discussed here?
I’d really like to be a part of community where I can get opinions from other experienced
people that share the same interest. If you have any suggestions, please let me know.
Appreciate it!
Unfortunately, Ping does not work with this method on the second interface. I read through several tutorials like yours and I also can use Traceroute, Mtr, … on the second interface, but Ping is just not working, at all. I can even SSH on it from the outside. See more information here, if anyone is interested: https://stackoverflow.com/questions/53616718/problem-with-routing-on-two-default-gateways
It is hard to read grey text on a black background (for the samples) . I found it easier to copy the whole post, and then paste it somewhere. Eg the first line — “say eth0 and eth1” eth0 and eth1 are not clear
Not working for UDP packets
Does someone have an example configuration for netplan?
Is there a way to do this via netplan?
Yes. Here is an example of the /etc/netplan/01-netcfg.yaml file using netplan:
Reference: https://askubuntu.com/questions/1191437/server-18-04-netplan-with-multiple-interfaces-on-different-networks-with-differe
52 Trackbacks/Pingbacks
[…] Tästä ohjeesta soveltaen väänsin sitten iproute.sh nimisen scriptin joka sisältää seuraavaa. #!/bin/sh […]
[…] Configuring Multiple Default Routes in Linux « Darien Kindlund’s Blog (tags: linux network networking routing gateway howto IP iptables policy) […]
[…] […]
[…] met enige goede wil (en beetje kennis of wil om man pages te lezen) kun je hier iets mee: https://kindlund.wordpress.com/2007/1…utes-in-linux/ Daarmee kun je de ene gateway aan de ene stateful table knopen, en de andere aan de 2e […]
[…] the method from this website, we create a policy route. I am using Ubuntu 10.04 Server 64bit in this example which has […]
[…] Lösung für dieses Problem findet mann in iproute2. In seinem Artikel Configuring Multiple Default Routes in Linux erkärt Darian Kindlund sehr gut, wie man mehrere default routs konfiguriert. Tags: […]
[…] Take a look at this site, it has an explanation and some information on how to do this with iproute2 (required). https://kindlund.wordpress.com/2007/1…utes-in-linux/ […]
[…] site for reference, but the one that explained exactly what I wanted to do and helped me most was https://kindlund.wordpress.com/2007/1…utes-in-linux/ Many thanks to this […]
[…] Configuring Multiple Default Routes in Linux « Darien Kindlund’s Blog. […]
[…] think I have found the answer at this link. I will be trying this out on my server and will post again here the results. […]
[…] Basically I followed this webpage to set it up : https://kindlund.wordpress.com/2007/1…utes-in-linux/ […]
[…] https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ […]
[…] This blog entry explains how to setup multiple routes for Linux in some detail. […]
[…] eventually found a solution by combining two excellent blog posts (this one and this one) as well as some trial and error in figuring out how to make the changes persist on a […]
[…] guide here 0.000000 0.000000 Like this:LikeBe the first to like this post. Comments RSS […]
[…] it's a bit complicated! You might want to start at http://lartc.org/howto/ Also, I found this blog entry which might give a simpler introduction to the […]
[…] has two gateway lines, including one for eth1, which I don't have) The same person suggested I do https://kindlund.wordpress.com/2007/1…utes-in-linux/ But then today, it seems to be working and I didn't change anything since last night. Why might it […]
[…] Originally Posted by kopper27 looks like I have to use iproute2 just don't know how any guidance thanks try this link https://kindlund.wordpress.com/2007/1…utes-in-linux/ […]
[…] and default routes for each interface: http://www.debian-administration.org…Policy_routing https://kindlund.wordpress.com/2007/1…utes-in-linux/ However, when ppp0 is up, "ping -I ppp0 8.8.8.8" from the firewall gets a response, but […]
[…] http://www.debian-administration.org…Policy_routing https://kindlund.wordpress.com/2007/1…utes-in-linux/ […]
[…] searched the solution, found Configuring Multiple Default Routes in Linux, followed the steps, and […]
[…] https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ Answered by lzap […]
[…] pulled most of this technique from this site. Works like a champ for my purposes! Post a […]
[…] with the old 'route' command, but it is doable with 'ip' by creating multiple route tables.. https://kindlund.wordpress.com/2007/1…utes-in-linux/ http://www.generationip.com/document…teway-on-linux To answer your question about running a […]
[…] here Share this:TwitterFacebookLike this:LikeBe the first to like this. Posted by baxteragu Filed in […]
[…] Hi Take a look over here […]
[…] https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ […]
[…] a look at this https://kindlund.wordpress.com/2007/1…utes-in-linux/ it's upper level stuff but I think it may be the key to getting it to work the way you want. It was […]
[…] https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ […]
[…] Here is the answer […]
[…] I've managed are strictly for backup networks or failover), so can't vouch for the instructions in this blog post, but they may […]
[…] https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ […]
[…] need 2 default gateways and 2 routing tables I did a little more Googling and found this article: https://kindlund.wordpress.com/2007/1…utes-in-linux/ The server is online and accessible correctly on both […]
[…] In the normal network stack you get one route to any destination. Regardless of which NIC receives a ping, for example, the reply will go out through the default gateway. To do advanced policy-based routing have a look at this: https://kindlund.wordpress.com/2007/1…utes-in-linux/ […]
[…] I believe I’ve achieved a table that routes packets from and to eth1/192.168.3.x through 192.168.3.1, and packets from and to eth0/192.168.1.x through 192.168.1.1 (helpful source). […]
[…] Configuring Multiple Default Routes in Linux […]
[…] More reading… […]
[…] pulled most of this technique from this site. Works like a champ for […]
[…] & relevant links https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ http://brainscraps.wikia.com/wiki/Setup_Gateway_Routing_On_Multiple_Network_Interfaces […]
[…] & relevant links https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ http://brainscraps.wikia.com/wiki/Setup_Gateway_Routing_On_Multiple_Network_Interfaces […]
[…] Configuring Multiple Default Routes in Linux […]
[…] addresses for Squid? [linux multihoming gateway] Multi gateway routing with iptables and iproute2 Configuring Multiple Default Routes in Linux Selective load balancing with Squid and iproute2 (utilizes ToS field for differentiating traffic) […]
[…] Проблема эта не сказать что новая — она существует, наверное, со времён появления систем с двумя интерфейсами (неважно, реальными или виртуальными), но как ни странно, нагуглить решение хоть и получается, но не прямо «сходу». Лично я это решение нашёл вот в этой статье. […]
[…] Check this out. I think it will at least get you started if this is what you are trying to accomplish. (This too.) […]
[…] here’s the problem: According to https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ I need to set up a second routing table for eth1. However, since both networks are on the same […]
[…] https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ […]
[…] too much effort to get this setup working. Most solutions I have tried so far go in this direction: https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ I hope some of you can tell me some obvious stuff I’m currently not seeing. I appreciate […]
[…] https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ […]
[…] SOLUTION: I solved my issue using both ip route and ip rule. ip route is used to create a new route, ip rule is used to tell linux when to use the new route. 1 – Create a new routing table that will be only used to solve this routing issue: echo 101 myroute>> /etc/iproute2/rt_tables 2 – Populate myroute with instructions to reply all traffic to the IP address of your Pfsense box: ip route add default via 192.168.0.4 table 101 The routing table myroute on its own wont affect anything yet. I needs a policy rule to activate it. Once activated, it will work alongside your standard routing table which will be unaltered. 3 – Add the rule: ip rule add from 192.168.0.78 table myroute 4 – flush routing table to reload settings: ip route flush cache Please notice that while doing this, all traffic sent to the server ip 192.168.0.78 via the local subnet (switch) will be replied via the 192.168.0.4 gateway, and thus local connections will fail. In order to understand why your new table supersedes the default routing table, you should read this great link dedicated to this issue. https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ […]
[…] https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ […]
[…] am following these instructions in order to configure multiple default routes for Linux (I am testing it on Ubuntu v12.04 and Kali […]
[…] https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/ […]