Skip navigation

Assume you have a Linux system with more than one network interface card (NIC) — say eth0 and eth1. By default, administrators can define a single, default route (on eth0). However, if you receive traffic (i.e., ICMP pings) on eth1, the return traffic will go out eth0 by default.

This can be a bit of a problem — especially when the two NICs share the same parent network and you’re trying to preserve sane traffic flows. In a nutshell, this post will explain how you can ensure traffic going into eth0 goes out only on eth0, as well as enforce all traffic going into eth1 goes out only on eth1.

You’ve found the one post that actually explains this issue; your googling has paid off. You wouldn’t believe how many advanced Linux routing websites out there explain how to route everything including your kitchen sink — yet fail to clearly explain something as simple as this.

As always, we’ll explain by example. Assume the following:

  • eth0 - 10.10.70.38 netmask 255.255.255.0
  • eth0's gateway is: 10.10.70.254
  • eth1 - 192.168.7.126 netmask 255.255.255.0
  • eth1's gateway is: 192.168.7.1

First, you’ll need to make sure your Linux kernel has support for “policy routing” enabled. (As a reference, I’m using a v2.6.13-gentoo-r5 kernel.)

During the kernel compilation process, you’ll want to:

cd /usr/src/linux
make menuconfig
Select "Networking --->"
Select "Networking options --->"
[*] TCP/IP networking
[*] IP: advanced router
Choose IP: FIB lookup algorithm (FIB_HASH)
[*] IP: policy routing
[*] IP: use netfilter MARK value as routing key

Next, you’ll want to download, compile, and install the iproute2 [1] utilities. (Most Linux distributions have binary packages for this utility.) Once installed, typing ip route show should bring up your system’s routing table. Type man ip for more information about this utility, in general.

Speaking of which, assume the system’s initial route configuration looks like this:

# netstat -anr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.7.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.10.70.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 192.168.7.1 0.0.0.0 UG 0 0 0 eth1

So, basically, the system is using eth1 as the default route. If anyone pings 192.168.7.126, then the response packets will properly go out eth1 to the upstream gateway of 192.168.7.1. But what about pinging 10.10.70.38? Sure, the incoming ICMP packets will properly arrive on eth0, but the outgoing response packets will be sent out via eth1! That’s bad.

Here’s how to fix this issue. Borrowing the method from a really sketchy website [2], you’ll first need to create a new policy routing table entry within the /etc/iproute2/rt_tables. Let’s call it table #1, named “admin” (for routing administrative traffic onto eth0).

# echo "1 admin" >> /etc/iproute2/rt_tables

Next, we’re going to set a couple of new entries within this “admin” table. Specifically, we’ll provide information about eth0‘s local /24 subnet, along with eth0‘s default gateway.

ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
ip route add default via 10.10.70.254 dev eth0 table admin

At this point, you’ve created a new, isolated routing table named “admin” that really isn’t used by the OS just yet. Why? Because we still need to create a rule referencing how the OS should use this table. For starters, type ip rule show to see your current policy routing ruleset. Here’s what an empty ruleset looks like:

0: from all lookup local
32766: from all lookup main
32767: from all lookup default

Without going into all the boring details, each rule entry is evaluated in ascending order. The main gist is that your normal main routing table appears as entry 32766 in this list. (This would be the normal route table you’d see when you type netstat -anr.)

We’re now going to create two new rule entries, that will be evaluated before the main rule entry.

ip rule add from 10.10.70.38/32 table admin
ip rule add to 10.10.70.38/32 table admin

Typing ip rule show now shows the following policy routing rulesets:

0: from all lookup local
32764: from all to 10.10.70.38 lookup admin
32765: from 10.10.70.38 lookup admin
32766: from all lookup main
32767: from all lookup default

Rule 32764 specifies that for all traffic going to eth0‘s IP, make sure to use the “admin” routing table, instead of the “main” one. Likewise, rule 32765 indicates that for all traffic originating from eth0‘s IP, make sure to use the “admin” routing table as well. For all other packets, use the “main” routing table. In order to commit these changes, it’s a good idea to type ip route flush cache.

Congratulations! You’re system should now properly route traffic to these two different default gateways. For more than 2 NICs, repeat the table/rule creation process as necessary.

Please provide comments, if you find any errors or have corrections to this post. I don’t claim that this method will work for everyone; this information is designed primarily to preserve my sanity, when configuring routing on future multi-NIC Linux systems.

References:
[1] http://www.policyrouting.org
[2] http://www.linuxhorizon.ro/iproute2.html

Update: Here are some additional resources, that I have found useful.
http://lartc.org/howto/lartc.rpdb.multiple-links.html
http://linux-ip.net/html/routing-tables.html

Update: Apparently, OpenBSD also now supports multiple default routes through a new feature called the Virtual Routing Table:
http://www.packetmischief.ca/2011/09/20/virtualizing-the-openbsd-routing-table/

About these ads

301 Comments

  1. What a fantastic article, you’ve saved my butt, I had my 2 nic system configured ok, but every reboot would change it, and voila my listening servers would nose dive. this works perfectly

  2. This is just exactly what I have been looking for, and you are right googling has paid off. I was looking for something that solved this issue, and went straight to the point. Many articles written out there tends to touch on the issue but never really explains what’s happening or maybe it seems to them so simplistic, they overlook the point all together. I don’t know if they ever think about the novices reading their articles…. This one sure did it for me, thank you very much.

  3. Great article indeed… but I’m in a situation where I need to be able to fail over to a different “default” gateway in case of the primary default gateway is unreachable. Lets say I have three upstream routers 10.0.0.1, 10.0.0.2, 10.0.0.3/24. Is there a way for me to inject all 3 IPs into a routing table as my “default” routes on a Linux host?

  4. RE: Failover Default Routes

    Here are some articles that describe what you’re looking for:

    http://www.uwsg.iu.edu/hypermail/linux/net/0009.1/0027.html

    http://www.linux.com/articles/113988

    http://mailman.ds9a.nl/pipermail/lartc/2002q1/002667.html

    OpenBSD also has a daemon called “hoststated” that provides this type of capability:

    http://www.openbsd.org/cgi-bin/man.cgi?query=hoststated&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html

  5. Thank you very much for the links…

    gc_timeout did the trick for me as the default is 5 min which is too long for a route to fail over.

  6. just wondering if there is a kernel value to control how long a “” arp entry may stay in the arp cache…

    #arp -a
    hosta (100.100.100.1) at on eth0

  7. not sure why the word “incomplete” did not show up in my previous post…

  8. RE: ARP Cache Timeout

    Try this link, for information about how to control how long entries stay in the ARP cache:

    http://www.ussg.indiana.edu/hypermail/linux/net/0012.2/0035.html

  9. great article
    thanks to all who rights this article

  10. Great article
    Thanks to all who Wright this article

  11. Thanks for this article. It helped me get past a roadblock in a project.

  12. i’m actually kind of curious – the most common, default situation is where eth0 is the interface that packets are routed through — heck, that’s noted in the first sentence — so i’m wondering why you constructed your example with the reverse configuration – with eth1 as the default? i found it kind of confusing, in an otherwise most excellent article….

  13. Sorry about the confusion; in my example, the Linux distribution I used in my example (Gentoo) seemed to always specify eth1 as the default route — when both eth0 and eth1 were present and active (and when automatic configurations were used).

    Let me ask you this, in your Linux distribution, when you have both an eth0 and an eth1 on the system with active links for both — does your distribution automatically select eth0 as the default route? I’m guessing it’s pretty much random, as the standard defaults are to use DHCP for both eth0 and eth1, in which case, whichever interface gets the DHCP lease (first? or last?) will ultimately decide which default route actually gets used by the system.

    I hope this explains where I was coming from.

  14. oh, okay, that makes sense. i’m always running servers with fixed IP’s, so DHCP never enters into the picture.

    for what i’m attempting, i actually found that going with bonded interfaces worked best – using a bond0 device and making both interfaces capable of “believing” that they’re the server’s IP, and the only path out. thus if one interface goes offline, traffic can still pass.

  15. I have a slightly different requirement.
    I have internet access both over WLAN and over DSL. I use the DSL for VoIP setup and WLAN for internet. But sometimes the WLAN or the DSL goes out of service. In this caseI want to use both the services – Internet and my VoIP setup over the either WLAN or DSL whichever is active.
    How could I do this?

    Thanks,
    Amit
    kahnoie@gmail.com

  16. Hi Amit,

    See my earlier comment about “Failover Default Routes”:

    http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/#comment-10

    — Darien

  17. Hi,

    Thank you for this helpfull article.
    Your exemples are related to 2 NICs : eth0 and eth1. But, what if instead of physical eth0 and eth1, we have a bonding logical one’s like this :
    bond0 : eth0 + eth1
    bond1 : eth3 + eth4.
    That is we have 4 NICS or a quad card in a server.

    In that case, what will be the configation.

    Tahnks,
    Bras

  18. Hi Bras,

    Actually the configuration would be almost identical with the example, but just replace eth0 with ‘bond0′ and eth1 with ‘bond1′.

    Linux treats ‘bond0′ and ‘bond1′ as physical NICs, so the rest of the networking subsystem will simply think that these are physical devices.

    Hope that helps,
    — Darien

  19. Hi,

    Many thanks for this wonderful Info !!!
    how can I make this permanent, so that
    when my server reboots, I will have the same ruling.

    Where can table admin and IP ruling
    mods be placed …

    Thanks in advance…

  20. Each Linux distribution is slightly different. Some distributions offer a way to save these configurations within a configuration file under /etc to be used upon reboot.

    Other distributions require you to add the “echo” and “ip rule” commands into a script file that gets called as one of the last commands when the system boots up.

    For example, on Ubuntu Linux, there is a script file called “/etc/rc.local” that you can add all the “echo” and “ip rule” commands into, that will apply these configurations upon complete server bootup.

    However, please be aware that if you have other network services that require the multiple default routes to be set up BEFORE those services can successfully load (e.g., an automatic VPN/IPSec tunnel), then you’ll have to create an rc.d init script, similar to those found in the “/etc/init.d” directory.

    The short answer:
    – Try adding the commands to “/etc/rc.local”.
    – If other services fail, then make a copy of a simple script in “/etc/init.d”; then edit that script to reflect the commands needed to be run at boot time; then register that script to be started upon bootup (“man update-rc.d”, for Ubuntu/Debian Linux).

    Hope this helps,
    — Darien

  21. Hello,
    This is a good article, but when you say : “However, if you receive traffic (i.e., ICMP pings) on eth1, the return traffic will go out eth0 by default.”, car you please explain or give examples where we will or can receive traffic on eth1 if we have eth0 as a gateway.

    Thanks
    Lasy

  22. Sure,

    Lasy, when you set a single default route, then that route is relevant for *all traffic outside your local subnet*.

    Let’s say you have a public class-C subnet of 128.253.1.0/255.255.255.0 and eth0 is 128.253.1.1 while eth1 is 128.253.1.2 and your default route is through eth0.

    Then, let’s say someone from a completely different subnet (from 128.5.1.1, for example) pings your ETH1 IP address (128.253.1.2). The ICMP request packets will arrive on eth1; however, when your system sends out the ICMP response packets, *because the destination is outside your local subnet*, these packets will go out on eth0, since that is your default route.

    The point of the article is that if you have two or more network interfaces and have only one default *route* set on the OS, then *for all non-local-subnet traffic*, all outgoing packets will go out on the interface marked as the default route — unless you follow the steps listed in this article.

    — Darien

  23. I have a backup server that has 2 NIC’s. One NIC is on my Server VLAN 10.1.0.0/16 and the other is on my storage VLAN 10.240.1.0/24. What I would like to do is force the backup traffic coming in on the 10.1.0.0/16 NIC to then be routed out to the backup NAS on the 10.240.1.0/24 NIC. How cna this be done?

    Thanks

  24. Thanx 4 that! Everything now seems to work… NICE DONE!

  25. Great article! Thank you very much for writing :o)

  26. At last! Thanks for writing this. While it didn’t save a me a lot of time I *already* wasted, it was the solution to my networking conundrum. You rock.

  27. Thanks for this writeup! It saved me a lot of time!

  28. That’s a very handy piece of writing, Darien ! Thanks for posting it.

    I can’t find IP_ROUTE_FWMARK in the new kernel for marking the packets (I’m using 2.6.27) or by googling. Has it been removed ?

    Any pointers would be great.

    GB

    • Hi Ghostbuster,

      Yes, IP_ROUTE_FWMARK was removed from linux v2.6.20 and later releases. See this link for details:

      http://cateee.net/lkddb/web-lkddb/IP_ROUTE_FWMARK.html

      Presumably, the feature still exists, but it’s probably named something else (or perhaps merged with some other feature).

      Hope this helps,
      — Darien

  29. Hi Darien,

    Thanks for that pointer.

    I’m thinking it’s the MARK command in Netfilter these days.

    http://www.linuxtopia.org/Linux_Firewall_iptables/x4368.html

    I’ve done some qdisc stuff so hopefully it won’t be too alien.

    GB

  30. Okay, scrap that last comment :-)

    I’ll try this instead hoping the support is all handled by iproute2 these days:

    http://www.physics.umd.edu/pnce/pcs-docs/Glue/linux-route-hack.html

    GB

  31. Kick-ass ! This fixed a problem I was having with an FTP server with two NICS. I could only get it to work with one default gateway at a time. Using what I found here, I created two additional entries, one for each NIC. Works perfectly now. I had to use a slight variation on the ip route statement (using CentOS) – it was: “ip route add 10.1.250.0/24 via 10.1.250.1 dev eth0 table inside” instead of dev eth0 src . Great article – thanks so much !!

  32. This is a really good article that gets me 90% of where I need to go, I really appreciate it. I was wondering if you also know how to configure a system with 10 nics with addresses on the same subnet to send packets directly to the same gateway instead of passing them indirectly through the dfault route on the machine. So, for example, 10.0.0.2->10.0.0.1 and 10.0.0.3->10.0.0.1 instead of
    10.0.0.2->10.0.0.1 and 10.0.0.3->10.0.0.2->10.0.0.1. (where 10.0.0.1 is the gateway for that subnet, and 10.0.0.2 is the address of the interface with the default local route.)
    I see how your example works for addresses on different subnets, but not for my situation. I would appreciate any ideas!

  33. Hi Howard,
    Yes, I’m familiar with that problem. To be honest, this problem is solved at a different layer than routing. What you’re looking for is “Link Aggregation” (aka. making multiple NICs in your server act as one large NIC). There is an entire page dedicated to the concept, available here:

    http://en.wikipedia.org/wiki/802.3ad

    In order to get it to work, though, you’ll need to make sure your upstream switch/router supports the 802.3ad protocol (most recent managed switches do support it; however, if you have an unmanaged switch, probably not).

    Hope this helps,
    Darien

  34. Hi,

    Am a little lost. On which file do you put all those commands? Secondly, after you have done all the configurations as shown in your solution, what will be the output when you do netstat -rn. Your assistance is highly appreciated.

  35. Hi Anderson,

    There is no specific file, as all the commands I’ve listed are performed from the command line. If you’re looking to run these commands every time your OS boots, then you can put them all in /etc/rc.local (for Ubuntu Linux).

    Second, ‘netstat -rn’ can’t show you “policy routes” properly. So even after you run the policy routing commands, it’s likely ‘netstat -rn’ will still show you the same thing before and after the policy routes are in place.

    If you’re looking to verify that policy routes are working properly, run the “ip rule show” before and after setting up policy routes. If you want more detailed information, see the man pages for the “ip” command, by typing “man ip”.

    — Darien

  36. Hi Darien,

    Thanks so much for your response. One more question. Should I include the respective gateways in each interface configuration files for my case ifcfcg-eth0 and ifcfcg-eth1? My OS is RedHat El5. My default gateway for eth0 I have defined it in /etc/sysconfig/network file. Where should I define the default gateway for eth1?

  37. Hi Anderson,
    You can define your eth1 gateway in the set of commands you enter into /etc/rc.local

    The “ip rule” commands Listed in the article will properly configure your eth1 gateway, so you won’t need to define it within /etc/sysconfig/network

    — Darien

  38. What do you change if both NICs are on the same network segment, using the same gateway IP? I tried following the steps above but I was left with all traffic going out eth0 while some went in eth1.

    • Hi nobodyfamous,
      If both NICs are on the same network segment, the policy routing isn’t going to help you. The problem has to do with Linux kernel’s TCP/IP stack employing short-circuit routing.

      Specifically, this problem is solved at a different layer than routing. What you’re looking for is “Link Aggregation” (aka. making multiple NICs in your server act as one large NIC). There is an entire page dedicated to the concept, available here:

      http://en.wikipedia.org/wiki/802.3ad

      In order to get it to work, though, you’ll need to make sure your upstream switch/router supports the 802.3ad protocol (most recent managed switches do support it; however, if you have an unmanaged switch, probably not).

      — Darien

      • Yeah, bonding was the initial goal, but we don’t have access to the switches in this case, so it had to be solved at the host level.

        We were able to make it work following a modified form of your instructions, FWIW:

        In this example, the gateway is 192.168.0.1 and the server’s IPs are 192.168.0.211 and 192.168.0.212 on eth0 and eth1 respectively:

        printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
        printf “2\tuplink1\n” >> /etc/iproute2/rt_tables

        ip route add 192.168.0.211/32 dev eth0 src 192.168.0.211 table uplink0
        ip route add default via 192.168.0.1 dev eth0 table uplink0
        ip rule add from 192.168.0.211/32 table uplink0
        ip rule add to 192.168.0.211/32 table uplink0

        ip route add 192.168.0.212/32 dev eth1 src 192.168.0.212 table uplink1
        ip route add default via 192.168.0.1 dev eth1 table uplink1
        ip rule add from 192.168.0.212/32 table uplink1
        ip rule add to 192.168.0.212/32 table uplink1

        By the way, in the second paragraph you say that the solution is applicable to cases where the two NICs are on the same network, that’s why I followed your lead on this.

  39. This is extremely helpful. Thank you.

    One question about routing in general, is there a way to have subnets share a common default gateway?

    • Hi Tekknogenius,
      If we’re talking about two subnets sharing a common, physical gateway (such as a router), then yes, but that common gateway must have a valid IP address on each subnet. For example, assume you have two subnets:

      192.168.0.0/24
      192.168.1.0/24

      You can then configure your router to have the IPs of:
      192.168.0.254 and 192.168.1.254 for use by members of their respective subnets.

      Bottom line is that the gateway must have an address in each subnet; otherwise, you would have to figure out some other way to reach the gateway’s IP address (e.g. static ARP entry if on the local LAN)

      — Darien

  40. Hi – i have another slightly different goal. My Linux box has an eth0 (connected via a default gw to the internet) and a on-demand ppp0 (gprs) connection. My goal is to programmatically create sockets via both interface at the same time. I think i am stuck setting up the correct routes for this – do you know how to handle this issue?

    • Hi Johnny,
      See my earlier comment as I’ve already addressed this issue:

      http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/#comment-10

      — Darien

      • Thanks for your quick reply. I’m not sure failover routing is the correct issue here, as both routes should be up all the time. My app decides on various criteria which connection should be used. I had read som article, whiches in short says: What goes out on one device, should comme in on that again. Is this the right idea? And how can I realize that?

        • Hi Johnny,
          Okay, thanks for the clarification. I had assumed when you mentioned ‘on-demand’, that you somehow wanted to switch from eth0 to ppp0 when eth0 failed. Instead, you want both eth0 and ppp0 alive all the time.

          In that case, just replaced ‘eth1′ with ‘ppp0′ in the example and you can still use policy routing to accomplish what you’re looking for.

          The real challenge, is if ppp0’s network configuration changes constantly. If the IP address, subnet, and route of ppp0 constantly changes, then you need to somehow have logic to detect when these changes occur and update your policy routes accordingly.

          In that case, you’ll probably want to research DHCP notification code. Depending on your Linux distribution, there are scripts which execute when your network changes, which you could use to also update your corresponding policy routes.

          — Darien

  41. Great article. This is so simple, yet so hard to find the information. Thanks a lot :-)

  42. Hey,

    Great article, helped me loads compared to the other sites were they just skimm abit on the top.

    Thou, having problem adapting it to my setup.

    I currently have 2 nic’s on my server that is acting as router/gateway.

    1 of the physical nic’s has 5 virtuell interfaces (g1-5), g1 goes to the router itself as it also goes as a webserver. The rest are nated straight to nic2(which has no virtuell interfaces). As i have 4 computer, g2-5 are nated 1:1 towards the inside local ip:s 10.10.1.X.

    I have a simple iptables rule to fix the NAT part, and i had it working “sortoff” since everything was working except that i could not reach the servers/pc’s from the outside, they could only talk from the inside->out (torrents etc works flawlessly thou).

    Any hints? If you got a mail or something it would be great to get some advice..

    • Hi DevizioN,

      It sounds like you have one-way NAT working and you want bi-directional NAT. From your post, it sounds like g1 goes to the router, as well as nic2 goes directly to the router … is that correct?

      Regardless, it sounds like setting up multiple default routes correctly will NOT fix your NAT problem… I would recommend you try and fix your NAT problem first, and then setup the multiple default routes.

      To be very specific, with your current NAT configuration, try configuring your server with only ONE upstream gateway and fix your NAT so that it’s working in both directions (insideout).

      Once you get a single route working, then add your second upstream gateway. If your bi-directional NAT breaks and no longer works correctly, then let me know and we can troubleshoot further (as configuring multiple default routes may help you).

      The general approach is: reduce the complexity of your configuration, verify each piece works first (e.g., NATing, virtual interfaces), and THEN verify the total configuration works. Otherwise, it can be a nightmare to troubleshoot multiple unknown pieces at the same time.

      — Darien

  43. Hi, if i have 2 or more subnets

    example

    10.10.70.0/24
    10.10.71.0/24

    ip rules add and from are with /32 or /24?

    example

    ip rule add from 10.10.70.38/32 table admin
    ip rule add to 10.10.70.38/32 table admin
    ip rule add from 10.10.71.38/32 table admin2
    ip rule add to 10.10.71.38/32 table admin2

    or

    ip rule add from 10.10.70.38/24 table admin
    ip rule add to 10.10.70.38/24 table admin
    ip rule add from 10.10.71.38/24 table admin2
    ip rule add to 10.10.71.38/24 table admin2

    I don’t hunderstand why you use all 32 bits.

    Thanks

  44. Excellent Post!

    I have followed the various bonding solutions available on Linux since I was doing load balancing work in the late 90’s but have never found anything that delivers on bonding two links to maximize bandwidth utilization.

    I have a FIOS and Cable link that I’d been trying with bonding modes 5,6 on with various 3com 509’s and netgear 100mbit nics, after following your references here to Astro Shapes pages (also very informative). mii-tool seems to work fine with the nics.. but can’t get them to attach correctly for ifenslave

    Given I’m interested in maximizing traffic on the links I wonder if anyone has rules that will monitor and direct traffic based on load on the individual nics?

    Thanks

  45. Thanks for writing a very useful article and subsequently answering so many questions. Here’s another :)

    Supposing you have one or more aliases on the public interface (e.g., eth0:0) would you repeat all of the above steps, substituting eth0:0 for eth0?

    • Yes. Using eth0:0 should work.

  46. Thanks for a great article. This looks like kinda what I need. However, my situation is a bit different (aren’t they all :) ). Anyway, I am basically, trying to setup openVPN on my Ubuntu 8.10 server. I got 2 NICs in my server. eth0 and eth1. eth0 is the external NIC while eth1 is the internal one (the LAN. eth0 is the default route. VPN connects through eth0 and I can ping my server and I can ping eth1, however I can’t ping anything behind eth1 (which is my local LAN). From the server console itself I can ping my internal LAN. I think the problem lies with traffice coming via eth0 and since the default route is eth0 it only goes through eth0. So if I ping anything which is behind eth1 it is routed via eth0 and not eth1. Is it possible to route all VPN traffic meant for eth1 through eth1 even though it’s not the default route?

    Does this make any sense or am I making you guys as confused as myself?

    Thanks for your help in advance.

    • Hi Mazzer,

      1) As a sanity check, are you able to ping your eth1 network if you TURN OFF your VPN? I just want to make sure that the problem isn’t more fundamental.

      2) On your server, once you have your VPN setup, you actually have THREE different interfaces: eth0, eth1, and (probably something like vpn0 — check your ‘ifconfig -a’ for details). Essentially, you have a third ‘virtual’ interface, representing the effective IP address of your server, once connected on the *inside* of your VPN network.

      Here’s what I need to know:
      – The IP address and netmask of your eth1 network.
      – The IP address and netmask of your *vpn0* network (see above for what I mean).

      Specifically, I believe that perhaps your eth1 network and your vpn0 network may have conflicting IP addressing, which is causing you to lose connectivity to your eth1 network once a VPN is established. However, I need more information in order to accurately say that this is the problem.

      — Darien

  47. I used the same kinda configuring (but created two tables instead of one) one per interface. This makes sure i get the response to through the right interface.

    # Assign IP address on the interfaces eth0 and eth1
    ip route add 10.1.0.0 dev eth0 src 10.1.0.250
    ip route add 10.0.0.0 dev eth1 src 10.0.0.250

    # IP route and default gateway for eth0 in the table 100
    ip route add 10.1.0.0 dev eth0 src 10.1.0.250 table 100
    ip route add default via 10.1.0.1 table 100
    # IP route and default gateway for eth1 in the table 101
    ip route add 10.0.0.0 dev eth1 src 10.0.0.250 table 101
    ip route add default via 10.0.0.1 table 101

    # Assign the rule to corresponding particular table
    ip rule add from 10.1.0.250 table 100
    ip rule add from 10.0.0.250 table 101
    ip route flush cache

    Now, i can ping the box to the corresponding box from behind the
    gateway. Everything works fine according to my requirement.

    Now if i use the ping utility (from the box) with a destination address not in 10.1.X.X or in 10.0.X.X network using both the interfaces (or the
    ip-address) it works
    ping -I eth0 some-address (works)
    ping -I eth1 some-address (works)
    ping some-address (not working) with “network is unreachable” as the error

    Now this force me to use the default route on the main table which i
    thought was not neccessary using below command
    ip route add default via 10.1.0.1.

    Since with out any interface or ip-address explicity given, ping
    should have chosen the source ip-address as either 10.1.0.250 or
    10.0.0.250. If so , rule should have hit in routing table 100 or table
    101. But just wondering why it is hiting the main routing table. This
    forces me to have a two gateway programming option for each interface
    and another default gateway to select among two. Not sure why..

    • Also how do we use the same concept in the DHCP enabled network. DHCP client running on both the interfaces. DHCP clients always deals with the main routing table (instead of the custom created routing table).

      • Handling DHCP leases is tough. You’ll need to use a DHCP client that supports custom extensions, so that you can call a script (which you create) that will setup the appropriate policy routes.

    • What does your ‘netstat -anr’ and ‘ip rule show’ look like?

  48. Hi Darien,

    Thank you for your quick reply. I was away from my computer for a while so couldn’t reply.

    As per your questions, I’ll answer the ones I can right now.

    1) I can ping eth1.

    2) I know what you mean about the 3rd virtual interface. Sorry for not providing more information. I do have one and it’s a tun interface (tun0). I’m using routing and not bridging.

    The IP address of eth1 is 10.10.2.102, netmask is 255.255.255.0 so thats the 10.10.2.0 subnet.

    The details about tun0 I’ll get for you when I go home tonight.

    I think the problem lies between the virtual Interface and eth1. From the VPN server I can ping both eth1 and eth0 and everything behind those interface. I can ping everything behind the eth1 subnet (internal network) and I can ping everyhting behind the eth0 subnet (the internet). When I’m connect via VPN to the server. I can ping both eth0 and eth1 but can’t ping anything behind eth1. Once connected via VPN, I can SSH into the VPN server (through the VPN connection) and one connected to the server console. I can ping everything behind eth1. I guess thats no surprise as I’m am then doing the pinging from the console. So as I said, I think the problem lies with the virtual interface having the default route of eth0 and so the pings arte going through eth0 even though they meant for eth1.

    Phew, I can’t believe I wrote all that down :). Thanks for your help though.

  49. Also Just so you know, I’m using OpenVPN as my VPN server and I’ve setup the redirect gateway option as well.

  50. Got the details for the tun0 interface.

    IP Address: 10.30.2.6
    Subnet Mask: 255.255.255.252
    Default Gateway: 10.30.2.5
    DHCP Server: 10.30.2.5
    DNS Servers: 10.10.2.11

    The 10.10.2.0 subnet is the internal eth1 network so looks like the VPN server is pushing the DNS server to the client correctly.

  51. Good article.
    Forgive my ignorance, but what’s the purpose of the “to” rule, ie

    ip rule add to 10.10.70.38/32 table admin

    that would be used only from incoming packets, and even in that case it doesn’t seem very useful. But I’m sure I’m missing something obvious.

    Thank you.

    • madinc,

      The “to” rule is to tell the tcp/ip stack that for all packets where the destination is 10.10.70.38/32, then route those packets to the “admin” interface. This is important for STATEFUL flow processing. When Linux constructs the appropriate reply (e.g., a TCP ACK), it will know which policy route to use, when sending the reply. Otherwise, without this rule, you’ll find that your ACKs will get sent using the default route in the main table.

      • Sorry for being slow to understand, but for ACK reply packets the “to” (destination IP) will NOT be 10.10.70.38, but rahter the IP of the host the ACK is directed to.
        So if you want to use a specific rule to route those packets out, you probably want a rule that matches on *from* 10.10.70.38, rather than “to”.

        I still fail to see how that “to” rule can match anything other than incoming packets.

        Thank you!

        • Hi madinc,

          The “to” rule is primarily important when BOTH of your ethernet interfaces are on the SAME SUBNET. If both interfaces are on separate subnets with NO possibility of getting overlapping traffic, then the “to” rule is less of an issue.

          However, when both interfaces are on the same subnet, then the “to” rule tells Linux which packets to ignore or listen to if each interface is getting duplicate traffic.

            • Darien
            • Posted November 10, 2009 at 6:14 pm
            • Permalink

            Just to clarify, when I mean SAME SUBNET, I mean that both ethernet interfaces are connected to the same upstream hub/switch/router, where both ethernet interfaces receive copies of all network traffic destined to any of the interfaces.

  52. When both ethernet interfaces are on the same subnet, only the one whose IP address is the destination IP address of the incoming packet will get it.
    It’s like having many hosts in a LAN: only the destination host will reply to ARP and thus get the packet, despite all other hosts being in the same subnet.

    Forgive me, but I still don’t see where the “to” rule comes into play :)

    Thank you.

    • Hm, on second thought…with Linux’s weak host model, both interfaces will reply. So yeah, now I see where the “to” can be useful.

      Thank you and sorry for the noise.

      • No problem; glad to help. In general, when getting real answers, I’d recommend actually testing against an actual Linux system, since a lot of these nuances could change, depending on which kernel patches (from which Linux distributions) are loaded on your test machine.

        On numerous occasions, I’ve found that “what should happen” is largely different than “what really happens”.

  53. I tried using the steps that you used to configure Multiple Routes and I ran into an error when I ran the command “ip route add default vi 10.141.50.50 dev eth2 table ccsadmin”. The error that I received was “error: either “to” is duplicate, or “10.141.50.50” is a garbage. I’m not sure what steps do I need to take to solve this problem that I’m having. I’m running CentOS 4.7 kernel version 2.6.9-78

    • Umm, I think you mis-typed the command:

      “ip route add default vi 10.141.50.50 dev eth2 table ccsadmin”

      Should be:

      “ip route add default via 10.141.50.50 dev eth2 table ccsadmin”

      Specifically, the “vi” should be “via”.

  54. Hmmm…

    I keep getting:

    [root@bjwww001 ~]# ip route add 192.168.20.0/24 dev eth1 src 192.168.20.2 table hkvpn
    RTNETLINK answers: Invalid argument

    All my ifcfgXX files are correct and have broadcast and network IP’s defined too….

    • Did you enable “policy routing” in the Linux kernel and install the iproute2 utilities? (Both were mentioned above in the article.)

      • iproute yes… policy routing – will have to check. This is a stock CentOS 5.4 64 bit. WIll I need to recompile the kernel or is there a kernel flag I can set in sysctl?

        • Policy routing requires a kernel recompilation, if it’s not enabled by default. There’s no way to dynamically enable/disable it via sysctl.

          Not sure about CentOS, but many Linux distributions will store a copy of the actual Linux kernel config in /proc/config.gz, so you could try inspecting that kernel config (if it exists) to see if Policy Routing has been defined. Otherwise, you’ll have to do a recompile.

  55. Worked great for ssh, but borked my apache server. I don’t know why but requests arriving on one device to the apache server will be sent out on the other one. And with the ip rule will try to go the wrong way. Maybe its an apache bug (or feature?)

  56. Since you were using Gentoo:

    How do you setup the IP address in the “admin” routing table during boot? How did your modify your “/etc/conf.d/net.eth0″-file?

    The default gateway can be added with “routes_eth0=( “default via 10.10.70.254 table admin” )

    The rules can be added in the “postup” section of the /etc/conf.d/net; but I simply cannot bring Gentoo to add the IP address in the other table. It either adds it in the default table; or throws a syntax error.

    • Instead of trying to figure out the proper way to add the routing table commands to the network startup scripts, just add them to your /etc/rc.local script (or equivalent). Essentially, you want to use the same absolute commands as listed in the examples (in the article) and write them to a script that will run after *all* automatic networking scripts are ran. On most systems, commands written to /etc/rc.local are literally the last commands executed before the system finished bootup and presents a login prompt to the user.

      • That would be “/etc/conf.d/local.start” on a Gentoo system.

        While it does work, of course, there is a problem if you restart the interface with an “/etc/init.d/net.eth0 restart”, since the routing table will be empty afterwards.

        Alas, at least it works now. Thanks for pointing me in the right direction.

        • Replying to my own comment:

          Using OpenRC with baselayout2 the /etc/conf.d/net.eth0 script would look like this:

          config_eth0=”192.168.1.100 netmask 255.255.255.0 broadcast 192.168.1.255″
          routes_eth0=”default via 192.168.1.1
          192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.150 table dmz
          default via 192.168.1.1 dev eth0 table dmz”
          dns_domain_eth0=”localdomain.local”
          dns_servers_eth0=”dns.ip.1 dns.ip.2″
          metric_eth0=”0″

          Modify to your own pleasure.

  57. Thanks so much for this! I had no idea, until I watched it with tcpdump, that a packet coming in on one interface would get an (attempted) response on another.

    The fix you give worked like a charm.

    –Al Evans

  58. Wow, this worked great! Really cool. Does anybody know if something like this can be done under Windows?

  59. Hi,
    Thanks for the tutorial, it helped a lot! One thing, is there a way to make the routing tables persistent? Or will the newly configured tables still be there after a system reboot?

    Regards,
    mdewet

    • Hi mdewet,

      The policy routing settings should persist across reboots. If, for whatever reason, they do not, you can usually add these settings to /etc/rc.local (or equivalent), so that these settings are applied at the very end of system boot.

      — Darien

  60. Great article. Much appreciated!!!

  61. Hi Darien,

    First of all great post.
    I just have one question. Is it possible to do something similar but with two interfaces connected to the same network? For instance two wireless cards connected to the same access point.

    • Hi rodmar,

      Yes, the same technique can be used on 2 wireless NICs connected to the same access point — Linux treats those interfaces the same when performing policy routing.

      — Darien

  62. Thanks for the tutorial. Does this also work with configuring multiple default routes on vlan interfaces?

    • Hi,
      Yes, this technique can be applied to vlan interfaces as well, because Linux treats those interfaces exactly the same as physical interfaces — just substitute eth0/eth1 with the corresponding vlan interfaces you want to perform policy routing on.
      — Darien

  63. Bless you, O Darien, for this gem on a subject which has stumped not only me but all who I have hit up for the solution, til now!

  64. Bless you, O Darien, for this gem on a subject which has stumped not only me but many who I have hit up for the solution!

  65. Classful routing hasn’t been used since 1994. The fact that the mask is included means it is a classless network. So stop saying ‘class C’, just because the Class C default mask was 255.255.255.0. It is simply a /24 subnet.

  66. Darien, Thnak you for this info..

    Do i have to remove the gateway in the interface file for this to work???

    • Hi Will,
      No, you don’t need to remove the gateway in the interface file for this to work. Essentially, you want to perform the policy route configuration *after* basic networking (including setting the default gateway) has been done. This is why I recommend setting the policy route in /etc/rc.local, so that it is the *last* thing initialized, which guarantees the settings will not conflict with another daemon upon boot up.
      — Darien

      • Can you look at this and see what I’m doing wrong??? I would gladly pay you for your time.
        We have 2 NIC cards and when I try to set the eth1 gatway we lose SHH to the server.
        ETH0 is for (SSH<WEB<ECT.)
        ETH1 is for SIP traffic only
        ifconfig –a
        eth0 Link encap:Ethernet HWaddr 00:1c:23:d0:0b:2c
        inet addr:192.168.1.69 Bcast:192.168.1.255 Mask:255.255.255.0
        inet6 addr: fe80::21c:23ff:fed0:b2c/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
        RX packets:6215 errors:0 dropped:0 overruns:0 frame:0
        TX packets:59877 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:908397 (887.1 KiB) TX bytes:13233015 (12.6 MiB)
        Interrupt:16 Memory:f8000000-f8012100

        eth1 Link encap:Ethernet HWaddr 00:1c:23:d0:0b:2e
        inet addr:192.168.3.70 Bcast:192.168.3.255 Mask:255.255.255.0
        inet6 addr: fe80::21c:23ff:fed0:b2e/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
        RX packets:271 errors:0 dropped:0 overruns:0 frame:0
        TX packets:123 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:181558 (177.3 KiB) TX bytes:13875 (13.5 KiB)
        Interrupt:16 Memory:f4000000-f4012100

        lo Link encap:Local Loopback
        inet addr:127.0.0.1 Mask:255.0.0.0
        inet6 addr: ::1/128 Scope:Host
        UP LOOPBACK RUNNING MTU:16436 Metric:1
        RX packets:197 errors:0 dropped:0 overruns:0 frame:0
        TX packets:197 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:68135 (66.5 KiB) TX bytes:68135 (66.5 KiB)

        vi interfaces
        auto lo
        iface lo inet loopback
        # The primary network interface[m
        auto eth0
        iface eth0 inet static
        address 192.168.1.69
        netmask 255.255.255.0
        gateway 192.168.1.10

        auto eth1
        iface eth1 inet static
        address 192.168.3.70
        netmask 255.255.255.0
        #gateway 192.168.3.11 —-When I try to use this line it locks me out oh SSH

        ip rule show
        0: from all lookup local
        32759: from 192.168.3.70 lookup sip
        32760: from all to 192.168.3.70 lookup sip
        32763: from all to 192.168.1.69 lookup admin
        32764: from 192.168.1.69 lookup admin
        32766: from all lookup main
        32767: from all lookup default

        ip route show
        192.168.3.0/24 dev eth1 proto kernel scope link src 192.168.3.70
        192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.69
        gla

        • Hi Will,
          Your last post looks like some information was cut off, but I think I have enough information to help you.

          1) In /etc/interfaces, do NOT define the gateway for eth1. You are correct that this declaration will cause problems. So, remove the line:
          “gateway 192.168.3.11″ completely from /etc/interfaces.

          2) It looks like you’re trying to define TWO (2) policy tables: “sip” and “admin”. You do NOT need to do this. Please remove *both* “sip” and “admin” tables, and clear out all the custom “ip rule” and “ip route” commands you have defined.

          3) Once you’ve down that, reboot your system and verify that there are NO custom “ip rule show” and “ip route show” commands listed, and that only the eth0 default gateway is showing up in “netstat -anr”.

          4) THEN, type the following commands:

          echo “1 admin” >> /etc/iproute2/rt_tables

          ip route add 192.168.3.0/24 dev eth1 src 192.168.3.70 table admin
          ip route add default via 192.168.3.11 dev eth0 table admin

          ip rule add from 192.168.3.70/32 table admin
          ip rule add to 192.168.3.70/32 table admin

          ip route flush cache

          Hope this helps,
          — Darien

  67. Great article – worked like a charm.
    Thanks very much!

  68. Thanks very much for the excellent info!

    However there was an error for me for the route setting for the default gateway, which I tried to set to same as in the /etc/network/interfaces file, but from an online IP-calculator I found that the gateway should be the first usable host address from the range of machines on the sub network. After changing to that IP, it worked like a charm.

  69. Many thanks for this I have been trying to consolidate some servers and I was combining our Proxy box with our openvpn box for a different subnet. I decided to use Untangle for the proxy as it was a quick easy set up. However, I knew that I would have to run my own instance of openVPN as Untangle only does routed VPNs and I needed a bridged VPN. All setup fine, but then when I pinged to the openVPN bridged interface through a second router from a PC on my LAN which is the one subnet being proxied I was getting no response, because it was killing them, I looked for a way to route the traffic back out the same interface it came in one for a couple of weeks and even posted questions about this at some forums, but your web page here finally provided me the information I needed to do it. I hope you don’t mind I have resolved the questions I asked on the forums and linked back to your article. Many thanks.

  70. Darien – thanks this is very helpful!
    nobodyfamous and howard refer to same subnet and same gatway with 2 (or multiple) NICs. But would anything change in your precription if I want 2 gateways (10.0.0.1 and 10.0.0.2) with 2 NICS (10.0.0.11 and 10.0.0.22) on the same subnet. (This is different from link aggregation).

    To stay with your example – what if eth0 has ip 192.168.7.127
    and gateway 192.168.7.1

    • oops I shouls have said:

      To stay with your example – what if eth0 has ip 192.168.7.127
      and gateway 192.168.7.2

      • Hi Dave,
        Take a look at the comments posted right after nobodyfamous:

        http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/#comment-102

        So, let’s assume:
        eth0 – 192.168.7.127 netmask 255.255.255.0
        eth0’s gateway is: 192.168.7.2
        eth1 – 192.168.7.126 netmask 255.255.255.0
        eth1’s gateway is: 192.168.7.1

        Here are the corresponding commands:
        printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
        printf “2\tuplink1\n” >> /etc/iproute2/rt_tables

        ip route add 192.168.7.127/32 dev eth0 src 192.168.7.127 table uplink0
        ip route add default via 192.168.7.2 dev eth0 table uplink0
        ip rule add from 192.168.7.127/32 table uplink0
        ip rule add to 192.168.7.127/32 table uplink0

        ip route add 192.168.7.126/32 dev eth1 src 192.168.7.126 table uplink1
        ip route add default via 192.168.7.1 dev eth1 table uplink1
        ip rule add from 192.168.7.126/32 table uplink1
        ip rule add to 192.168.7.126/32 table uplink1

        • I have set it up as stated. It boots fine but I am unable to ping 10.0.0.211 (eth1) from another machine on the subnet – but I can ping 10.0.0.201 (eth0). From the machine itself I can ping 10.0.0.211. Here is some output:
          oft201:~# ip rule
          0: from all lookup local
          32766: from all lookup main
          32767: from all lookup default
          oft201:~# vi /etc/rc.local
          oft201:~# /etc/rc.local
          oft201:~# ip rule
          0: from all lookup local
          32762: from all to 10.0.0.211 lookup uplink1
          32763: from 10.0.0.211 lookup uplink1
          32764: from all to 10.0.0.201 lookup uplink0
          32765: from 10.0.0.201 lookup uplink0
          32766: from all lookup main
          32767: from all lookup default
          oft201:~# netstat -nra
          Kernel IP routing table
          Destination Gateway Genmask Flags MSS Window irtt Iface
          10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
          10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
          0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 eth0

          What gives? [I am assuming that policy routing must be enabled in the latest stable debian kernel:
          Linux oft201 2.6.26-2-686-bigmem #1 SMP Thu Sep 16 20:29:07 UTC 2010 i686 GNU/Linux]

            • Darien
            • Posted November 24, 2010 at 6:58 pm
            • Permalink

            Okay, can you paste the contents of your /etc/rc.local file?

            • Darien
            • Posted November 24, 2010 at 7:00 pm
            • Permalink

            Also, I assume you entered “ip route flush cache” and still ran into problems?

            • Dave
            • Posted November 24, 2010 at 7:03 pm
            • Permalink

            I am able to get it to work if the line:
            ip route add 192.168.7.127/32 dev eth0 src 192.168.7.127 table uplink0
            is changed to
            ip route add 192.168.7.0/24 dev eth0 src 192.168.7.127 table uplink0
            and the line
            ip route add 192.168.7.126/32 dev eth1 src 192.168.7.126 table uplink1
            is changed to
            ip route add 192.168.7.0/24 dev eth1 src 192.168.7.126 table uplink1

            • Darien
            • Posted November 24, 2010 at 7:05 pm
            • Permalink

            Okay, glad it’s working for you.

  71. My rc.local with what I beleive to be the incorrect lines commented out:
    #!/bin/sh -e
    #
    # rc.local
    #
    # This script is executed at the end of each multiuser runlevel.
    # Make sure that the script will “exit 0″ on success or any other
    # value on error.
    #
    # In order to enable or disable this script just change the execution
    # bits.
    #
    # By default this script does nothing.

    #ip route add 10.0.0.201/32 dev eth0 src 10.0.0.201 table uplink0
    ip route add 10.0.0.0/24 dev eth0 src 10.0.0.201 table uplink0
    ip route add default via 10.0.0.1 dev eth0 table uplink0
    ip rule add from 10.0.0.201/32 table uplink0
    ip rule add to 10.0.0.201/32 table uplink0

    #ip route add 10.0.0.211/32 dev eth1 src 10.0.0.211 table uplink1
    ip route add 10.0.0.0/24 dev eth1 src 10.0.0.211 table uplink1
    ip route add default via 10.0.0.2 dev eth1 table uplink1
    ip rule add from 10.0.0.211/32 table uplink1
    ip rule add to 10.0.0.211/32 table uplink1

    exit 0

    • Turns out that the above settings work but not as expected.
      If I run ‘ifdown eth1′ and then try
      ping or ssh 10.0.1.211 (the interface associated with eth1) – I can still get through (and vice versa). This may be seen as a feature rather than a bug – but I am spooked by it because I dont understand it. I have abandoned the quest of dual gateway on same subnet. There is no compelling reason for having this anyway. I can live with the original prescription of Darien – which works as expected. (I used the latest debian lenny kernel and did not need to make any special kernel compile to get Dariens prescription working). Thanks again.

  72. Would have been a great article if it had explained where one wrote the admin routing information:

    ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
    ip route add default via 10.10.70.254 dev eth0 table admin

    Which file is this placed into? I don’t know. The article does not say.

    Therefore, I am back to googgeling once more for another site. But at least I know I should Google for policy routing. Thus a partial thank-you.

    • PS. I am using SLES 11. Maybe its obvious on gentoo.

  73. Hi,
    I have had a hard time following the post, and am unsure its what I have to do. Would you be able to have a look at this scenario and add some helpers for it?

    Both interfaces are on different VLANs (Should not make an once of difference). The default route is the one that goes to the commodity Internet, while the 10/8 destinations are for the internal network.

    eth0 IP 10.254.60.90/22
    default 10.254.60.74 255.255.252.0 eth0
    10.0.0.0 10.254.60.51 255.0.0.0 eth0

    eth1 IP 10.254.41.13/22
    default 10.254.40.74 255.255.252.0 eth1
    10.0.0.0 10.254.40.51 255.0.0.0 eth1

    • I’ll take a look, but why are you defining two different netmasks for each interface? That makes no sense.

      If you’re going to define a full Class-A /8, then you don’t need to explicitly define a /22 for the upstream Internet gateway. The /22 serves no purpose in this context.

      I’ll generate some rules that show you what I mean…

      • I think I see what you’re trying to do… each interface has two _virtual_ IPs assigned to it:

        eth0: 10.254.60.90, 10.254.60.51
        eth1: 10.254.41.13, 10.254.40.51

        I’ll adjust accordingly…

        • Actually, can you confirm each interface has two virtual IPs? The way you wrote the initial IP information is confusing…

            • whynot
            • Posted December 21, 2010 at 4:08 pm
            • Permalink

            These are both IPs plumbs on real NICs, not virtual NIC (not eth0:1 etc), and neither are these bonded in anyway.

      • Hi Darie,

        Apologies for the confusion.

        The 10/8 reference was only mean to refer to the second route below for all traffic to the 10.0.0.0 network, and this understandingly caused confusion.

        All networks use /22. The default route is for all traffic other than the 10.0.0.0/22 network, (which is really the public Internet and other private nets 192.168.n.n. and 172.16.n.n networks).

        The current system has one NIC eth0. Its routing table as defined is :
        default 10.254.64.74 255.255.252.0 –
        10.0.0.0 10.254.64.51 255.0.0.0 –

        I have to plumb a new eth1 onto it on another VLAN and keep the traffic separate. (I had never heard of priority routing until now.) The eth1 has to send the default Internet traffic via the router 10.254.40.74 on eth1, and the rest back to the internal LAN via 10.254.40.51 on eth1.

        • ^Hi Darie,^Hi Darien,

        • Okay, ignore my previous post… will provide documentation based on this new information, shortly.

            • whynot
            • Posted December 21, 2010 at 4:13 pm
            • Permalink

            FYI
            These are all routers somewhere on the network:
            10.254.64.74
            10.254.64.51
            10.254.40.74
            10.254.40.51

        • Okay, the real issue is that you have a single IP address defined for each physical interface (eth0 and eth1), and yet you have multiple _overlapping_ netmasks defined for the routes in each interface.

          For example, eth0:
          Class-A /8 – respond directly
          /22 – out to gateway

          This makes the issue not as easy to solve, because normally you would assign the same netmask for each interface — or you could define multiple virtual interfaces with different IP addresses and netmask combinations.

          In any event, I believe the following commands will work without having to define any virtual interfaces, but I have never tested this scenario, so you’ll have to try it out and see for yourself:

          Here are the corresponding commands:
          printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
          printf “2\tuplink1\n” >> /etc/iproute2/rt_tables
          printf “3\tuplink2\n” >> /etc/iproute2/rt_tables
          printf “4\tuplink3\n” >> /etc/iproute2/rt_tables

          ip route add 10.254.60.51/22 dev eth0 src 10.254.60.51 table uplink0
          ip route add default via 10.254.60.74 dev eth0 table uplink0
          ip rule add from 10.254.60.51/22 table uplink0
          ip rule add to 10.254.60.51/22 table uplink0

          ip route add 10.254.60.51/8 dev eth0 src 10.254.60.51 table uplink1
          ip rule add from 10.254.60.51/8 table uplink1
          ip rule add to 10.254.60.51/8 table uplink1

          ip route add 10.254.40.51/22 dev eth1 src 10.254.40.51 table uplink2
          ip route add default via 10.254.40.74 dev eth1 table uplink2
          ip rule add from 10.254.40.51/22 table uplink2
          ip rule add to 10.254.40.51/22 table uplink2

          ip route add 10.254.40.51/8 dev eth1 src 10.254.40.51 table uplink3
          ip rule add from 10.254.40.51/8 table uplink3
          ip rule add to 10.254.40.51/8 table uplink3

            • whynot
            • Posted December 21, 2010 at 11:28 am
            • Permalink

            For example, eth0:
            Class-A /8 – respond directly
            /22 – out to gateway

            Yes, this is exactly this. There is a multitude of overlapping netmasks here.

            I shall try all this on a test server, and then report back here. This’ll be later tomorrow, as its nearly home time here.

    • Okay, here’s the set of rules for this scenario. I assume that each interface actually has _two_ virtual IP addresses. In this case, you would be defining “eth0:0 and eth0:1″ as the virtual interfaces for eth0 and “eth1:0 and eth1:1″ for the virtual interfaces for eth1. If you’re not sure what this means or how to set it up, the SLES documentation about virtual interfaces should help.

      Regardless, instead of thinking about two interfaces, think about this problem in terms of _four_ virtual interfaces. If you split the problem based upon each virtual interface, it’s much easier to solve. See below for details.

      Assumptions:
      eth0:0 – 10.254.60.90 netmask 255.255.252.0
      eth0:0’s gateway is: 10.254.60.74

      eth0:1 – 10.254.60.51 netmask 255.0.0.0

      eth1:0 – 10.254.41.13 netmask 255.255.252.0
      eth1:0’s gateway is: 10.254.40.74

      eth1:1 – 10.254.40.51 netmask 255.0.0.0

      Here are the corresponding commands:
      printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
      printf “2\tuplink1\n” >> /etc/iproute2/rt_tables
      printf “3\tuplink2\n” >> /etc/iproute2/rt_tables
      printf “4\tuplink3\n” >> /etc/iproute2/rt_tables

      ip route add 10.254.60.90/22 dev eth0:0 src 10.254.60.90 table uplink0
      ip route add default via 10.254.60.74 dev eth0:0 table uplink0
      ip rule add from 10.254.60.90/22 table uplink0
      ip rule add to 10.254.60.90/22 table uplink0

      ip route add 10.254.60.51/8 dev eth0:1 src 10.254.60.51 table uplink1
      ip rule add from 10.254.60.51/8 table uplink1
      ip rule add to 10.254.60.51/8 table uplink1

      ip route add 10.254.41.13/22 dev eth1:0 src 10.254.41.13 table uplink2
      ip route add default via 10.254.40.74 dev eth1:0 table uplink2
      ip rule add from 10.254.41.13/22 table uplink2
      ip rule add to 10.254.41.13/22 table uplink2

      ip route add 10.254.40.51/8 dev eth1:1 src 10.254.40.51 table uplink3
      ip rule add from 10.254.40.51/8 table uplink3
      ip rule add to 10.254.40.51/8 table uplink3

  74. Apologies. I;m confused.

    Actually, your original thread was so close what I want to do, yet the ip rules you defined only ever mentioned eth0:
    # eth0 – 10.10.70.38 netmask 255.255.255.0
    # eth0’s gateway is: 10.10.70.254
    But never mentioned eth1:
    # eth1 – 192.168.7.126 netmask 255.255.255.0
    # eth1’s gateway is: 192.168.7.1

    Since the eth1 is not mentioned, then I presume that it has to be indirectly decided upon by iproute, but this is where I lost the plot. There are only these entries:
    ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
    ip route add default via 10.10.70.254 dev eth0 table admin
    ip rule add from 10.10.70.38/32 table admin
    ip rule add to 10.10.70.38/32 table admin
    Nor does it say what to add or remove for the /etc/sysconfig/network/routes table, which currently has some routing information in it

    Anyway, cheers for your time.

    • The default route for anything other than that subnet specified in the table is that which is set for the network. In other words: if those rules don’t apply to the packets, they head on out eth1 towards 192.168.7.1 (unless the packets came from the 7.x network, in which case they would head on over to the machine that made the request.)

  75. This may be an old post, but wonderfully explains how to properly set up the routes. I’m running Ubuntu 10.04. One note, if it’s not clear, I defined a gateway (192.168.1.1) for eth0 as my default in the /etc/network/interfaces file. For eth1, I did not define a gateway, but rather set up the route as noted above. Again, many thanks.

  76. Excellent article. It served most of my problem, specially where multiple vlan interfaces setup for different network.

  77. Splendid post!! Thanks =))

  78. Thanks.

  79. This is really great. Finally got my bridged virtual VBOX server environment connected to the internet through a separate nic, leaving my primary nic for the physical ubuntu server access only.

    This makes my virtual business environment finally completely safe and separate from my physical private environment.

    Thanks a lot !

    (If someone wants the setup details, leave a reply and I will post them.)

  80. Two years later, and this is the best post i found to accomplish the job without need to learn all about iptables, routes, rules, etc of a linux box (but still is necesary ;)).
    Thank you very much. You make my day a happy day ;)

  81. Thank you Thank you Thank you,
    I’ve been trying to shovel several vlans across as many openvpn tunnels on a virtual server without success for what felt like eternity, now in 2 minutes flat it actually does stuff.

    Thanks again

  82. Man, Thank You !!!!!!

    I spent days trying to solve this. This worked right away.

  83. Darien,
    Thanks to you for great article and thanks to everyone else for some great reading.

    Set this up on BackTrack 5 – 32 bit and it worked fine. Now trying the same thing on BT5 R1 – 64 bit and it’s not playing

    I have two distinct networks and never the twain shall meet.

    My home network (eth0) is a 192.168.2.0/24 with the gateway as 192.168.2.1 and this connects to the internet
    My ‘special’ network (eth1) is a 10.0.0.16/28 with the gateway as 10.0.0.17 and this connects nowhere due to some of the software sitting on the VM’s that live on this network (vm’s are fixed to eth1 only using VMWare Virtual Network Editor). Both nics have static ip’s though it did work with DHCP when I had it running pre 64 bit install.

    Here is what i’m getting
    have added following to rc.local
    ip route add 192.168.2.0/24 dev oth0 src 192.168.2.30 table admin
    ip route add default via 192.168.2.1 dev eth0 table admin
    ip rule add from 192.168.2.30/32 table admin
    ip rule add to 192.168.2.30/32 table admin
    ip route flush cache

    ip rule show
    0: from all lookup local
    32764: from all to 192.168.2.30 lookup admin
    32765: from 192.168.2.30 lookup admin
    32766: from all lookup main
    32767: from all lookup default

    ip route show
    default via 192.168.2.1 dev eth0 metric 100
    default via 10.0.0.17 dev eth1 metric 100
    10.0.0.16/28 dev eth1 proto kernel scope link src 10.0.0.20
    192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.30

    netstat -anr
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt Iface
    0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 eth0
    0.0.0.0 10.0.0.17 0.0.0.0 UG 0 0 0 eth1
    10.0.0.16 0.0.0.0 255.255.255.240 U 0 0 0 eth1
    192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

    With eth1 enabled, I can’t access the internet or if I use ‘route del default gw 10.0.0.17′ it will let me connect to the internet until next time I reboot. I had it working before and am now baffled as to why it’s playing silly buggers. Have I missed something obvious above or can you think of something else I can try?

    This system is a training rig and I need to be able to access the internet from the main machine whilst using the VM’s on their special network.

    • Here’s your problem:

      “netstat -anr

      0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 eth0
      …”

      You’ve _already_ specified a default route in the _admin_ table for eth0; as such, you should NOT specify a default route for eth0 in the _main_ table.

      To correct this, remove the following command [FLAGGED] from your list:
      “ip route show
      default via 192.168.2.1 dev eth0 metric 100 <— [REMOVE THIS ONE]
      default via 10.0.0.17 dev eth1 metric 100
      10.0.0.16/28 dev eth1 proto kernel scope link src 10.0.0.20
      192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.30"

  84. Thanks! This article really helped me.

  85. Thanks man ! you saved my life :) i was looking this information from a long time ago.

    cheers!

  86. hi, and thank you for your post! i have a question….

    eth0 – ip 10,60,90,45 gw 10,60,90,1
    eth1 – ip 10,60,91,45 gw 10,60,91,1
    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    10.60.90.0 * 255.255.255.0 U 0 0 0 eth0
    10.60.91.0 * 255.255.255.0 U 0 0 0 eth1
    link-local * 255.255.0.0 U 0 0 0 eth0
    loopback * 255.0.0.0 U 0 0 0 lo
    default 10.60.90.1 0.0.0.0 UG 0 0 0 eth0

    if i need that a machine on 69,50,20,xx subnet can connect to 10,60,91,45 i add a new route

    69,51,20,0 10,60,91,1 255,255,255,0 UG 0 0 0 eth1

    but if i don’t know all the subnets or more easy, how can i do to all traffic that goes through an interface is returned by the same interface.

    • Hi sun,
      Take a look at the comments posted right after nobodyfamous:

      http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/#comment-102

      So, let’s assume:
      eth0 – 10.60.90.45 netmask 255.255.255.0
      eth0′s gateway is: 10.60.90.1

      eth1 – 10.60.91.45 netmask 255.255.255.0
      eth1′s gateway is: 10.60.91.1

      Here are the corresponding commands:

      printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
      printf “2\tuplink1\n” >> /etc/iproute2/rt_tables

      ip route add 10.60.90.0/24 dev eth0 src 10.60.90.45 table uplink0
      ip route add default via 10.60.90.1 dev eth0 table uplink0
      ip rule add from 10.60.90.45/32 table uplink0
      ip rule add to 10.60.90.45/32 table uplink0

      ip route add 10.60.91.0/24 dev eth1 src 10.60.91.45 table uplink1
      ip route add default via 10.60.91.1 dev eth1 table uplink1
      ip rule add from 10.60.91.45/32 table uplink1
      ip rule add to 10.60.91.45/32 table uplink1

      ip route flush

  87. THANK YOU

  88. Thank you! Your page made it so simple to fix my routing dilemma! I had two NICs on two separate dmz networks and needed to keep traffic on the the same NIC it started on. And have a default gateway for each NIC. This helped! Great Work!

  89. Four years on, this is still the clearest guide to this. It just works.

    Thanks.

  90. Thanks for the article.

    Unfortunately it does not work in my setup.
    How can I make sure my kernel has support for policy routing?
    I use Proxmox Virtualization Environment (http://pve.proxmox.com) with kernel 2.6.32-6-pve.

    My setup shows:

    vserver3:~# ip rule show
    0: from all lookup local
    32764: from all to 192.44.37.62 lookup vmbr1
    32765: from 192.44.37.62 lookup vmbr1
    32766: from all lookup main
    32767: from all lookup default

    vserver3:~# ip route show
    192.168.20.0/24 dev vmbr101 proto kernel scope link src 192.168.20.1
    192.44.37.0/24 dev vmbr1 proto kernel scope link src 192.44.37.62
    192.168.19.0/24 dev vmbr100 proto kernel scope link src 192.168.19.1
    153.96.8.0/24 dev vmbr0 proto kernel scope link src 153.96.8.76
    default via 153.96.8.1 dev vmbr0

    vserver3:~# ip route show table vmbr1
    192.44.37.0/24 dev vmbr1 scope link src 192.44.37.62
    default via 192.44.37.101 dev vmbr1

    I have vmbr100 connected via iptables NAT rules to vmbr0
    and vmbr101 via iptables NAT rules to vmbr1.

    Connectivity of vmbr0 and vmbr100 works.
    Connectivity of vmbr1 and vmbr101 is limited to network 192.44.37.0/24.

    Any ideas? Thanks.

    • Hi Birger,

      “vmbr*” interfaces are not standard network interfaces (http://pve.proxmox.com/wiki/Vlans) — they are bridging interfaces, which is why policy routing does NOT work properly.

      In order to use policy routing, you need to assign one or more NICs (eth*) interfaces to a bridging interface (vmbr*), and then assign the routing to the attached eth* interface, accordingly.

  91. Hi Darien,

    With respect to multiple default routes on OpenBSD: the virtual tables are primarily meant to create separate logical networks over top of a common base network. For instance you could create a private network for contractors which would keep them logically separated from your employees.

    However within a routing table, OpenBSD actually supports equal-cost multipath route entries. The kernel will choose a path based on the result of a hash and send the packets on their way. It’s really easy to setup and does not require the complexity of virtual tables. Details are in the OpenBSD FAQ (http://www.openbsd.org/faq/faq6.html#Multipath).

  92. Thanks for the feedback.

    Special Thanks to Darien for the very valuable blog post.

    Everything works as expected. We tested the configuration with the command “ping -I vmbr1 …”, which gave an error “Destination host unreachable”. Now we found that ping is buggy.

    “ping -I 192.44.37.62 …” works as expected. “wget -q -O – -bind-address=192.44.37.62 http://whatismyip.org” works also.

    It does work with bridging interfaces and it does work in combination with NAT.

    • Okay, glad to hear it works with bridging interfaces… I wasn’t 100% sure of this, as I had never tested it before.

  93. thx for this. you made my day. if you got 2 nics make sure you add two route tables

    # echo “1 uplink0″ >> /etc/iproute2/rt_tables
    # echo “2 uplink1″ >> /etc/iproute2/rt_tables

    # ip route add 192.168.0.0/24 dev eth0 src 192.168.0.101 table uplink0
    # ip route add default via 192.168.0.1 dev eth0 table uplink0
    # ip rule add from 192.168.0.101/32 table uplink0
    # ip rule add to 192.168.0.101/32 table uplink0

    # ip route add xxx.xxx.xxx.0/24 dev eth3 src xxx.xxx.xxx.219 table uplink1
    # ip route add default via xxx.xxx.xxx.1 dev eth3 table uplink1
    # ip rule add from xxx.xxx.xxx.219/32 table uplink1
    # ip rule add to xxx.xxx.xxx.219/32 table uplink1
    (xxx.xxx.xxx. replace with your settings :P)

    to make it permanent under gentoo you can use

    # nano /etc/local.d/network.start (http://de.gentoo-wiki.com/wiki/Local.d)
    (add #!/bin/sh & the ip xxx commandos; check with rc-update show if the local is added to the default level)
    # chmod +x /etc/local.d/network.start

    have fun and enjoy!

  94. Hi. Great post and the comment stream is going strong…

    I used your post to set up my system.

    I have a gateway box with 2 NICs.
    One nic sits on the internal network (eth1; 192.168.0.7), the other nic (eth0; 110.xxx.xxx.173) connected to a 10 MBit pipe used for incoming VPN.
    There is an ADSL router (192.168.0.1) sitting on the network as well…
    Everyone on the network uses 192.168.0.1 as their gateway.
    I want people to be able to set 192.168.0.7 as their gateway if they have special speed needs.
    So what happens is that when the user’s (say user 192.168.0.10) gateway is set to 192.168.0.7 the traffic still goes via 192.168.0.1… whereas I want it to go through 110.xxx.xxx.173…

    Host Loss% Snt Last Avg Best Wrst StDev
    1. gateway1.e3(192.168.0.7) 0.0% 4 0.1 0.2 0.1 0.2 0.0
    2. firewall.e3(192.168.0.1) 0.0% 4 0.4 0.5 0.4 0.5 0.0
    3. ras8.nap.net.au 0.0% 3 30.0 146.9 30.0 263.7 165.2
    4. g3.cor1.adp.net.au 0.0% 3 280.6 229.0 140.8 280.6 76.8
    5. g1.ap.net.au 50.0% 3 155.1 155.1 155.1 155.1 0.0

    here are my configs:

    /etc/iproute2/rt_tables:

    2 gatewayeth1
    3 gatewayeth0

    /etc/network/interfaces:
    # The primary network interface
    auto eth1
    iface eth1 inet static
    address 192.168.0.7
    netmask 255.255.255.0
    gateway 192.168.0.1

    # 10Mbit interface
    auto eth0
    iface eth0 inet static
    address 110.xxx.xxx.174
    netmask 255.255.255.252
    gateway 110.xxx.xxx.173

    /etc/rc.local:
    ip route add 110.xxx.xxx.172/30 dev eth0 src 110.xxx.xxx.174 table gatewayeth0
    ip route add default via 110.xxx.xxx.173 dev eth0 table gatewayeth0
    ip route add 192.168.0.0/24 dev eth1 src 192.168.0.7 table gatewayeth1
    ip route add default via 192.168.0.1 dev eth1 table gatewayeth1

    ip rule add from 110.xxx.xxx.174/32 table gatewayeth0
    ip rule add to 110.xxx.xxx.174/32 table gatewayeth0
    ip rule add from 192.168.0.7/32 table gatewayeth1
    ip rule add to 192.168.0.7/32 table gatewayeth1

    iptables rules:
    #! /bin/bash

    iptables –flush

    #NAT TABLE
    iptables -t nat –policy PREROUTING ACCEPT
    iptables -t nat –policy POSTROUTING ACCEPT
    iptables -t nat –policy OUTPUT ACCEPT
    iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
    iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

    #FILTER TABLE
    iptables –policy INPUT DROP
    iptables –policy OUTPUT DROP
    iptables –policy FORWARD DROP

    # localhost rules
    iptables -A INPUT -i lo -j ACCEPT
    iptables -A OUTPUT -o lo -j ACCEPT

    # in chain for eth0 (outward facing telstra port on network 110.xxx.xxx.172/30)

    iptables -X IN_RULES_ETH0
    iptables -N IN_RULES_ETH0
    iptables -A INPUT -i eth0 -j IN_RULES_ETH0

    # in chain for eth1 (inward facing internal network 192.168.0.0/24)

    iptables -X IN_RULES_ETH1
    iptables -N IN_RULES_ETH1
    iptables -A INPUT -i eth1 -j IN_RULES_ETH1

    # out chain

    iptables -X OUT_RULES
    iptables -N OUT_RULES
    iptables -A OUTPUT -j OUT_RULES

    # forward chain for eth1

    iptables -X FORWARD_ETH1
    iptables -N FORWARD_ETH1
    iptables -A FORWARD -j FORWARD_ETH1

    # drop stuff for eth0
    # IN_RULES_ETH0
    iptables -A IN_RULES_ETH0 -s 10.0.0.0/8 -j DROP
    iptables -A IN_RULES_ETH0 -s 172.16.0.0/12 -j DROP
    iptables -A IN_RULES_ETH0 -s 192.168.0.0/16 -j DROP
    iptables -A IN_RULES_ETH0 -s 224.0.0.0/4 -j DROP
    iptables -A IN_RULES_ETH0 -s 240.0.0.0/5 -j DROP
    iptables -A IN_RULES_ETH0 -s 127.0.0.0/8 -j DROP
    iptables -A IN_RULES_ETH0 -s 0.0.0.0/8 -j DROP
    iptables -A IN_RULES_ETH0 -d 255.255.255.255 -j DROP
    iptables -A IN_RULES_ETH0 -s 169.254.0.0/16 -j DROP

    # drop stuff for eth1 allow only a few addresses
    # IN_RULES_ETH1
    iptables -A IN_RULES_ETH1 -m iprange –src-range 192.168.0.2-192.168.0.9 -j DROP
    iptables -A IN_RULES_ETH1 -m iprange –src-range 192.168.0.11-192.168.0.26 -j DROP
    iptables -A IN_RULES_ETH1 -m iprange –src-range 192.168.0.27-192.168.0.59 -j DROP
    iptables -A IN_RULES_ETH1 -m iprange –src-range 192.168.0.61-192.168.0.254 -j DROP
    iptables -A IN_RULES_ETH1 -s 169.254.0.0/16 -j DROP

    # drop stuff out rules
    iptables -A OUT_RULES -d 224.0.0.0/4 -j DROP

    # allow previously established / related connections
    iptables -A OUT_RULES -m state –state ESTABLISHED,RELATED -j ACCEPT
    iptables -A IN_RULES_ETH0 -m state –state ESTABLISHED,RELATED -j ACCEPT
    iptables -A IN_RULES_ETH1 -m state –state ESTABLISHED,RELATED -j ACCEPT

    # forward all on eth1
    iptables -A FORWARD_ETH1 -i eth1 -j ACCEPT
    iptables -A FORWARD_ETH1 -o eth1 -j ACCEPT

    ####################################################################
    # out services / ports

    # ICMP
    iptables -A OUT_RULES -p icmp -j ACCEPT
    # Port 22 (SSH)
    iptables -A OUT_RULES -p tcp –dport 22 -j ACCEPT
    # Port 1723 (VPN)
    iptables -A OUT_RULES -p tcp –dport 1723 -j ACCEPT

    # Port 123 (NTP)
    iptables -A OUT_RULES -p udp –dport 123 -j ACCEPT
    iptables -A OUT_RULES -p tcp –dport 123 -j ACCEPT

    iptables -A OUT_RULES -j LOG –log-prefix “BLOCKED OUTPUT ”
    iptables -A OUT_RULES -j REJECT

    ####################################################################
    # input services / ports

    # ICMP
    iptables -A IN_RULES_ETH0 -p icmp -j ACCEPT
    iptables -A IN_RULES_ETH1 -p icmp -j ACCEPT

    # Port 22 (SSH)
    iptables -A IN_RULES_ETH0 -p tcp –dport 22 -m state –state NEW -j ACCEPT
    iptables -A IN_RULES_ETH1 -p tcp –dport 22 -m state –state NEW -j ACCEPT

    # Port 1723 (VPN)
    iptables -A IN_RULES_ETH0 -p tcp -m tcp –dport 1723 -j ACCEPT
    iptables -A IN_RULES_ETH0 -p gre -j ACCEPT

    ######################################################################

    # Drop MS NetBIOS scanning early (it spams the log)
    iptables -A IN_RULES_ETH0 -p tcp –dport 137 -j DROP
    iptables -A IN_RULES_ETH1 -p tcp –dport 137 -j DROP
    iptables -A IN_RULES_ETH0 -p udp –dport 137 -j DROP
    iptables -A IN_RULES_ETH1 -p udp –dport 137 -j DROP
    iptables -A IN_RULES_ETH0 -p tcp –dport 138 -j DROP
    iptables -A IN_RULES_ETH1 -p udp –dport 138 -j DROP

    # Log and reject
    iptables -A IN_RULES_ETH0 -j LOG –log-prefix “BLOCKED INPUT ”
    iptables -A IN_RULES_ETH1 -j LOG –log-prefix “BLOCKED INPUT ”
    iptables -A IN_RULES_ETH0 -j REJECT
    iptables -A IN_RULES_ETH1 -j REJECT

    Thanks.
    Hopefully you can help me…

    Best
    Luke.

    • > Hopefully you can help me…

      Luke, I need some additional clarification… I assume you posted this information because multiple default routing for you isn’t working correctly — is that correct? If so, can you clarify as to what is and is not working correctly? (e.g. which route works properly?)

      Also, remove _all_ the firewall rules and just test to make sure multiple default routing works from the 192.168.0.7 system, first (without any further NAT’ing). If multiple default routes for that host _fail_ to work, then the problem isn’t with your iptables rules — however, if multiple default routing _works_ from (just) the host without the iptable rules, then you’ve isolated the problem to your iptables rules.

      The overall idea here, is to reduce the number variables in your configuration, in order to identify the root cause.

      Regards,
      — Darien

      • Hi Darien. Thanks for your reply.
        The problem is that users on the 192.168.0.0 network who set their gateway to 192.168.0.7 should have their traffic go straight out through the 110.xxx.xxx.173 interface, but it goes first to the 192.168.0.1 interface…
        I will try what you suggest and get back here…

      • Ive removed the gateway statement from the local NIC for 192.168.0.7 and it seems to have fixed the problem.
        I thought that having routes added later (as in /etc/rc.local) would have overridden the default gateway set in /etc/network/interfaces but apparently not…

      • Hi Darien. Lastly is there a way I can edit my posts … I seem to have mentioned my ip address in my original post, although I thought I had edited them all, one still remains, could you replace with xxx… thanks

        • Okay, this is fixed..

      • Thanks. You’re the man!

  95. I nee a help, I m delete device from ‘network configuration’ and again add it. It may be wrong so I unable to ping eventhough default gateway of my router. so plz guide me to configure properly to access internet via netgare 150 adsl router. plz replay in mail id : paradise_437@yahoo.com

  96. anybady online

  97. Thanks Darien, this has moved things forward for me. I hope that someone can help with the 1 (!) remainng problem.

    I have eth0 on the 10.10.170 subnet and eth1 on the 10.10.177 subnet. There is more than 1 gateway on each subnet, for reasons I won’t go into. The suggestion in the initial post, modified for my circumstances, allowed me to ssh and ping into either interface. So far, so good. I wasn’t able to do that before. Now however, I’m having trouble transitting from one subnet to the other. I wanted to use the 10.10.177.1 gateway, and that is how I set it up. However, a tracert tries to send me through the 10.10.177.2 gateway. I’ve searched the system and there is no mention of that ip address anywhere to be found.

    Here’s the commands I used.

    ip route add 10.10.177.0/24 dev eth1 src 10.10.177.20 table voice
    ip route add default via 10.10.177.1 dev eth1 table voice
    ip rule add from 10.10.177.20/32 table voice
    ip rule add to 10.10.177.20/32 table voice

    This is under Centos 6

    Cheers,
    Chris

    • Hi Chris,

      I’ll need more information about your network setup, including the configuration used to setup eth0.

      Regards,
      — Darien

  98. Hi, i’ve follow your step, but when i run /etc/init.d/networking restart , the ip table admin is gone, how to save it?
    thanks

      • Thanks for the ansewer, but i still confuse about this (i’m new in ubuntu) , i’ve problem about connection, you can see here (please) http://i41.tinypic.com/nl2as6.jpg

        in that case, i want computer on lan, can connect through squid proxy to the internet, when i setting up without client connect, it can’t ping to LAN because of 10.10.7.0 default route (that connect to internet)
        when i disable that default route, i can successfully ping to lan, But link to internet is off.

        How could i make two connection, so i can connect to LAN and the internet, i refer to your site but still not work.

        Many thanks

        • Hi sandm3n,

          The problem is that your Squid proxy is running inside VMware. I assume you are running VMware Workstation. The problem you are running into is specific to VMware’s network configuration.

          1) You need to change eth0 to use a vmnet that is _bridged_ with a physical network interface card (NIC).
          2) You need to change eth1 to use a vmnet that is _bridged_ with a _different_ physical network interface card (NIC).

          Because you are using (host-only) and (NAT) for these interfaces, VMware is not letting the policy routing settings work correctly.

          • i will let you know my whole simulated network, see here http://i40.tinypic.com/2f061hd.jpg , using GNS3 virtual box edition with cisco router and inter vlan on it, i use Virtual Box as client, you can see cloud on the right that is connect to vmware (ubuntu server), client to client between vlan can connect .
            first question is my topology correct? on the cloud (server) i run Samba file sharing and Squid (not running)
            Do i must running samba and squid separately ?
            many thanks!

            • Darien
            • Posted January 20, 2012 at 11:41 am
            • Permalink

            Hi sandm3n,

            You have enough complex components that I’d recommend testing the multiple default route settings on the Ubuntu server in _isolation_ from the rest of your network. The samba and squid services running on the Ubuntu server do not matter.

            Specifically, if you have two separate clients connected directly to the two interfaces on the Ubuntu server, and you can ping each interface separately from each client, then your multiple default route settings should be working properly.

            Once you have established that this works, you can slowly increase the complexity of your network architecture and re-test these basic checks at each step (e.g., pinging each interface). That way, when the test fails, you will be able to isolate which component is responsible for the problem.

          • Thanks for your nice advice, want to test your suggest first, i’ve to learn more about this, sorry if i bothering you Darien.

  99. Great Article…

    Thank you

    I bookmarked this page

    regards
    Samy

  100. Wow, this absolutely saved my bacon; thanks for posting this. So easy to do and works like a champ (once you add everything into rc.local that is!).

  101. I apologize if the question is stupid, I’m new with linux/ubuntu.
    I have a pc (ubuntu-server) with two network cards (eth0 and eth1).
    eth1 (ip 192.168.1.40) is connected to the internet (ADSL) via a gateway-router (ip 192.168.1.1).
    eth0 (ip 192.168.0.5) is connected to the Internet (another provider DSL) via another gateway-router (ip 192.168.0.1).
    I use eth1 for all internet and local connections.
    I just want to do a reverse ssh tunnel “ssh -R 9999: localhost: 22 xxxxx@123.432.xxx.xxx” that go out to 123.432.xxx.xxx through eth0 so that if I’m connecting (from another PC on the internet) to port localhost:9999 of 123.432.xxx.xxx I am redirected to my pc/ubuntu-server at it’s port 22 using the internet line of eth0 which has an upload bandwidth much greater than ADSL to which eth1 is connected.
    How can I do this?
    I must do what we are talking about in this article? Or just a rule of static route? Can you help? thanks infinitely

    • Hi Jack,

      If you want all traffic destined for “123.132.x.x” to go out eth0, then you would need to create a static route that forces all “123.132.0.0/16″ out through eth0, accordingly.

      Specifically, the command is:
      route add -net 123.132.0.0 netmask 255.255.0.0 dev eth0

      • Thanks, it works fine.
        I used:

        route add -host xxx.xxx.xxx.xxx gw 192.168.0.1 dev eth0

        and to make it permanent I added to /etc/network/interfaces:

        up route add -host xxx.xxx.xxx.xxx gw 192.168.0.1 dev eth0

        where “xxx.xxx.xxx.xxx” is the ip of a ssh server.

  102. Very Nice info, neatly explained.

  103. Good day Darien,

    I nearly have the same situation you described.
    I have a strato vserver (2.6.18-028stab091.2) OpenSuSE 11.1:

    >ifconfig
    venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
    inet addr:127.0.0.1 P-t-P:127.0.0.1 Bcast:0.0.0.0 Mask:255.255.255.255
    UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1

    venet0:0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
    inet addr:85.214.156.xxx P-t-P:85.214.156.xxx Bcast:0.0.0.0 Mask:255.255.255.255
    UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1

    venet0:1 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
    inet addr:81.169.166.xxx P-t-P:81.169.166.xxx Bcast:0.0.0.0 Mask:255.255.255.255
    UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1

    >uname -a
    Linux hxxxxxxx 2.6.18-028stab091.2 #1 SMP Fri Jun 3 00:02:40 MSD 2011 i686 athlon i386 GNU/Linux

    >netstat -rn
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt Iface
    127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
    0.0.0.0 0.0.0.0 0.0.0.0 U 0 0 0 venet0

    >route
    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    loopback * 255.0.0.0 U 0 0 0 lo
    default * 0.0.0.0 U 0 0 0 venet0

    >cat /etc/sysconfig/network/ifcfg-venet0
    STARTMODE=onboot
    BOOTPROTO=static
    BROADCAST=0.0.0.0
    NETMASK=255.255.255.255
    IPADDR=127.0.0.1
    IPADDR_0=85.214.156.xxx
    LABEL_0=0
    IPADDR_1=81.169.166.xxx
    LABEL_1=1

    >cat /etc/sysconfig/network/ifroute-venet0
    default – – venet0
    default :: – venet0

    venet0:0 – IP 85.214.156.xxx netmask 255.255.255.255
    venet0:0 – gateway is: 0.0.0.0
    venet0:1 – IP 81.169.166.xxx netmask 255.255.255.0
    venet0:1 – gateway is: 0.0.0.0

    My goal is to do exactly like that:

    http://www.linuxquestions.org/questions/linux-networking-3/ssh-tunneling-with-multiple-ips-720679/

    Case 1) User (A) connects with putty sock5 tunnel to venet0:0 (outgoing traffic has to go back venet0:0)
    “C:\Program Files\putty\plink.exe” -ssh -T -D 7777 -load
    I configured my firefox for case 1)
    ->Options->Advanced->Settings..->Manual proxy configuation->SOCKS Host: localhost, Port: 7777
    Now I can browse through that tunnel with the IP of linux server venet0:0
    Test with http://whatismyipaddress.com shows 85.214.156.xxx -> OK.

    Case 2) User (B) connects with putty sock5 tunnel to 2nd IP (outgoing traffic has to go back venet0:1)
    “C:\Program Files\putty\plink.exe” -ssh -T -D 8888 -load
    I configured my firefox for case 2)
    ->Options->Advanced->Settings..->Manual proxy configuation->SOCKS Host: localhost, Port: 8888
    Now I want to browse through that tunnel with the IP of linux server venet0:1

    I tried to setup you wrote above, but the case 2) always show the IP of venet0:0
    Hope you and the other can help me.

    Thanks and regards,
    Tien.

    • Hi Tien,

      Some of the information you provided in your example conflicts.

      1) Earlier, you had said:
      > venet0:1 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
      > inet addr:81.169.166.xxx P-t-P:81.169.166.xxx Bcast:0.0.0.0 Mask:255.255.255.255
      > UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1

      Notice that the netmask for this interface is 255.255.255.255

      Then, you said:
      > venet0:1 – IP 81.169.166.xxx netmask 255.255.255.0
      > venet0:1 – gateway is: 0.0.0.0

      In this case, the netmask is 255.255.255.0.

      Which is it? The netmask is either 255.255.255.255 or 255.255.255.0, but not both.

      2) Also, both venet0:0 and venet0:1 need valid gateways:
      venet0:0 – gateway is: 0.0.0.0
      venet0:1 – gateway is: 0.0.0.0

      The gateways you specified “0.0.0.0” are not valid, in this case. You need specific IP addresses for each gateway on each interface.

      3) Therefore, let’s assume you had the following information:
      venet0:0 – 85.214.156.2 netmask 255.255.255.0
      venet0:0’s gateway is: 85.214.156.1

      venet0:1 – 81.169.166.2 netmask 255.255.255.0
      venet0:1’s gateway is: 81.169.166.1

      Here are the corresponding commands you would run:

      printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
      printf “2\tuplink1\n” >> /etc/iproute2/rt_tables

      ip route add 85.214.156.0/24 dev eth0 src 85.214.156.2 table uplink0
      ip route add default via 85.214.156.1 dev venet0:0 table uplink0
      ip rule add from 85.214.156.2/32 table uplink0
      ip rule add to 85.214.156.2/32 table uplink0

      ip route add 81.169.166.0/24 dev eth1 src 81.169.166.2 table uplink1
      ip route add default via 81.169.166.1 dev venet0:1 table uplink1
      ip rule add from 81.169.166.2/32 table uplink1
      ip rule add to 81.169.166.2/32 table uplink1

      ip route flush

      • Hi Darien,
        Many thanks for the feedback.
        Yes you’re right, I pasted wrong regarding the netmask.
        Here is the right one:

        venet0:0 – 85.214.156.2 netmask 255.255.255.255
        venet0:0′s gateway is: 0.0.0.0

        venet0:1 – 81.169.166.2 netmask 255.255.255.255
        venet0:1′s gateway is: 0.0.0.0

        I asked the guys from Strato provider (http://www.strato.de/server/virtual-linux-server/) to give me the gateways, they said this is a v-server and has gateway = 0.0.0.0
        With the route command I got that:
        >route
        Kernel IP routing table
        Destination Gateway Genmask Flags Metric Ref Use Iface
        loopback * 255.0.0.0 U 0 0 0 lo
        default * 0.0.0.0 U 0 0 0 venet0

        As I can see in your example with eth0 & eth1, I do only have venet0, venet0:0 & venet0:1

        • Hi Tien,

          I’m skeptical that the real gateway is 0.0.0.0… The best way to know for sure, would be to run tcpdump on your two interfaces (venet0:0 and venet0:1), and see what the destination IP address is, when you issue a traceroute to a server on the internet (e.g., “traceroute yahoo.com”).

          So, to be clear, the steps would be:

          1) tcpdump -ni venet0:0
          2) traceroute yahoo.com
          3) Look at the destination IP addresses listed in the traceroute output and packet dump

          You can repeat these steps for venet0:1, also.

            • Anonymous
            • Posted May 14, 2012 at 10:52 pm
            • Permalink

            Hi,Darien
            Very thanks for your article,i have same situation and i need your help.my vps have 2ips,and they all use a NIC venet0,vent0:0 vent0:1
            DEVICE = venet0
            BOOTPROTO = static
            ONBOOT = yes
            IPADDR = 127.0.0.1
            NETMASK = 255.255.255.255
            BROADCAST = 0.0.0.0
            IPV6INIT = “yes”

            DEVICE = venet0:0
            BOOT = yes
            IPADDR=119.XX.XX.206
            NETMASK = 255.255.255.255

            DEVICE = venet0:1
            ONBOOT = yes
            IPADDR = 119.XX.XX.245
            NETMASK = 255.255.255.255
            how to achive accessing ip1 get ip1,acessing ip2 get ip2? I think i need a correct code similar the following for my case.
            —-quote—
            printf “1\tuplink0\n” >> /etc/iproute2/rt_tables
            printf “2\tuplink1\n” >> /etc/iproute2/rt_tables

            ip route add 85.214.156.0/24 dev eth0 src 85.214.156.2 table uplink0
            ip route add default via 85.214.156.1 dev venet0:0 table uplink0
            ip rule add from 85.214.156.2/32 table uplink0
            ip rule add to 85.214.156.2/32 table uplink0

            ip route add 81.169.166.0/24 dev eth1 src 81.169.166.2 table uplink1
            ip route add default via 81.169.166.1 dev venet0:1 table uplink1
            ip rule add from 81.169.166.2/32 table uplink1
            ip rule add to 81.169.166.2/32 table uplink1

            ip route flush
            —–quote—-
            pls write a correct codes smilar to the above,thanks very much. i am not good at programming and linux so i expect to get a very correct code and input in my vps.pls attention to if the “eth0″,”eth1″ is correct,because i pass these codes to my providers to help me they told me the eth0 cause error.eth0 don’t exist and replace it with venet0:0 cause error :RTNETLINK answers:invalid argument,so looking forwards to your help.

            Best Regards
            Thomas

            • Darien
            • Posted May 22, 2012 at 8:56 am
            • Permalink

            Hi Thomas, what is the default gateway for each interface in your setup?

            • Martin Kealey
            • Posted June 11, 2012 at 1:41 am
            • Permalink

            vservers are “thin” virtual machines which hand off to the real machine to do actual routing, so the “gateway” is meaningless and left as 0.0.0.0. If you want policy routing it would have to be done in the real machine.

            • Darien
            • Posted June 11, 2012 at 3:34 am
            • Permalink

            Thanks for the clarification, Martin!

  104. I am connected to 2 interfaces simultaneously like eth0 and wlan0. And eth0 is the default interface. Now I start downloading something on the eth0 interface. In midst of the download if I change the default interface to wlan0, the download breaks. I don’t understand why this happens. Can you please tell me ??

    • Abhay, that is because you only have one default route setup. During your download, the application probably got temporarily disconnected and tried to resume the download after the default gateway switched.

  105. I love you man!

  106. Hi Darien,

    Awesome post. I have followed your post, and definitely learnt more about routing in linux. Thanks.

    Now to my problem. I have a web server with two NIC’s:

    eth0 is used as the local network nic: 192.168.250.43/24 (no gateway)
    eth1 is used as the public network nic: 41.222.8.146/29 (gateway 41.222.8.145)

    eth1 is the default route, and the only device with a specified gateway.

    If I ping from an external address, I get replies on the external/public IP for roughly 5 min, and then the time-outs start, which continue indefinitely. If, however, I ping the public IP’s primary DNS, on the webserver itself, I start getting replies again on the external PC I am polling connectivity on.

    Is this a routing issue on my webserver, or a routing issue on my router, or is it a non-routing issue.

    Thanks again for the great post.

    • While this isn’t a multi-default route issue, It sounds like you have some sort stateful packet inspection (SPI) firewall between your 2 NIC system and the external system you are pinging from. The firewall could be built into the 2 NIC system — not 100% sure. Regardless, this issue is quite common in those types of setups. Specifically the outbound traffic from the 2 NIC system through the SPI firewall triggers the firewall to allow reply traffic for a set period of time (a max timeout). Therefore, you’ll need to check the systems for corresponding firewalls. Disabling them altogether should alleviate this problem.

      Regards,

  107. Thank you very much. I was having this issue since yesterday. Your email just save my day and my job too. You are a star.

  108. Many thanks to You !!!!!!

    I spent days trying to solve this. I even gave up on it but had to start again several times. This worked right away and finally I can move up with my network upgrade.

  109. Exactly what I was looking for, this worked perfectly.

  110. Is not that possible that we can add a static route on one ethernet and add a default gateway on one ethernet ?

    • Yes, that is possible, but I’m not sure that is what you mean.

      Regards,

  111. is not that possible that we can add a static route on one ethernet and keep another ethernet on default gateway

  112. Wow 200 posts (this is 201 for those keeping count) and 5 1/2 years later, amazing article…

    2 things.. first let me quote you… “this information is designed primarily to preserve my sanity”, hows that working out for you?

    And second, Im using Ubuntu 12.04.
    > netstat -anr

    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt Iface
    0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 control
    0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 wlan
    192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 control
    192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 wlan

    the was after adding “route add default gw 192.168.1.1″ to /etc/rc.local
    the second line “0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 wlan” should not be there, ever. But when ubuntu boots up, the wlan device is the once to get the lease first. How can I set the default gateway in this multi-nic system… At the same time, the machine needs to talk to everything on the 192.168.2.X network as well.

    Thanks, and again… wow.. really impressive article, I read every post, you are a trove of information.

    • Never mind. I remembered something you said in one of the comments, tried it, and it works…

      I’m not saying what or what comment, because people should really read all of it before posting, all the information is there.. really amazing.

  113. i did as per ur article but the problem is not solved
    eth0 = 10.0.0.2
    eth1= 192.168.5.102
    default gateway is 192.168.5.1
    gateway for eth0 is 10.0.0.138
    when i am pinging it tries to go via 192. which i dont want it should go thru 10…
    so i did
    ip route add 10.0.0.0/24 dev eth0 src 10.0.0.2 table admin
    ip route add default via 10.0.0.138 dev eth0 table admin

    ============================

    ip rule add from 10.0.0.2/32 table admin

    ip rule add to 10.0.0.2/32 table admin

    ==================================

    ip rule show
    0: from all lookup 255
    32764: from all to 10.0.0.2 lookup admin
    32765: from 10.0.0.2 lookup admin
    32766: from all lookup main
    32767: from all lookup default
    ============================

    but still

    ping smtp.bizmail.yahoo.com
    PING smtp.bizmail.yahoo.com (98.139.221.126) 56(84) bytes of data.
    From 192.168.5.102 icmp_seq=1 Destination Host Unreachable
    From 192.168.5.102 icmp_seq=2 Destination Host Unreachable
    From 192.168.5.102 icmp_seq=3 Destination Host Unreachable

    it is still going thru 192…

    • Hi Ahmed,

      Okay, then switch your configuration where eth0 is set by default and use table “admin” to define your eth1 settings.

      Regards,

      • i did the opposite

        ip rule show
        0: from all lookup 255
        32764: from all to 192.168.5.102 lookup admin
        32765: from 192.168.5.102 lookup admin
        32766: from all lookup main
        32767: from all lookup default
        ==============================================
        still the problem exist
        when i am pinging to outside it is try to go via 192.168.. , i want it to go via eth0 10.0.0. … bcause
        dont know y its not working for me

        • Okay, I see another issue. Your first rule has:

          0: from all lookup 255

          And that should be instead:

          0: from all lookup local

          Perhaps “255” = “local” on your system, but I don’t know that for sure.

          Regards, Darien

  114. Hi, This is a great post. thanks again,.
    One question – How do you print the routing table the one you created as “admin” in the above example? how do I see my connection to the particular IP uses the correct gateway?

    • “ip route show table main” and “ip route show table default”

      Regards,

  115. Hi all,

    I am trying to create the following set up

    I have following linux networking set up.

    Machine A Ip-10.8.11.100 (Source machine)
    Machine B ip-10.8.11.1 (acting as a gateway for 10.5.0.0 network )
    Machine C ip-10.8.11.2 (acting as a gateway for 192.68.11.0 network )
    Machine D ip-10.5.0.1 (Dest machine)
    Machine E ip-192.168.11.1 (Dest machine)
    Machine F ip-10.8.11.5 (default agteway)

    Now I want to add both the destination networks into Machine A so that I should able to ping ip address of both the networks from it.

    The rules and routes that I ave added on machine A a below :-

    $ip rule add from 10.8.11.100 table 2
    $ip route add 10.5.0.0/24 via 10.8.11.1 dev eth0 src 10.8.11.100 table 2
    $ip route add 192.168.11.1 via 10.8.11.2 dev eth0 src 10.8.11.100 table 2
    $ip route add default via 10.8.11.5 table 2
    $ip route flush cache

    Here the rules on the Machine A

    $ip rule show
    0: from all lookup local
    32763: from 10.8.11.100 lookup 2
    32766: from all lookup main
    32767: from all lookup default

    Here are the routes in table 2 of machine A

    $ip route show table 2
    192.168.11.0/24 via 10.8.11.2 dev eth0 src 10.8.11.100
    10.5.0.0/24 via 10.8.11.1 dev eth0 src 10.8.11.100
    default via 10.8.11.5 dev eth0

    Now what is happening is when I trying to ping any destination from Machine A, I am not getting any ping reply from any of the network and I think the packet is also not reaching to destination machine as I am not seeing anything in the tcpdump traces at the destination.(Machine D & Machine E).

    Now when I am adding the same rules to the main table of linux everything is working fine . I can able to ping all the static as well as default networks from machine A.

    I don’t know whts was happening when I am adding the same routes to table 2.

    A part from that I am directly using the table index 2 without adding its entry in /etc/iproute2/rt_tables file.
    Is it necessary to add the table name into that file.?

    I did that as well but looks like it is not working out.
    How can we know which routing table currently linux is using.?

    Please help me here

  116. Thanks alot, worked for me!

  117. Finally found the webpage that sorted my multi nic networking issues. Excellent post. Followed it, and it worked a treat.

  118. I am having trouble getting broadcast message to their respective networks/interface. All broadcast messages to 255.255.255.255 go out the default route in the main table even if I set up default routes in other tables and create the rules to use those routes.

    Should the advanced routing features handle broadcasting or is the kernel/library somehow changing the broadcast message to the default route address before getting to the routing library?

  119. Wow! Such a great post! It still comes up at the top although it was written so long ago!
    I don’t know if you keep up the comments but I am really hoping that you do.
    I have a question for you… I am trying to use your guide with multiple wifi connections. Is that any different than wired ones? I’m thinking that it shouldn’t right?
    For example I want all peer 2 peer traffic to go through wlan1. I’ve selected the appropriate option in ktorrent for limiting all traffic to wlan1 and created a p2p rule table following your guide. Before reading your post (and realizing that connections still go out from the default route) when I was limiting ktorrent to wlan1, I would see no traffic…0.
    Now I created a new table rule like you are doing with admin, however when I do ‘netstat -n | grep LISTENING’ established connections are still bound to the primary ip address (wlan0) and not wlan1. Is there a reason why those new connections are not listening to the additional ip address? The weird thing is that now I do see traffic in ktorrent, albeit slow, but I do see some. But even regardless of peer to peer traffic, not even 1 connection is listed as LISTENING for the secondary wlan1 ip? All connecitons are listening on the primary (wlan0) ip. What do you make of that?

    • Hi,

      Yes, policy routing works on any type of network interface — wired or wireless.

      It sounds like some sort of ‘ktorrent’ problem, then. Specifically, if you configured ktorrent to only use wlan1, but it _still_ uses wlan0, then it sounds like a but with ktorrent, regardless of policy routing settings.

      I’d recommend reaching out to the ktorrent support mailing lists and find out if this is a known or unknown issue. If you completely turn off policy routing _and_ ktorrent _still_ uses wlan0 (even when you explicitly set wlan1), then completely disable wlan0 and see if ktorrent still works properly. If not, then it sounds like ktorrent has a bug where it always uses the first registered interface of the system.

      Regards,

      • Thank you for the reply,
        so you were right, it was a bug with ktorrent. I managed to make it work with Vuse though.
        Also I made it work with the following two commands:

        ip route add table 2 default via 192.168.1.1
        ip rule add from 192.168.1.0/24 table 2

        here’s my rules table:
        0: from all lookup local
        32765: from 192.168.1.0/24 lookup 2
        32766: from all lookup main
        32767: from all lookup default

        going over your post again I noticed that 32764 is missing. I don’t mean to sound stupid but would you mind explaining a bit more why do we need 32764 *and* 32765?

        thank you!

        PS: It’s such a shame that you stopped blogging! Got too busy? You have some really nice posts, and judging by the number of comments pretty good audience.

        • > so you were right, it was a bug with ktorrent. I managed to make it work with Vuse though.

          I figured. ;)

          > Also I made it work with the following two commands: > > ip route add table 2 default via 192.168.1.1 > ip rule add from 192.168.1.0/24 table 2 > > here’s my rules table: > 0: from all lookup local > 32765: from 192.168.1.0/24 lookup 2 > 32766: from all lookup main > 32767: from all lookup default > > going over your post again I noticed that 32764 is missing. I don’t mean to sound stupid but would you mind explaining a bit more why do we need 32764 *and* 32765?

          Sure. Without 32764, any traffic from the internet going to 192.168.1.0/24would get directly routed your default interface, instead of the secondary interface (e.g., 192.168.1.1). This only matters if your secondary interface is connected to some sort of upstream network translation device (NAT) that is able to route your RFC 1819 addressable interface out to the internet. If that’s the case (most likely), then you’ll find that any internet-based traffic going to IP 192.168.1.1 will likely go through your primary interface, instead of your secondary interface.

          You can run tcpdump on all interfaces to confirm this, by issuing a simple ping out to the internet, where you specify that it should use the secondary interface for all outbound traffic.

          > PS: It’s such a shame that you stopped blogging! Got too busy? You have some really nice posts, and judging by the number of comments pretty good audience.

          Thanks. I don’t have much time to blog publicly, currently. As such, I tend to try and stick to topics that aren’t already discussed on the internet, elsewhere. Therefore, if I come across an issue that isn’t easily found via standard Google searches, I’ll likely create a future blog post about it. ;)

          Regards,

          • Ok so, when I do traceroute -i wlan0 and tracertoute -i wlan1 I do see traffic going out from the appropriate routes

            So rule 32764 forwards any traffic FROM the internet (meaning coming in from other interfaces?) TO the secondary interface gateway? Because rule 32765 reads “from 192.168.1.0/24 lookup 2″ so I am assuming it is taking care of traffic coming in the secondary interface making sure that it goes out from 192.168.1.1 right?

            And if so, Vuse is working fine without rule 32764, because all traffic is generated directly from the secondary interface and also arriving into the interface? Is that why so far everything seems to be working just with rule 32765 ?

            THAT’s what confuses me!

            thank you,

            • Darien
            • Posted November 6, 2012 at 7:43 pm
            • Permalink

            >So rule 32764 forwards any traffic FROM the internet (meaning coming in from other interfaces?) TO the secondary interface gateway?

            Correct.

            > Because rule 32765 reads “from 192.168.1.0/24 lookup 2″ so I am assuming it is taking care of traffic coming in the secondary interface making sure that it goes out from 192.168.1.1 right?

            Correct.

            > And if so, Vuse is working fine without rule 32764, because all traffic is generated directly from the secondary interface and also arriving into the interface? Is that why so far everything seems to be working just with rule 32765 ?

            Pretty much, yes.

            > THAT’s what confuses me!

            Yup, policy routing is not straight forward, at all.

            Regards, Darien

          • The conversation reached the max limit depth so I’ll reply here..
            Thanks again for the last clarification, I got it now!
            Oh and please continue blogging good stuff!

      • Hi again Darien,
        sorry to bother you again, I was just going through the comments section in this page checking to see how others might be using policy routing for any smart scenarios.
        In the last comment made by user “anestis” where he used the -I parameter to bind to ping to an interface, I tried it too. I’m having the same issue as well. Binding ping works only if I bind it to the IP of the secondary interface (i.e. -I 192.168.1.12). If I bind it directly to the interface (i.e. -I wlan1) it’s giving me destination host unreachable. With the primary interface, it works both ways (either with -I ip or -I wlan0). Maybe -I is not a reliable way to test?

        Thanks again,

        • Yeah, I think you’re right. It seems that option is not a reliable way to test policy routing.

          Regards, Darien

          Regards,

          • When I get the change I’ll test it with centos or RHEL too. Maybe it’s a debian based distro issue. Thanks again for the reply back.

  120. Hi Daren, awesome post!! Googling paid off indeed!
    Let me ask you a question though,
    I’m having an issue and I don’t know if it’s my configuration

    # traceroute -i eth0 8.8.8.8 (goes out eth0 gw)
    # traceroute -i eth1 8.8.8.8 (goes out eth1 gw)
    # ping -I eth0 8.8.8.8 (works ok)
    # ping -I eth1 8.8.8.8 Destination Host Unreachable)
    # ping -I 192.168.2.2 (eth1 ip) (works ok)

    So for some reason if I use the direct ip of eth1 ping works but if I use the interface, I get Destination Host Unreachable.
    Any ideas as to what might be the problem here?

    • last line I meant # ping -I 192.168.2.2 8.8.8.8
      oh and sorry for butchering your name… I meant Darien :)

    • Please provide full network and routing information for each interface, as I don’t have enough information in order to identify the core problem.

      Regards, Darien

      On Nov 8, 2012, at 3:16 AM, “Darien Kindlund’s Blog”

      • Oh I’m sorry I should have added some more details.
        So here’s my setup:

        eth0: 192.168.1.2 gw0:192.168.1.1
        eth1: 192.168.2.2 gw1:192.168.2.1

        /etc/iproute2/rt_tables
        #added:
        1 table1
        2 table2

        ##route1
        ip route add default via 192.168.1.1 dev eth0 table table1
        ip rule add from 192.168.1.0/24 table1

        ##route2
        ip route add default via 192.168.2.1 dev eth1 table table2
        ip rule add from 192.168.2.0/24 table2

        so, that’s what I did pretty much and then tried to test with traceroute -i and ping -I
        Traceroute seems to work in both instances (going out from the correct gateways).
        Ping worked for my default interface (eth0), however for eth1 it only works if I put the actual IP but not with the eth1
        ie.
        ping -I 192.168.2.2 8.8.8.8 (works)
        ping -I eth1 8.8.8.8 destination host unreachable

        Isn’t that weird? Do you thing I misconfigured something? Let me know if you need any more details.

        Thank you,

        • After issuing the “ip rule” commands, did you issue “ip route flush cache” at the end, so that your commands will take immediate effect?

          Regards, Darien

          On Nov 9, 2012, at 12:19 AM, “Darien Kindlund’s Blog”

            • anestis
            • Posted November 11, 2012 at 8:38 pm
            • Permalink

            Sorry for the long delay but I had to wait to get access to that system.
            So, I did the simple setup from scratch with the two routes and I tried flushing the cache like you suggested but same thing. Any thoughts?

            • Darien
            • Posted November 13, 2012 at 11:08 pm
            • Permalink

            It looks like using the -I option with ping is not a reliable way to test policy routing; see sxty1’s earlier comment for details.

            Regards, Darien

            Regards,

  121. For the 2 local networks 192.168.7.0 and 10.10.70.0 in your example, you defined 2 default gateways for them nice enough to access the Internet in 2 distinct paths. That is very good but I need more than that. I also desire the 2 local networks be able to ping each other meanwhile. What’s the trick? Please teach me.

    • I assume you want _all_ the systems on both networks to be able to ping each other — not just the two interfaces on your Linux system. In that case, you need to look at standard routing guides, which clearly explain this. You don’t need policy routing, specifically.

      Here are some useful references on general routing for Linux: http://www.google.com/url?sa=t&source=web&cd=3&ved=0CCAQFjAC&url=http%3A%2F%2Flinux-ip.net%2Fhtml%2Frouting-intro.html&ei=UziqUOTPFqnB0QGm7YG4DQ&usg=AFQjCNG8qeJPOA12XiW69xbSt0WKg7uArQ

      http://www.google.com/url?sa=t&source=web&cd=2&ved=0CBUQFjAB&url=http%3A%2F%2Flartc.org%2Fhowto%2F&ei=UziqUOTPFqnB0QGm7YG4DQ&usg=AFQjCNGcAW7fcd573mwGwbp6fk3ZjXZ5kg

      Regards,

      • Actually I have setup a Ubuntu linux gateway with 2 interfaces. For instance

        auto eth1
        iface eth1 inet static
        address 192.168.1.20
        netmask 255.255.255.0
        network 192.168.1.0
        broadcast 192.168.1.255

        auto eth2
        iface eth2 inet static
        address 10.10.10.20
        netmask 255.255.255.0
        network 10.10.10.0
        broadcast 10.10.10.255

        I have enabled the IP forwarding such that PC in 192.168.1.0/24 and PC in 10.10.10.0/24 could ping each other though this machine as their default gateway.

        TWO other routers, A and B, have independing Internet connections respectively. Each one has an LAN interface directly connected to the 2 LANs. The LAN port IP of router A is 192.168.1.19. The LAN port IP for router B is 10.10.10.19. I desire each network go through a specified router for Internet access. For instance; PC in 192.168.1.0/24 go through router A. PC in 10.10.10.0/24 go through router B. To do this, I have added 2 tables to the file: /etc/iproute2/rt_tables
        100 pma
        200 ma

        Then I issued the following commands:

        ip rule add from 192.168.1.0/24 table pma
        ip rule add from 10.10.10.0/24 table ma
        ip route add default via 192.168.1.19 table pma
        ip route add default via 10.10.10.19 table ma
        ip route flush cache

        Now that the 2 networks could have the Internet access independently. But they cannot ping each other anymore. If I reboot this gateway without issuing the previous 5 commands. PC from both networks could each other again. And none of the PC could get Internet access anymore. Of course I could add a gateway statement to the network configuration file, for instance, the last line of the following stance.
        auto eth1
        iface eth1 inet static
        address 192.168.1.20
        netmask 255.255.255.0
        network 192.168.1.0
        broadcast 192.168.1.255
        gateway 192.168.1.19

        But that is not I want since it would become a default gateway to all connected networks. I want each connected network go through different path for Internet access, namely router A and router B. In fact, I wish I could dynamically assigned some selected PC from 10.10.10.0/24 go through route A in additional to the whole 192.168.1.0/24 network.

        This is what I mean I want “more” than that. Sorry for my fault of ambiguous illustration previously. Again I appreciate your document. And I am expecting further teaching from you. Thank you in advance.

        • Hi,

          I see what you mean. I appreciate your detailed description, as that really helps me better understand what the possible issues are. First, I agree with you that you do _not_ want to include the gateway directive in your individual network configuration files.

          Basically, you want: 1) policy routing from this “router” system out to the internet, so that each upstream link is used independently -AND- 2) proper routing between other systems on networks A and B through this router, so that they don’t have to go out to the internet to reach each other

          Assuming your initial IP configuration for eth1 and eth2, as well as the default gateways defined for each interface, you indicated your tables are setup as such:

          /etc/iproute2/rt_tables 100 pma 200 ma

          Issue the following commands:

          ip rule add from 192.168.1.0/24 table pma ip rule add to 192.168.1.0/24 table pma

          ip rule add from 10.10.10.0/24 table ma ip rule add to 10.10.10.0/24 table ma

          ip route add 192.168.1.0/24 dev eth0 src 192.168.1.20 table pma ip route add default via 192.168.1.19 dev eth0 table pma

          ip route add 10.10.10.0/24 dev eth1 src 10.10.10.20 table ma ip route add default via 10.10.10.19 dev eth1 table ma

          ip route flush cache

          Regards,

            • Shawn
            • Posted November 26, 2012 at 10:47 pm
            • Permalink

            Thanks for your teaching Darien. For the following suggested statements:
            ip rule add from 192.168.1.0/24 table pma ip rule add to 192.168.1.0/24 table pma
            ip rule add from 10.10.10.0/24 table ma ip rule add to 10.10.10.0/24 table ma
            Could you explain the functions? Again I would like PC in these 2 networks 192.168.1.0/24 and 10.10.10.0/24 could ping each other through this Linux gateway. How the 2 statements could fullfill this requirement?

            Suppose I would like some selected nodes (e.g. 10.10.10.100 is the PC of the General Manager and requires a preference connection to the Internet) to reach Internet by going through eth0 with table pma instead of the deault path of the subnet the node belongs to. What statement shoud be added? I understand that there is a PRIORITY paramenter setting for ip rule statements. How could I make use of it?

            How to use statement: “ip route add” and “ip route append” What’s the difference? Could you provide some examples to illustrate the difference?

            That’s a lot of questions but I’ve found your teaching the most comprehensive to me and will help in a more effiective way to complete my study on policy routing. Thank you very much Darien. Good teaching.

            • Darien
            • Posted November 26, 2012 at 10:55 pm
            • Permalink

            Hi Shawn,

            Sorry about the confusion. It looks like wordpress combined multiple lines into a single statement. Here was my original recommendation, but this time, I’ve added multiple lines between each command, so that it’s clearer to see when one command ends and another begins:

            Assuming your initial IP configuration for eth1 and eth2, as well as the default gateways defined for each interface, you indicated your tables are setup as such:

            /etc/iproute2/rt_tables

            100 pma

            200 ma

            Issue the following commands:

            ip rule add from 192.168.1.0/24 table pma

            ip rule add to 192.168.1.0/24 table pma

            ip rule add from 10.10.10.0/24 table ma

            ip rule add to 10.10.10.0/24 table ma

            ip route add 192.168.1.0/24 dev eth0 src 192.168.1.20 table pma

            ip route add default via 192.168.1.19 dev eth0 table pma

            ip route add 10.10.10.0/24 dev eth1 src 10.10.10.20 table ma

            ip route add default via 10.10.10.19 dev eth1 table ma

            ip route flush cache

            Hopefully, the extra spaces will help. I will reply to your other questions separately.

            Regards,

            • Darien
            • Posted November 26, 2012 at 11:04 pm
            • Permalink

            > Could you explain the functions?

            “ip rule add from …” and “ip rule add to …” directives indicate when Linux should use the corresponding routing table (e.g., “pma” or “ma”) when processing incoming packets. Only packets matching these “rules” will get processed by the corresponding table.

            “ip route add …” directives indicate _what_ Linux should do with the packets that matched on the corresponding “ip rule …” directives for a given routing table.

            > Again I would like PC in these 2 networks 192.168.1.0/24 and 10.10.10.0/24 could ping each other through this Linux gateway. How the 2 statements could fullfill this requirement?

            The “ip rule …” and “ip route …” statements help the Linux gateway route packets through the right interfaces. I think your confusion was due to the fact that wordpress combined the “ip rule” and “ip route” statements onto a single line, which is incorrect. Let me know of this is still unclear.

            > How to use statement: “ip route add” and “ip route append” What’s the difference?

            Use “ip route add” when you want to add a new routing entry to the TOP of the corresponding routing table.

            Use “ip route append” when you want to add a new routing entry to the BOTTOM of the corresponding routing table.

            I believe, Linux processes entries in a routing table from TOP to BOTTOM; first match wins.

            Regards,

            • Shawn
            • Posted November 27, 2012 at 1:34 am
            • Permalink

            How about this: Suppose I would like some selected nodes (e.g. 10.10.10.100 is the PC of the General Manager and requires a preference connection to the Internet) to reach Internet by going through eth0 with table pma instead of the deault path of the subnet the node belongs to. What statement shoud be added? I understand that there is a PRIORITY paramenter setting for ip rule statements. How could I make use of it?

            • Darien
            • Posted December 11, 2012 at 9:37 pm
            • Permalink

            Defining route priorities is documented here: http://linux.die.net/man/8/ip

            Regards,

  122. This is the closest thing to an answer I have! Thank you
    I have a question. What if i Don’t always know the network IP of the second interface?
    EG
    192.168.0.10 Eth0 Gateway 192.168.0.1 This will always be like this.
    but then i have a VPN connection where the network IP might change, but i ALWAYS know it will be on link PPP0
    Is there a way to route based soley on the interface or link ?

    • You would have to setup some sort of script to run the policy route commands _after_ the VPN is established. The details of this depend on which VPN software you’re using and also which Linux distribution you’re running.

      Regards,

  123. 2 snaps up for this thread

  124. Hello, Darien

    I’m working on a selective VPN routing on tomato router which is really useful here in China.
    Using routing table, getting custom list of host and net to route through VPN is working perfectly with one big problem, I will have to manually add every single IPs for all the site that I found is block to the script and it is very time consuming. I am in the searching for some kind of script or technique to make this process somewhat automated.
    (I am not routing all connections through VPN since that it would make everything slow.)

    >>Here is the part of the script I’m using
    VPNGW=$(nvram get pptp_client_gateway)
    ##### begin batch route #####
    # DNS
    route add -host 8.8.8.8 gw $VPNGW
    route add -host 8.8.4.4 gw $VPNGW
    route add -host 209.244.0.3 gw $VPNGW
    route add -host 74.207.242.213 gw $VPNGW
    # twitter.com
    route add -net 37.61.54.0/24 gw $VPNGW
    route add -net 120.88.53.0/24 gw $VPNGW
    route add -net 184.169.84.0/24 gw $VPNGW
    route add -net 199.16.156.0/22 gw $VPNGW
    route add -net 199.59.148.0/22 gw $VPNGW
    route add -net 168.143.161.0/24 gw $VPNGW
    # fbcdn
    route add -net 125.252.224.0/24 gw $VPNGW
    route add -net 60.254.185.0/24 gw $VPNGW
    route add -net 96.17.69.0/24 gw $VPNGW
    route add -net 60.254.175.0/24 gw $VPNGW

    >>This is part of my #ip route show
    50.112.0.0/16 via 10.9.0.1 dev ppp40
    174.129.0.0/16 via 10.9.0.1 dev ppp40
    54.242.0.0/15 via 10.9.0.1 dev ppp40
    23.22.0.0/15 via 10.9.0.1 dev ppp40
    15.240.0.0/15 via 10.9.0.1 dev ppp40
    54.234.0.0/15 via 10.9.0.1 dev ppp40
    54.236.0.0/15 via 10.9.0.1 dev ppp40
    50.16.0.0/14 via 10.9.0.1 dev ppp40
    10.9.0.1 dev ppp40 proto kernel scope link src 10.9.28.24
    XXX.XXX.XXX.XXX via 192.168.1.1 dev vlan3 #XXX is my vpn server
    10.0.0.0/24 dev br0 proto kernel scope link src 10.0.0.1
    192.168.1.0/24 dev vlan3 proto kernel scope link src 192.168.1.198
    127.0.0.0/8 dev lo scope link
    default via 192.168.1.1 dev vlan3

    I have spent about half a month on this already, trying to find some way to automate this.
    Please give some advice or help if you have time,
    Noch

    • I’m thinking about something close to Automatic Failover, if default route is not responsive then if a request is repeated more than 2 times, try the next alternate gateway in order. (not switching completely from default gateway, but only use alternate gateway with some hosts that is blocked on default route)

      ## Sorry for my English language, I am Thai :)

  125. Thank you. I’m curious about a mix of IPv4 and IPv6. With an IPv4 address on eth0 and an IPv6 address on eth0, plus an IPv4 address on eth1, what should one setup? Our eth1 interface is generally to a private network, but eth0 goes to both IPv4 and IPv6 default gateways. Thanks! Alan

  126. This is a great tutorial! But how do I get it to work for 8 NIC’s?

    • Simply define unique routes for each interface. As long as there is no overlap, you should be all set.

      Regards,

      • Thanks for the amazingly fast reply! I’ve got another quick question, I’ve noticed that you use static IP’s a lot, is there any way to get this working with DHCP?

        • It’s possible, but you would have to customize your DHCP scripts to issue the appropriate policy routes once successful.

          Also, you would have to make sure your DHCP servers issue leases on non-overlapping subnets.

          Lastly, your DHCP servers need to be on separate, logical networks, so that way they don’t see overlapping requests from clients.

          Regards,

            • dev
            • Posted February 9, 2013 at 7:09 pm
            • Permalink

            I’m sorry if my question seems silly (I’m a novice) but would this work if both interfaces were on the same gateway?

            • Darien
            • Posted February 11, 2013 at 9:19 am
            • Permalink

            As long as the DHCP server assigns a fixed IP address for each interface, then yes.

            Regards,

            • dev
            • Posted February 9, 2013 at 10:02 pm
            • Permalink

            Hello Darien,

            Well, I’ve tried your setup and it works great when I want to send data from the client to the server. However, the server (with 8 NIC’s) can’t seem to ping the client…would you happen to know whats wrong?

            • Darien
            • Posted February 11, 2013 at 10:36 am
            • Permalink

            I would need a lot more information about your setup, in order to troubleshoot.

            Regards,

  127. After struggling most of the day, this post came to the rescue. I use Ubuntu 12.04 LTS. Thanks Darien!!!

  128. 5 years later and people still sending thanking you notes. And the list is soo long…!

    Here’s yet one more: THANK YOU! It saved my day – or rather night.

    I have s set-up with a couple of multi-homed hosts belonging to two networks. Each network is connected to the Internet via a separate NAT router. For various reasons I want outbound traffic using one of the routers, but inbound (port forwarded ssh) traffic from the other. After much grief I realised that for inbound traffic to work, it has to go back to the same router as it came from (i.e. exactly the case described here). Not sure why, as I always thought both source and destination were translated: in which case the inside sshd should believe stream originate from the router and send responses back to it, but obviously this is not so. This really is not my area and a few lines of enlightenment would be appreciated (your band-width permitting of course).

    • Yeah, sshd doesn’t perform any packet routing by itself — that’s the kernel’s job. Therefore, all packets going in and out of your system are handled through the standard routing tables, which is why you saw this type of activity. The persistence you’re describing is typical of any sort of “stateful packet inspection” (SPI) feature on a firewall or NAT device. Specifically, once an initial TCP connection is established and SPI is enabled, the firewall/NAT will maintain a record of the mapping and route all packets pertaining to that flow through to the mapped destination, by default. It’s the same reason why you may need to send initial packets out from the trusted side through a firewall to the untrusted side — in order to receive equivalent packets back through the firewall (from untrusted to trusted side).

      Regarding your “stateful route” issue with sshd… yeah, even though you think it would make sense to route outbound SSH packets through a completely different NAT’d route out, the problem with that approach, is that the two NAT devices have _no_ complete view of the TCP flow state, which can cause problems when managing the full TCP flow. In order for NAT to work correctly, a single NAT device needs to see both sides of the flow; otherwise, you will run into the issues you’ve described. The other aspect to this, is that if you tried to configure asymmetric routing through multiple NAT devices for a single ssh session, the original SSH client would have no idea why they’re receiving TCP packets from a completely different IP address than the original IP address that the client sent TCP packets to.

      In order to discover these issues, it’s best recommended to run tcpdump or wireshark on both the server and client sides, in order to witness this asymmetric route issue while you try running TCP connection tests — the problem will become much clearer when you have this type of visual as a reference point.

      Regards,

  129. Dear Sir,

    I need the information regarding multiple interface card.

    My Scenario is :

    Eth0: 172.X.X.106 / 255.255.255.252 GW 172.X.X.105
    Eth1: 192.168.1.222 / 255.255.255.0 GW 192.168.1.1

    My Requirement is that:

    1) Eth0 traffic out/in gateway 172.X.X.105
    2) Eth1 traffic in/out gateway 192.168.1.1

    Mean that traffic generate from eth0 forward to its own gate way, same as eth1.

    I am using the elastix server (centos 5.7) with two sip account, one on eth0 and other eth1. Port number 5060.

    I am confused how to do this.

    Waiting for your quick response.

    I try , which is mention on above, no success.

    Regards

    Shahid Iqbal

    • Hi Shahid,

      So, let’s assume this is your network:

      eth0 – 172.16.0.106 netmask 255.255.255.252 (/30) eth0’s gateway is: 172.16.0.105

      eth1 – 192.168.1.222 netmask 255.255.255.0 (/24) eth1’s gateway is: 192.168.1.1

      Here are the corresponding commands:

      printf 1\tuplink0\n >> /etc/iproute2/rt_tables printf 2\tuplink1\n >> /etc/iproute2/rt_tables

      ip route add 172.16.0.106/30 dev eth0 src 172.16.0.106 table uplink0 ip route add default via 172.16.0.105 dev eth0 table uplink0 ip rule add from 172.16.0.106/30 table uplink0 ip rule add to 172.16.0.106/30 table uplink0

      ip route add 192.168.1.0/24 dev eth1 src 192.168.1.222 table uplink1 ip route add default via 192.168.1.1 dev eth1 table uplink1 ip rule add from 192.168.1.0/24 table uplink1 ip rule add to 192.168.1.0/24 table uplink1

      ip route flush

      Regards,

      • I am using elastix server, i put all above mention, which was sent to me, but nothing happen.
        Basically using 5060 port to forward traffic to sip ip. but sip registration is failed.(eth0)
        I dont know how to manage.

  130. I have a Linux box wtih 4 interfaces: eth0 and eth1 serve as default gateways of network A and network B respectively. eth2 and eth3 serve as gateways to the Internet through 2 different ADSL. I want the incoming traffic from network A to go through eth2 to reach Internet, the incoming traffic of network B to go through eth3 to reach Internet. Meanwhile network A and network B could communicate with each other through their default gateways, namely eth0 and eth1 respectively. Forwarding is enabled in this box. I have added 2 entries to the rt_tables. I know that I have to add some ip rules and ip routes statements. But how could I manage to have 2 different default gateways to route Internet traffic for 2 different networks while enabling the inter-network communication between this 2 networks. Please advise.

    • Please provide example network subnets for: 1) network A 2) network B 3) eth2 ADSL network 4) eth3 ADSL network

      Regards,

    • Configuration of the Linux Box…

      eth0: 192.168.1.20
      eth1: 10.10.10.20
      eth2: 192.168.6.20 (reach Internet through 192.168.6.1)
      eth3: 192.168.0.20 (reach Internet through 192.168.0.1)
      LAN A: 192.168.1.0/24 Default Gateway: 192.168.1.20
      LAN B: 10.10.10.0/24 Default Gateway: 10.10.10.20

      If you copy the following ASCII text and paste it to the Notepad, you could see a better picture.

      ________
      +—————+ /
      | ISP 1 | /
      +——–+ (MA) +——+
      +—————-+ | | gw 192.168.0.1| /
      | LAN A | +———–+–+ +—————+ /
      | 192.168.1.0/24 +—-+ eth0 eth3 | /
      +—————-+ | | |
      | Linux Box | |
      +—————-+ | | | Internet cloud
      | LAN B +—-+ eth1 eth2 | |
      | 10.10.10.0/24 | +———–+–+ +—————+ |
      +—————-+ | | ISP 2 | \
      +——–+ (PMA) +—-+
      |gw 192.168.6.1 | \
      +—————+ \
      \
      \________

      If I issue the following commands, it seems that traffic between LAN A and LAN B is not possible as they all go their own gateway as listed in the routing tables, ma and pma respectively. The traffic could eventually go to the Internet. I desire LAN A and LAN B could communicate with each other. Meanwhile LAN A and LAN B could go to the Internet through the assigned gateways.

      ip rule add from 192.168.1.0/24 table ma
      ip rule add to 192.168.1.0/24 table ma
      ip rule add from 10.10.10.0/24 table pma
      ip rule add to 10.10.10.0/24 table pma

      I understand that I may need to additionally issue the following comands:

      ip route add 192.168.1.0/24 dev eth0 table ma
      ip route add default via 192.168.0.1 dev eth0 table ma
      ip route add 10.10.10.0/24 dev eth1 table pma
      ip route add default via 192.168.6.1 dev eth1 table pma

      Still don’t work. Is there a precedence order for “ip rule” and “ip route”? If so, which one goes first? Getting confused…

      Appreciate you advise in advance.

      • Oops! It seems that WordPress doesn’t like the graphic in ASCII text. Sorry. I hope you understand anyway.

  131. Hi Darien,

    Once again, great post! I always come back to this post about linux pbr :-)
    I have a question for you.. when I connect to a vpn my default route is overwritten and all my internet traffic goes through the vpn. So first question, is this normal behaviour? Why does the vpn client feels the need to route ALL traffic through the vpn and not just the traffic destined to the vpn’s subnet.
    Second question, what is the best way to prevent this behaviour and route only the vpn’s subnet traffic through the vpn?

    Here’s what I did but had no success:

    –before vpn:
    $ ip r
    default via 192.168.1.1 dev wlan0 proto static
    192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.3 metric 2

    –after vpn:
    $ ip r
    default dev ppp0 proto static
    10.72.27.161 dev ppp0 proto kernel scope link src 10.72.27.166 <—-(*)

    so my obvious thought was to delete the default route for the ppp0 and recreate the old defaut for the wlan0. This worked for the internet traffic however, traffic destined for the 10.72.27.0/24 (vpn subnet) is still routed via the default route (wlan0) EVEN though the line that I have pointed out with the (*) above STILL exists in the routing table. Shouldn't that line be routing traffic through the vpn even though the default route points to something else?

    What am I missing?

    thank you!

  132. Hmm it seems like your website ate my first comment (it was extremely long) so I guess I’ll just sum it up what I wrote and say, I’m
    thoroughly enjoying your blog. I too am an aspiring blog writer but I’m still new to everything. Do you have any tips for rookie blog writers? I’d certainly appreciate it.

  133. You are so cool! I don’t think I have read something like this before. So good to discover another person with some original thoughts on this subject matter. Seriously.. thank you for starting this up. This web site is one thing that is required on the internet, someone with a little originality!

  134. Hi Darien. Your article is great but didn’t help in my problem.

    I have 2 machines directly connected to each other (im doing this because i have to test one protocol in a controlled environment). Both machines (A and B) have two NICS [etho (A) eth0 (B) and eth1(A) eth1(B)].
    A:
    eth0 address: 10.1.1.1/24 gateway: 10.1.1.2
    eth1 address: 10.1.2.1/24 gateway: 10.1.2.2
    B:
    eth0 address: 10.1.1.2/24 gateway: 10.1.1.1
    eth1 address: 10.1.2.2/24 gateway: 10.1.2.1

    I have configured two routing tables (T1 and T2) like i saw here and in other websites. But the problem is that ping command shows “Host Unreachable” in both ways and command ARP -a shows the message “Incomplete”.

    Here is my configs in machine A (in B are the same with, of course, corrects IPs):

    IP Rule
    32762: from all to 10.1.2.1 lookup T2
    32763: from all to 10.1.1.1 lookup T1
    32764: from 10.1.2.1 lookup T2
    32765: from 10.1.1.1 lookup T1

    IP route show table T1
    default via 10.1.1.2 dev eth0 src 10.1.1.1
    10.1.1.0/24 dev eth0 scope link src 10.1.1.1

    IP route show table T2
    default via 10.1.2.2 dev eth1 src 10.1.2.1
    10.1.2.0/24 dev eth1 scope link src 10.1.2.1

    I have been following this website http://lartc.org/howto/lartc.rpdb.multiple-links.html#AEN2970 and im doing something wrong. Can u help me in my problem?

  135. This webpage finally explained how to make policy routing accessible. Thank you so much. In about 30 minutes I fixed all of my routing woes.

  136. Hello Darien.
    Thank you very much for this article very clear.
    Can you add to your article the corresponding configuration for / etc / network / interfaces file?

    because I do not know what to put too.

    Take care of yourself

  137. Hi,
    I’ve unexpectedly found myself needing to create multiple default routes, I’m using haproxy with keepalived on two servers, so I have:

    server1 server2 floating ip required gateway
    eth0: 192.168.52.251 192.168.52.252 192.168.52.250
    eth1: 192.168.51.251 192.168.51.252 192.168.51.250
    eth2.11 192.168.100.3 192.168.100.4 192.168.100.5 192.168.100.250
    eth2.12 192.168.101.3 192.168.101.4 192.168.101.5 192.168.101.250
    eth2.13 192.168.102.3 192.168.102.4 192.168.102.5 192.168.102.250
    eth3.5 192.168.50.3 192.168.50.4 192.168.50.5 192.168.50.250

    keepalived also includes the rest of the 192.168.100.0/24, 192.168.101.0/24 and 192.168.102.0/24 as virtual_ipadress_excluded.
    192.168.51.0/24 and 192.168.52.0/24 outbound traffic is SNAT’d to these virtual_ipaddress_excluded addresses
    (this is just test at the moment, the addresses on eth2.11, eth2.12, and eth2.13 will be changed to public ip’s for production)

    your instructions show ip rule add being used with the physical ip addresses, what would I do in this case? do I create rules using the physical addresses, the virtual addresses, or do I need to create rules for both physical and virtual addresses?

    thanks for any advice you can give.

  138. damn, formatting makes that hard to read,
    server1:
    eth0: 192.168.52.251
    eth1: 192.168.51.251
    eth2.11 192.168.100.3
    eth2.12 192.168.101.3
    eth2.13 192.168.102.3
    eth3.5 192.168.50.3

    server2:
    eth0: 192.168.52.252
    eth1: 192.168.51.252
    eth2.11: 192.168.100.4
    eth2.12: 192.168.101.4
    eth2.13: 192.168.102.4
    eth3.5: 192.168.50.4

    floating ips:
    192.168.52.250
    192.168.51.250
    192.168.100.5
    192.168.101.5
    192.168.102.5
    192.168.50.5

    required gateways:
    192.168.100.250
    192.168.101.250
    192.168.102.250
    192.168.50.250

    hopefully this makes it clearer.

    • I believe you will need to treat each interface (physical and virtual) that has a unique IP as a separate interface for policy routing. The commands should work on both interface types, I believe. You will have to likely experiment to see what works for you.

      Regards,
      — Darien

      • Hi,

        after a bit of experimentation, I’ve found I can get it done in just a few rules.
        192.168.50.0/24, 192.168.51.0/24, and 192.168.52.0/24 don’t require anything beyond their default settings
        192.168.100.0/24 has the default route.
        so it’s just rules for all the VIP’s on 192.168.101.0 and 192.168.102.0

        create two tables in /etc/iproute2/rt_tables : 101 table_101 and 102 table_102

        ip route add default via 192.168.101.250 dev eth2.12 table table_101
        ip rule add from 192.168.101.0/24 table table_101
        ip route add default via 192.168.102.250 dev eth2.13 table table_102
        ip rule add from 192.168.102.0/24 table table_102

        outbound from any of these addresses now use the correct interface. not created inbound rules yet, still working on that, but with what i have to test with, from haproxy to firewall to adsl router, internet bound traffic is getting NATted 3 times, so currently working on inbound traffic problems upstream from this machine first.

        hopefully this helps anyone else who finds themselves needing to create rules for large ranges of ip’s.
        I certainly wasn’t looking forward to the possibility of creating tables and rules for every individual ip.

  139. Hey Darien,

    I just wanted to thank you for the post and for keeping up with replies to everyone asking questions, even after 5 years! This post was exactly what I needed. I used this with my Raspberry Pi to simulate hosts for a Cisco lab network (and linked back): http://toddjames.io/post/64816553890/using-a-raspberry-pi-to-simulate-hosts-on-a-network

    Todd

    • Nice little project but for what you need it to do gns3 or virtualbox would be much better and more convenient

  140. Thanks! Found your blog while watching inbound http requests have responses sent out of the wrong interface.

    You’re the man.

  141. Great article. I was wondering if you can help me with a peculiar problem I am having.

    I have a Tomato Router with internet connection and public dynamic IP, x.x.x.x and 192.168.1.254 private interface.

    x.x.x.x (Tomato) 192.168.1.254

    I have port forwarded from (5000) that router using DNAT to a machine called Frodo 192.168.1.2. Frodo has a OpenVPN client that connects to VPN service so that my torrenting some other activities remain anonymous from Frodo. This has VPN address of 10.4.x.x (also dynamic).

    192.168.1.2 (Fordo) 10.4.x.x

    So prior to the VPN tunnel being brought up. When I get a request into Tomato on port 5000 that is forwarded to Frodo and all is well, because the default route on Frodo without the VPN running is 192.168.1.254 (default GW)

    Now when the OpenVPN system starts up on Frodo the default route changes to 10.4.1.1 since it eats up all trafffic. So now when a forwarded request reaches Frodo via the DNAT, it cannot make it back since its going back to internet IP:port because the default GW is via the VPN and so the forwaded port doesn’t respond to external traffic.

    I suspect I having a similar issue to what you suggested. I have tried whats on your post but it didn’t do the trick.. any suggestions?

    I have two interfaces eth0 and tun0 (VPN)

    Thanks!

    PS: All ports and IPs appearing in this post are fictitious. Any resemblance to real ports and ips, is purely coincidental.

    • Hi Manojav,

      Yes, this is a known issue with OpenVPN. You’re essentially looking to setup what’s known as a “split-tunnel VPN”. If you search Google for “openvpn split tunnel”, you will like find additional resources on how to accomplish this, such as: http://swimminginthought.com/routing-traffic-vpn/

      If those search results don’t yield anything useful, then my next recommendation would be to contact the OpenVPN support forums to look for a possible answer — mention or search for the “split tunnel” keywords, as well.

      Regards,

      • Darien,

        agreed this is due to default routes the vpn sets up. but the issue is that when i send a request that originates in the 192.168.1.x network the response is sent back over that interface, this only occurs for DNATed connections that need to traverse my NAT firewall and reach the internet through the default network GW 192.168.1.254 rather than the VPN default GW.

        so OpenVPN folks send me back to ip route folks and ip route folks bounce me back to openvpn.. suspect this is something I cannot do..

        what i need is to have a default gw as the vpn for all requests except for those to or from a certain port.. i tried setting fwmarks as well. but i dont know how to debug those packets..

        Thanks

        • Got it. Yeah, the bouncing back and forth between the forums can get quite old. Even though the 192.168.1.x network is internal, you could try treating that as an external network within the OpenVPN configuration, so that it thinks there is a split tunnel route altogether. Part of the problem, is I think OpenVPN may shim the kernel in a way that makes packet debugging a real pain…

          Regards,

            • Manojav
            • Posted October 30, 2013 at 1:20 pm
            • Permalink

            Big D!

            Thanks to you I have worked it out.

            ip route del table LAN
            ip route add 192.168.1.0/24 dev eth0 src 192.168.1.2 table LAN
            ip route add default via 192.168.1.252 dev eth0 table LAN

            ip rule add from 192.168.1.2/32 table LAN
            ip rule add to 192.168.1.2/32 table LAN

            ip route add default table LAN via 192.168.1.252
            ip rule add fwmark 1 table LAN
            ip route flush cache
            iptables -t mangle -I PREROUTING -p tcp –sport SRC_PORT_OF_SERVICE -j MARK –set-mark 1

            I stuck this in a shell script and called it on route-up on the opevpn connection.

            I am not sure what I am doing here.. but seems to work, new connections head out to the VPN GW. port forwarded connections also work.. do i have some hole open that I am not seeing?

            thanks a bunch! (for nothing :-))

  142. Une solution permettant d’avoir plusieurs réseaux différents. Donc avec plusieurs passerelles. Et ceci sur une seule machine ayant pour chaque réseau une carte réseau dédié.
    Cette solution fonctionne pour des réseaux public ou comme ici pour un réseau de classe C.
    Cette solution peut fonctionner également derrière une architecture VLAN.

    # The loopback network interface
    auto lo
    iface lo inet loopback

    # The network 0 interface
    auto eth0
    iface eth0 inet static
    address 192.168.0.1
    netmask 255.255.255.0
    broadcast 192.168.0.255
    network 192.168.0.0
    gateway 192.168.0.254
    up /sbin/ip route add default via 192.168.0.254 dev eth0 table network0
    up /sbin/ip rule add from 192.168.0.0/24 table network0
    post-down /sbin/ip rule del from 192.168.0.0/24 table network0
    post-down /sbin/ip route del default via 192.168.0.254 dev eth0 table network0
    # dns-* options are implemented by the resolvconf package, if installed
    ### Z1.Z2.Z3.Z4 U1.U2.U3.U4 : Les IPs de vos serveurs DNS
    dns-nameservers Z1.Z2.Z3.Z4 U1.U2.U3.U4
    dns-domain yourdomain.tld
    dns-search yourdomain.tld

    auto eth0:0
    iface eth0:0 inet static
    address 192.168.0.2
    netmask 255.255.255.0

    auto eth0:1
    iface eth0:1 inet static
    address 192.168.0.3
    netmask 255.255.255.0

    # The network 1 interface
    auto eth1
    iface eth1 inet static
    address 192.168.1.1
    netmask 255.255.255.0
    broadcast 192.168.1.255
    network 192.168.1.0
    ### gateway 192.168.1.254
    up /sbin/ip route add default via 192.168.1.254 dev eth1 table network1
    up /sbin/ip rule add from 192.168.1.0/24 table network1
    post-down /sbin/ip rule del from 192.168.1.0/24 table network1
    post-down /sbin/ip route del default via 192.168.1.254 dev eth1 table network1
    # dns-* options are implemented by the resolvconf package, if installed
    ### Z1.Z2.Z3.Z4 U1.U2.U3.U4 : Les IPs de vos serveurs DNS
    dns-nameservers Z1.Z2.Z3.Z4 U1.U2.U3.U4
    dns-domain yourdomain.tld
    dns-search yourdomain.tld

    auto eth1:0
    iface eth1:0 inet static
    address 192.168.1.2
    netmask 255.255.255.0

    auto eth1:1
    iface eth1:1 inet static
    address 192.168.1.3
    netmask 255.255.255.0

    # The network 2 interface
    auto eth2
    iface eth1 inet static
    address 192.168.2.1
    netmask 255.255.255.0
    broadcast 192.168.2.255
    network 192.168.2.0
    ### gateway 192.168.2.254
    up /sbin/ip route add default via 192.168.2.254 dev eth2 table network2
    up /sbin/ip rule add from 192.168.2.0/24 table network2
    post-down /sbin/ip rule del from 192.168.2.0/24 table network2
    post-down /sbin/ip route del default via 192.168.2.254 dev eth2 table network2
    # dns-* options are implemented by the resolvconf package, if installed
    ### Z1.Z2.Z3.Z4 U1.U2.U3.U4 : Les IPs de vos serveurs DNS
    dns-nameservers Z1.Z2.Z3.Z4 U1.U2.U3.U4
    dns-domain yourdomain.tld
    dns-search yourdomain.tld

    auto eth2:0
    iface eth2:0 inet static
    address 192.168.2.2
    netmask 255.255.255.0

    auto eth2:1
    iface eth2:1 inet static
    address 192.168.2.3
    netmask 255.255.255.0

    Enjoy ! ;-)

  143. Awesome article! This was the only way my vpn would even start with two nics a while back. Thanks!
    Here is my goal: Seperate network traffic. I want use my paid vpn service provider to encrypt the connection for a single application. The rest of my network applications I want to operate outside of my vpn. It doesn’t matter to me how I manage this goal: I have two network cards available – the default one in my motherboard and I have a spare pci NIC. I will install that extra nic if I need to but I’d prefer to somehow use virtual interfaces. I want to be able to use port forwarding for the rest of my network applications so they can hit the internet so i access them remotely I’m using Ubuntu 12.04. I’ve been playing with the bind and local options but still haven’t been able to figure this out. i’m sure this is similiar to what others have been saying on this thread, but I think this is slightly different and more basic. Do virtual interfaces work the same way as physical nics?

    Thanks Darien

    • The virtual NICs should operate the exact same way as the physical NICs. The major challenge, will be understanding all the custom IPs that your application uses, which will require custom routes for each netmask.

      Regards,
      — Darien

      • Thanks for the advice Darien and for getting back to me so quickly. I will answer sooner next time, I was just away for the week. So it seems like I shouldn’t use the second NIC and just use virtual interfaces. Let’s say for educational sake that
        ETH0 = 4.4.4.4
        and ETH0:1 = 4.4.4.5

        I want to run rtorrent + rutorrent in 4.4.4.5. Everything else is fine at 4.4.4.4. Can you help me configure the routes?

        • It really depends on the applications. You need to figure out all the TCP/UDP ports that the torrent clients use, in order to develop a same routing policy. My best recommendation, would be to run your torrent clients while capturing full PCAPs in order to see what the common ports/IPs are, which would be the basis for your defined routes.

          I am not intimately familiar with all the protocols used by standard torrent clients.

          Regards,
          — Darien

            • Anonymous
            • Posted December 16, 2013 at 10:42 am
            • Permalink

            Hey Darien,

            I have another follow up question about my above post. I think I have everything working properly.
            I have two interfaces-
            So right now everything is working fine with the below routing command in webmin custom commands.
            sudo echo “1 admin” >> /etc/iproute2/rt_tables; sudo ip route add 4.4.4.0/24 dev eth0 src 4.4.4.4 table admin; sudo ip route add default via 4.4.4.1 dev eth0 table admin; sudo ip rule add from 4.4.4.4/32 table admin; sudo ip rule add to 4.4.4.4/32 table admin; sudo ip route flush cache; sudo ip rule show

            Would there be any benefit to creating a table with the commands below for the vpn?
            sudo echo “1 vpn” >> /etc/iproute2/rt_tables; sudo ip route add 4.4.4.0/24 dev eth0:1 src 4.4.5.4 table vpn; sudo ip route add default via 4.4.4.1 dev eth0:1 table vpn; sudo ip rule add from 4.4.5.4/32 table vpn; sudo ip rule add to 4.4.5.4/32 table vpn; sudo ip route flush cache; sudo ip rule show

            Thanks!

            • Darien
            • Posted December 16, 2013 at 2:57 pm
            • Permalink

            I’d recommend changing the “1 vpn” string to “2 vpn”, because you already have “1 admin” defined, and your follow on commands will overwrite, rather than append to the routing policy.

            Aside from that, the VPN commands you have seem like it would work. My recommendation would be to try and see, as I know others were able to get it to work on virtual interfaces successfully.

            Regards,
            — Darien

            • Anonymous
            • Posted December 16, 2013 at 3:52 pm
            • Permalink

            Thanks Darien I updated my code for the vpn table- sudo echo “2 vpn2″ >> /etc/iproute2/rt_tables; and called the table vpn2

            0: from all lookup local
            32762: from all to 4.4.5.4 lookup vpn2
            32763: from 4.4.5.4 lookup vpn2
            32764: from all to 4.4.4.4 lookup admin
            32765: from 4.4.4.4 lookup admin
            32766: from all lookup main
            32767: from all lookup default ”

            This is my output. So these two connections should be totally isolated at this point.

            No my question is which becomes the default route for any internet connection? I want 4.4.4.4 admin table to be the default route, but I always thought lower numbers in linux get the priority.

            Thanks Darien this page is awesome!

            • Darien
            • Posted December 16, 2013 at 4:13 pm
            • Permalink

            Yes, you’d want to swap the ordering, so that the admin table gets lower priority numbering (which means it gets evaluated first).

            Aside from that, I’d recommend testing it out, to make sure everything is working, by running tcpdump, accordingly.

            Regards,
            — Darien

  144. Darien. I use two interfaces in parallel, basically setup as your example. Now the problem I am having is that when the primary interface (The one with the default gateway in the main routing table) is gone (And there is no default gateway route in the main table anymore), I am not able to do any dns lookups anymore. (Traceroute still finds the dns server though) Are there perhaps other ip rules to add for this case or should you fall back to adding the secondary gateway to the main table?

    • Basically, policy routing is not going to help you achieve failover routing, which is what you’re looking to do. I would recommend doing a Google search (terms: “linux failover routing”) to see what options are available for Linux fail over routing. It is likely you will need to install a custom tool to accomplish this.

      Regards,
      — Darien

      • Darien, this is a fantastic article! I came back to read it again, as I am having problems configuring my interfaces with Debian Wheezy. I made it work on my previous server, but sadly the motherboard died and when trying to replicate I am failing. One thing I noticed for a start, the routing table with one network card looks like this:
        Destination Gateway Genmask Flags MSS Window irtt Iface
        0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0
        192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
        239.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0

        The default route appears first unlike previous time. Does that matter? From there on, the ip route add and ip rule add commands do get executed, but no change on routing table occurs. Any pointers?
        Thanks a lot,

        Ivo

        • Hi Ivo,

          It sounds like the kernel doesn’t have policy routing compiled in, which means you may need to custom compile your kernel with that support, accordingly.

          Regards,

            • ivo
            • Posted April 16, 2014 at 7:39 pm
            • Permalink

            Well I checked my /boot/config and policy routing is enabled. I do thing is something to do with the way interfaces are configured at boot time, but seem unable to find exactly what is wrong. I have two interfaces and use same gateway. My /etc/network/interfaces:
            auto eth0
            allow-hotplug eth0
            iface eth0 inet static
            address 192.168.0.11
            netmask 255.255.255.0
            network 192.168.0.0
            broadcast 192.168.0.255
            gateway 192.168.0.1
            post-up ip route add 192.168.0.0/24 dev eth0 src 192.168.0.11 table uplink0
            post-up ip route add default via 192.168.0.1 dev eth0 table uplink0
            post-up ip rule add from 192.168.0.11/32 table uplink0
            post-up ip rule add to 192.168.0.11/32 table uplink0
            auto eth1
            allow-hotplug eth1
            iface eth1 inet static
            address 192.168.0.21
            netmask 255.255.255.0
            network 192.168.0.0
            broadcast 192.168.0.255
            # gateway 192.168.0.1
            post-up ip route add 192.168.0.0/24 dev eth1 src 192.168.0.21 table uplink1
            post-up ip route add default via 192.168.0.1 dev eth1 table uplink1
            post-up ip rule add from 192.168.0.21/32 table uplink1
            post-up ip rule add to 192.168.0.21/32 table uplink1
            # #dns-* options implemented by resolver
            dns-nameservers 8.8.8.8 8.8.4.4
            dns-search mydomain.com
            If I enable gateway on eth 1, interface does not get configured. If I disable eth0 gateway, the routing table gets configured with routing table configured for both interfaces correctly, but no gateway therefore unable to reach internet at all. If however I boot with the above configuration, the routing table is:

            Kernel IP routing table
            Destination Gateway Genmask Flags MSS Window irtt Iface
            0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0
            192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
            192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
            239.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0

            and I am back to the original problem, all output goes through eth0. Any hints?
            Thanks a lot for taking your time.

  145. WOW!!! This is brilliant!! This helped me a lot with an engineering project that I’ve been on involving a Linux server. You are a life saver!!! Thank you so much!

  146. thanks for your help. you save my job. you are boss.

  147. Wyjątkowo interesujący tekst, polecam wszystkim

  148. Let me add more info to this excellent article:

    https://access.redhat.com/site/solutions/53031

    That is a alternative to the problem: it allows the traffic to pass through the default route dev.

  149. Thanks for the info and it helped. This info is nicely putup.


34 Trackbacks/Pingbacks

  1. By Core » Tunkkausmaanantai o/ on 11 May 2009 at 8:18 pm

    [...] Tästä ohjeesta soveltaen väänsin sitten iproute.sh nimisen scriptin joka sisältää seuraavaa. #!/bin/sh [...]

  2. By links for 2009-10-20 « Donghai Ma on 21 Oct 2009 at 4:05 am

    [...] Configuring Multiple Default Routes in Linux « Darien Kindlund’s Blog (tags: linux network networking routing gateway howto IP iptables policy) [...]

  3. [...] met enige goede wil (en beetje kennis of wil om man pages te lezen) kun je hier iets mee: http://kindlund.wordpress.com/2007/1…utes-in-linux/ Daarmee kun je de ene gateway aan de ene stateful table knopen, en de andere aan de 2e [...]

  4. [...] the method from this website, we create a policy route.  I am using Ubuntu 10.04 Server 64bit in this example which has [...]

  5. [...] Lösung für dieses Problem findet mann in iproute2. In seinem Artikel Configuring Multiple Default Routes in Linux erkärt Darian Kindlund sehr gut, wie man mehrere default routs konfiguriert. Tags: [...]

  6. By How do I setup Multi-Path route iptables on 14 Jun 2010 at 8:28 pm

    [...] Take a look at this site, it has an explanation and some information on how to do this with iproute2 (required). http://kindlund.wordpress.com/2007/1…utes-in-linux/ [...]

  7. [...] site for reference, but the one that explained exactly what I wanted to do and helped me most was http://kindlund.wordpress.com/2007/1…utes-in-linux/ Many thanks to this [...]

  8. [...] Configuring Multiple Default Routes in Linux « Darien Kindlund’s Blog. [...]

  9. [...] think I have found the answer at this link. I will be trying this out on my server and will post again here the results. [...]

  10. [...] Basically I followed this webpage to set it up : http://kindlund.wordpress.com/2007/1…utes-in-linux/ [...]

  11. [...] This blog entry explains how to setup multiple routes for Linux in some detail. [...]

  12. [...] eventually found a solution by combining two excellent blog posts (this one and this one) as well as some trial and error in figuring out how to make the changes persist on a [...]

  13. [...] guide here 0.000000 0.000000 Like this:LikeBe the first to like this post. Comments RSS [...]

  14. By 2 NIC's 2 seperate networks, traffic routing on 26 Dec 2011 at 8:01 am

    [...] it's a bit complicated! You might want to start at http://lartc.org/howto/ Also, I found this blog entry which might give a simpler introduction to the [...]

  15. By dual homed centos on 22 Feb 2012 at 7:56 am

    [...] has two gateway lines, including one for eth1, which I don't have) The same person suggested I do http://kindlund.wordpress.com/2007/1…utes-in-linux/ But then today, it seems to be working and I didn't change anything since last night. Why might it [...]

  16. By Routing 2 IPs on 05 Mar 2012 at 6:49 am

    [...] Originally Posted by kopper27 looks like I have to use iproute2 just don't know how any guidance thanks try this link http://kindlund.wordpress.com/2007/1…utes-in-linux/ [...]

  17. [...] and default routes for each interface: http://www.debian-administration.org…Policy_routing http://kindlund.wordpress.com/2007/1…utes-in-linux/ However, when ppp0 is up, "ping -I ppp0 8.8.8.8" from the firewall gets a response, but [...]

  18. [...] searched the solution, found Configuring Multiple Default Routes in Linux, followed the steps, and [...]

  19. [...] pulled most of this technique from this site.  Works like a champ for my purposes! Post a [...]

  20. By Add simple script to "service network restart" on 05 Jun 2012 at 7:15 am

    [...] with the old 'route' command, but it is doable with 'ip' by creating multiple route tables.. http://kindlund.wordpress.com/2007/1…utes-in-linux/ http://www.generationip.com/document…teway-on-linux To answer your question about running a [...]

  21. By Multiple gateways on Linux « Install Process on 30 Jun 2012 at 5:24 pm

    [...] here Share this:TwitterFacebookLike this:LikeBe the first to like this. Posted by baxteragu Filed in [...]

  22. By Multiple "default" gw on 30 Jul 2012 at 8:18 am

    [...] Hi Take a look over here [...]

  23. By OpenVPN struggles on 17 Jan 2013 at 10:23 pm

    [...] a look at this http://kindlund.wordpress.com/2007/1…utes-in-linux/ it's upper level stuff but I think it may be the key to getting it to work the way you want. It was [...]

  24. […] I've managed are strictly for backup networks or failover), so can't vouch for the instructions in this blog post, but they may […]

  25. By 1 server, 2 nics, 2 networks on 10 Dec 2013 at 9:24 am

    […] need 2 default gateways and 2 routing tables I did a little more Googling and found this article: http://kindlund.wordpress.com/2007/1…utes-in-linux/ The server is online and accessible correctly on both […]

  26. By problem in adding static routes in centos 6.5 on 01 Apr 2014 at 9:11 am

    […] In the normal network stack you get one route to any destination. Regardless of which NIC receives a ping, for example, the reply will go out through the default gateway. To do advanced policy-based routing have a look at this: http://kindlund.wordpress.com/2007/1…utes-in-linux/ […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: