My ISP, EPB of Chattanooga, will kindly provide more than one publicly routable IP address via DHCP. They do this, I think, in order to allow customers to use only a Layer-2 switch instead of a router to network all of their Windows boxen and game consoles, at their own Wild-West peril (observed at my boss’s house, which is how I got this idea). I’m not sure how many addresses EPB would hand out per fiber gateway, but I suspect it would be way more than I would ever use. Even with the imminent IPv4 crunch, we dine like kings…

I wanted to use this nice feature to multi-home my server. Never mind that IPv6 will obsolete the benefits. Name-based virtual hosting has its limitations (e.g. SSL) and I’m not waiting around!

One easy way to do this is to run virtual machines and bridge them to the port. But that has complications of its own and I didn’t feel like adding a new VM for every new low-traffic website I want to host. Of course, the VMs could be pretty small and optimized for routing, in which case I could run a bunch of them and send the traffic to a master Apache installation in Ring #0. But that’s not what I did.

So here’s the method:

1. Run multimac to give us virtual TAP ports with real-looking MACs.

/etc/init.d/multimac:

#!/sbin/runscript

opts="start stop restart"

depend() {
   need localmount
}

makemac() {
   killall -q multimac
   /usr/local/bin/multimac 1
   #one greater than eth1's MAC. The Gentoo scripts can do this as well but this seemed safer.
   ifconfig tap1 hw ether YOUR FAKE MAC HERE
}

start() {
   ebegin "Starting multimac"
   makemac
   eend $?
}

stop() {
   ebegin "Stopping multimac"
   killall -q multimac
   eend $?
}

restart() {
   ebegin "Restarting multimac"
   makemac
   eend $?
}

rc-update add multimac default

2. Policy routing for multiple uplinks

/etc/conf.d/net (excerpts):

modules=("dhcpcd")
dhcp="nontp nonis"

bridge_br1="eth1 tap0"
config_eth1=( "null" )
config_tap0=( "null" )
config_br1=( "dhcp" )
dhcpcd_br1="-L -t 0 -G"

#This is a static address "aliased" via "ip rule" to tap1
config_lo=(
   "172.16.0.60/32"
)

config_tap1=( "dhcp" )
#Supply a custom hostname so as not to confuse the DHCP server.
dhcpcd_tap1="-L -t 0 -G -h penguin2"
dhcp_tap1="nontp nonis nodns"

RC_NEED_tap0="multimac"
RC_NEED_br1="multimac net.eth1 net.tap0"
RC_NEED_tap1="multimac net.br1"

/etc/conf.d/local.start :

#The first three rules are for LAN and VPN traffic
ip rule add order 100 to 10.0.1/24 table main
ip rule add order 101 to 172.16.10/24 table main
ip rule add order 102 to 172.16.11/24 table main
ip rule add order 103 from 172.16.0.60 table tap1

Some iptables stuff (excerpts):

IPT=/sbin/iptables
#Traffic coming from the real world should have real addresses. This does basically the same thing as
#rp_filter but rp_filter is broken for multiple routing tables.
$IPT -N rp_filter_t
$IPT -A rp_filter_t -s 192.168.0.0/16 -j DROP
$IPT -A rp_filter_t -s 172.16.0.0/12 -j DROP
$IPT -A rp_filter_t -s 10.0.0.0/8 -j DROP
#else RETURN

for i in br1 tap1; do
   for j in FORWARD INPUT; do
      $IPT -A $j -i $i -j rp_filter_t
   done 
done

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/default/rp_filter

for j in br1 tap1; do
   $IPT -t nat -A POSTROUTING -o $j -j MASQUERADE
done;

#For Apache:
$IPT -t nat -A PREROUTING -i br1 -p tcp --dport 443 -j DNAT --to 10.0.1.1
$IPT -t nat -A PREROUTING -i br1 -p tcp --dport 80 -j DNAT --to 10.0.1.1
$IPT -t nat -A PREROUTING -i tap1 -p tcp --dport 80 -j DNAT --to 172.16.0.60

/etc/iproute2/rt_tables :

255	local
254	main
253	default
0	unspec
100	br1
101	tap1

3. dhcpcd’s run-hooks feature to reconfigure routes when addresses change

/etc/dhcpcd.exit-hook (where the action happens):

#!/bin/bash

#useful commands:
#   dhcpcd -V to see available variables.
#   dhcpcd -T to spit out all variables received by the DHCP server

echo `date` "$reason   $interface" >> /var/log/multi_epb_log

if [ -z "$new_ip_address" ] || [ "$reason" == "RENEW" ]; then
   #various events that don't result in new IPs
   exit
fi

save_tables() {
   ip route > $1
   echo >> $1
   echo "br1:" >> $1
   ip route show table br1 >> $1
   echo >> $1
   echo "tap1:" >> $1
   ip route show table tap1 >> $1
   echo >> $1
   echo "rules:" >> $1
   ip rule >> $1
   echo >> $1
   date >> $1
}

save_tables /pre_tables

#try to clear out some cruft--------
#one or more of these could cough up an error, but it does not indicate failure.

ip route flush table $interface

if [ "$interface" == "br1" ]; then
   #the main table default gw
   ip route del default
fi

if [ -n "$old_network_number" ] && [ -n "$old_subnet_cidr" ]; then
   OLD_NET_STR="$old_network_number/$old_subnet_cidr"
   #In the case that the old and new were the same subnet (maybe different IPs, though), there will be two almost-matching entries (save for the
   #"src" field present on the older one)
   #This deletes one of them and the other will be deleted below.
   ip route del "$OLD_NET_STR" dev $interface
fi

NET_STR="$new_network_number/$new_subnet_cidr"

#for instance, "ip route del 123.123.123.123/24 dev br1"--delete the route added by dhcpcd. Supply the dev in case
#another interface is on the same subnet.
ip route del "$NET_STR" dev $interface

#------------------
#now add some new stuff

if [ "$interface" == "br1" ]; then
   ip route add "$NET_STR" dev br1 src "$new_ip_address"
   ip route add default via "$new_routers" dev br1

   #maybe the only way to do this "right" would be to establish a mapping between interfaces and integers,
   #since this approach only works if there is only one simulated interface (tap1). Order/preference
   #numbers have to be unique.
   ip rule del order 200
   ip rule add from $new_ip_address table $interface order 200
else
   #the next route shouldn't be needed, but is referenced (hardlinked?) by the tap1 table gateway.
   #The only time it would ever be referenced would be for locally originated connections addressed
   #to tap1's subnet, if it is different from br1's. That is why tap1 needs a masquerade rule in iptables.
   ip route add "$NET_STR" dev $interface metric 2000 src "$new_ip_address"

   #see note above
   ip rule del order 201
   ip rule add from $new_ip_address table $interface order 201
fi

ip route add "$NET_STR" table $interface dev $interface
ip route add default via "$new_routers" table $interface dev $interface

#Critical! It seems like arp_filter should work, but it doesn't for some reason.
#I was prepared to implement this using arptables before I found out the kernel does it for me.
echo 2 > /proc/sys/net/ipv4/conf/"$interface"/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/"$interface"/arp_ignore

#This should be off anyway, but make SURE! It's broken with multiple tables and iptables rules do the same thing.
echo 0 > /proc/sys/net/ipv4/conf/"$interface"/rp_filter

ip route flush cache

save_tables /post_tables

4. Bind Apache to an internal static IP and use DNAT

5. Finally, some gotcha’s

One interesting thing is that the netfilter PREROUTING hook apparently occurs *after* the RPDB lookup, since unless ip rule 103 from step 2 is included, the DNAT iptables rule is not sufficient to cause the tap1 table to be selected. So much for “pre” routing! See this. “Stateless” NAT seems like it would be a better method, but it’s been deprecated.

See the note about arp_announce and arp_ignore. I think that arp_filter may fail for the same reason as rp_filter in this scenario, since my arp_announce and arp_ignore options seem to describe what arp_filter should have done. Read about it in /usr/src/linux/Documentation/networking/ip-sysctl.txt .