Setting up jails with multiple IPs and providing it with internet access
I have multiple VMs up and running and generally I have an internal to the VMs only network set up, in the current project I am working on I have three completely different networks set up.
- DHCP assigned network that routes to the outside network on the host (192.168.0.1/24)
- An internal to the VM's network interface (10.99.12.1/24)
- A network that is tied to the hosts wireless network (setup to be an AP) (10.99.11.1/24)
So within my FreeBSD VM I have the following: em0, em1, em2 as "network cards". In my FreeBSD VM, I am creating two jails that are going to contain different pieces of server software. The jails need ip's in the following ranges, 10.99.12.1/24 (to connect to another VM), 10.99.11.1/24 (Wifi network) and then it needs to have an IP address that is considered "local" and we can NAT on, so in addition to the above networks we also needed to add another network. The easiest way to create new interface with IP address' is using a bridge. They can be assigned IP addresses, which is all we need.
At this point inside of a jail it kinda looks like this:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:6b:6a:49 media: Ethernet autoselect (1000baseT <full-duplex>) status: active em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:2f:a5:55 inet 10.99.12.4 netmask 0xffffffff broadcast 10.99.12.4 media: Ethernet autoselect (1000baseT <full-duplex>) status: active em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:bb:b6:43 inet 10.99.11.14 netmask 0xffffffff broadcast 10.99.11.14 media: Ethernet autoselect (1000baseT <full-duplex>) status: active lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=3<RXCSUM,TXCSUM> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether da:de:63:c9:55:6c inet 10.0.1.4 netmask 0xffffffff broadcast 10.0.1.4 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
em0 doesn't have an address since that is our DHCP address, em1, em2 and bridge0 all have IP addresses, in the jail configuration the bridge0 IP address is listed first, and thus when something binds or sends it sends from that IP address. The thing is that at this point the jail does not have internet access.
Jail's inherit the routing table from their host environment (yes, you can compile in more routing tables ... but I don't want to modify stock), any traffic that goes to the outside is going to go out over the default route, thing is you'd expect that traffic to show up on bridge0 where the IP address exists, instead using tcpdump you see it show up on em0, which is where the default route is. I would have expected having to do NAT on the bridge, not just the external interface.
Here is the pf configuration.
ext_if="em0" jail_if="bridge0" set optimization aggressive set block-policy drop set skip on lo nat on $ext_if from $jail_if:network:0 to any -> ($ext_if) pass quick all
And we've got internet within our jail. I am looking forward to setting this all up when the new VIMAGE stuff lands in 9.0-RELEASE so that I can have virtual network cards that are not tied to a physical network card!