Personal.X-Istence.com

Bert JW Regeer (畢傑龍)

Setting up jails with multiple IPs and providing it with internet access

I have multiple VMs up and running and generally I have an internal to the VMs only network set up, in the current project I am working on I have three completely different networks set up.

  1. DHCP assigned network that routes to the outside network on the host (192.168.0.1/24)
  2. An internal to the VM's network interface (10.99.12.1/24)
  3. A network that is tied to the hosts wireless network (setup to be an AP) (10.99.11.1/24)

So within my FreeBSD VM I have the following: em0, em1, em2 as "network cards". In my FreeBSD VM, I am creating two jails that are going to contain different pieces of server software. The jails need ip's in the following ranges, 10.99.12.1/24 (to connect to another VM), 10.99.11.1/24 (Wifi network) and then it needs to have an IP address that is considered "local" and we can NAT on, so in addition to the above networks we also needed to add another network. The easiest way to create new interface with IP address' is using a bridge. They can be assigned IP addresses, which is all we need.

At this point inside of a jail it kinda looks like this:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
    ether 52:54:00:6b:6a:49
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
    ether 52:54:00:2f:a5:55
    inet 10.99.12.4 netmask 0xffffffff broadcast 10.99.12.4
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
    ether 52:54:00:bb:b6:43
    inet 10.99.11.14 netmask 0xffffffff broadcast 10.99.11.14
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=3<RXCSUM,TXCSUM>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether da:de:63:c9:55:6c
    inet 10.0.1.4 netmask 0xffffffff broadcast 10.0.1.4
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0

em0 doesn't have an address since that is our DHCP address, em1, em2 and bridge0 all have IP addresses, in the jail configuration the bridge0 IP address is listed first, and thus when something binds or sends it sends from that IP address. The thing is that at this point the jail does not have internet access.

Jail's inherit the routing table from their host environment (yes, you can compile in more routing tables ... but I don't want to modify stock), any traffic that goes to the outside is going to go out over the default route, thing is you'd expect that traffic to show up on bridge0 where the IP address exists, instead using tcpdump you see it show up on em0, which is where the default route is. I would have expected having to do NAT on the bridge, not just the external interface.

Here is the pf configuration.

ext_if="em0"
jail_if="bridge0"

set optimization aggressive
set block-policy drop
set skip on lo

nat on $ext_if from $jail_if:network:0 to any -> ($ext_if)

pass quick all

And we've got internet within our jail. I am looking forward to setting this all up when the new VIMAGE stuff lands in 9.0-RELEASE so that I can have virtual network cards that are not tied to a physical network card!