Networking for untrusted Xen domUs in Debian

From Wiki
Jump to: navigation, search

The recommended setup for Xen networking is to use the network-bridge script. In a typical setup, this will look like:

# brctl show
eth0		8000.0017a442088c	no		peth0
							vif1.0
							vif2.0
							vif3.0
							vif4.0

This has placed the dom0's eth0 interface and all domU virtual network interfaces into the same layer 2 network, like so:

           dom0 -------- bridge --> eth0 --> the network
                           ^
                           |
domU 1:    eth0 -> vif1.0 -+
                           |
domU 2:    eth0 -> vif2.0 -+
                           |
domU 3:    eth0 -> vif3.0 -+
                           |
domU 4:    eth0 -> vif4.0 -+

Often, this is what you want. But what if, however, we do not trust the owners of the domUs to play nicely?

The Problem

  • The owner of an untrusted domU could assign themselves any IP address they wish, including ones intended for other domUs or the default gateway for the network. Technically speaking, that would be bad.
  • There is no control over domU ingress or egress traffic because the dom0 is not a router.

The Solution

iptables has the capability to filter traffic at layer 2, which is exactly what is needed for controlling traffic on a bridge interface. Calling network-bridge with the antispoof=yes argument is supposed to set this up, but I found two problems with this:

  • Rules are only created to permit egress traffic from the domUs, no ingress ACCEPT rules are created.
  • The script adds a rule each time a vif is torn down and recreated irrespective of whether a rule already exists. If several xm create/xm destroy cycles are run, this makes the iptables rules fairly messy.

I prefer to disable the automatic iptables rule manipulation and take control myself.

The network-bridge script calls the vif-common.sh script to alter the iptables rules, so this can be altered like so:

# vi /etc/xen/scripts/vif-common.sh

 # jsjsjsjs disable iptables commands, I'll do it myself thanks.
  #iptables "$c" FORWARD -m physdev --physdev-in "$vif" "$@" -j ACCEPT \
  #  2>/dev/null &&
  #iptables "$c" FORWARD -m state --state RELATED,ESTABLISHED -m physdev \
  #  --physdev-out "$vif" -j ACCEPT 2>/dev/null

You can tell which vif is being used by which domU like so...

# xm network-list my.xen.domu
Idx BE     MAC Addr.     handle state evt-ch tx-/rx-ring-ref BE-path
0   0  00:16:3E:E5:B9:22    0     4      10    768  /769     /local/domain/0/backend/vif/2/0  

...but ensuring that each domU gets the same vifN.0 interface assigned to it each time will simplify matters greatly. Edit the vif= line of each domU so that it includes at least the vif interface number to be used. It is helpful but not necessary to include the IP address, this makes it easier to identify which IP addresses should be permitted on which vif:

vif = [ 'ip=172.16.0.36,mac=00:15:DD:0B:2E:14,vifname=vif4.0' ]

Next, create a file which can be used with iptables-restore, or edit your existing iptables script as appropriate:

# vi /etc/iptables.rules 
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
#
## These are the anti-spoofing rules for our domUs
-A FORWARD -m state --state RELATED,ESTABLISHED -m physdev --physdev-out vif1.0 -j ACCEPT
-A FORWARD -p udp -m physdev --physdev-in vif1.0 -m udp --sport 68 --dport 67 -j ACCEPT
-A FORWARD -s 172.16.0.16/32 -m physdev --physdev-in vif1.0 -j ACCEPT
-A FORWARD -d 172.16.0.16/32 -m physdev --physdev-out vif1.0 -j ACCEPT
-A FORWARD -s 172.16.0.15/32 -m physdev --physdev-in vif1.0 -j ACCEPT
-A FORWARD -d 172.16.0.15/32 -m physdev --physdev-out vif1.0 -j ACCEPT
-A FORWARD -s 172.16.0.14/32 -m physdev --physdev-in vif1.0 -j ACCEPT
-A FORWARD -d 172.16.0.14/32 -m physdev --physdev-out vif1.0 -j ACCEPT
-A FORWARD -s 172.16.0.22/32 -m physdev --physdev-in vif1.0 -j ACCEPT
-A FORWARD -d 172.16.0.22/32 -m physdev --physdev-out vif1.0 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -m physdev --physdev-out vif3.0 -j ACCEPT
-A FORWARD -p udp -m physdev --physdev-in vif3.0 -m udp --sport 68 --dport 67 -j ACCEPT
-A FORWARD -s 172.16.0.29/32 -m physdev --physdev-in vif3.0 -j ACCEPT
-A FORWARD -d 172.16.0.29/32 -m physdev --physdev-out vif3.0 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -m physdev --physdev-out vif2.0 -j ACCEPT
-A FORWARD -p udp -m physdev --physdev-in vif2.0 -m udp --sport 68 --dport 67 -j ACCEPT
-A FORWARD -s 172.16.0.10/32 -m physdev --physdev-in vif2.0 -j ACCEPT
-A FORWARD -d 172.16.0.10/32 -m physdev --physdev-out vif2.0 -j ACCEPT
-A FORWARD -p udp -m physdev --physdev-in vif4.0 -m udp --sport 68 --dport 67 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -m physdev --physdev-out vif4.0 -j ACCEPT
-A FORWARD -s 172.16.0.36/32 -m physdev --physdev-in vif4.0 -j ACCEPT
-A FORWARD -d 172.16.0.36/32 -m physdev --physdev-out vif4.0 -j ACCEPT
## If it's not an IP address allowed on that vif, log and drop it.
-A FORWARD -m limit --limit 15/min -j LOG --log-prefix "Dropped by firewall: " --log-level 7
-A FORWARD -j DROP
#
## And now any access rules for the dom0 machine itself:
-A INPUT -j ACCEPT
COMMIT

Apply with:

# iptables-restore < /etc/iptables.rules

You should find that attempting to assign a 'forbidden' IP address on a domU does not work – you will see the packets being dropped in the dom0's syslog.

To make permanent across reboots:

# vi /etc/network/interfaces

auto eth0
iface eth0 inet static
 address p.q.r.s
 netmask 255.255.255.0
 gateway n.o.p.q
 pre-up iptables-restore < /etc/iptables.rules

A Subnet for the DomUs

Even better is to allocate a subnet for the use of the domU machines. This makes it easier to secure the dom0, since it's now very easy to permit traffic to the domUs but block traffic to the dom0 based on destination address. Handily, this is often this is how hosting companies will supply IP address ranges if you have ordered additional IPs. Using the example from earlier, this would look like:

                                    dummy0 
                                172.16.2.0/24 
                                    ^  |
                                    |  |
                                    |  v
                         bridge --> dom0 --> eth0 --> the network
                           ^
                           |
domU 1:    eth0 -> vif1.0 -+
                           |
domU 2:    eth0 -> vif2.0 -+
                           |
domU 3:    eth0 -> vif3.0 -+
                           |
domU 4:    eth0 -> vif4.0 -+

Effectively, this uses a router-on-a-stick topology within our Xen host. The 172.16.2.0/24 subnet has been routed to the address of the dom0's eth0 interface, so we have created a dummy0 interface and assigned 172.16.2.1 to it in order to route this traffic.

Important: Don't forget to enable ip forwarding if it isn't already:

# vi /etc/sysctl.conf

# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

Assign an address to dummy0:

# ifconfig dummy0 172.16.2.1 netmask 255.255.255.0

Make it a permanent addition:

# vi /etc/network/interfaces

auto dummy0
iface dummy0 inet static
 address 172.16.2.1
 netmask 255.255.255.0

You should be able to reach other hosts using the address on dummy0:

# ping -I 172.16.2.1 8.8.8.8 -c3
PING 8.8.8.8 (8.8.8.8) from 88.198.170.177 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_req=1 ttl=51 time=5.68 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=51 time=5.93 ms
64 bytes from 8.8.8.8: icmp_req=3 ttl=51 time=6.66 ms
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 5.684/6.095/6.664/0.415 ms

If not, you need to stop and check your routing at this point. Otherwise, continue by configuring Xen to bridge with the dummy0 interface rather than eth0:

# vi /etc/xen/xend-config.sxp

#(network-script 'network-bridge)
(network-script 'network-bridge netdev=dummy0')

At this point, it's best to reboot the dom0. There's a lot of stopping and starting of bridges and renaming eths to peths and it's far too easy to screw yourself right in the pooper if you try to do it manually.

When the dom0 comes back, you'll find that it has created a bridge between dummy0 and the domUs:

# brctl show
dummy0		8000.c28f9fe86b7e	no		pdummy0
							vif1.0
							vif2.0
							vif3.0
							vif4.0

This is good news. You can now assign your domUs addresses in the 172.16.2.0/24 range and it should work fine.

On to iptables. The setup is virtually the same as it was when the domUs were bridged with eth0, but we can add some simple rules which will protect the dom0 from the domUs and other hosts:

# vi /etc/iptables.rules

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
#
## These are the anti-spoofing rules for our domUs
-A FORWARD -m state --state RELATED,ESTABLISHED -m physdev --physdev-out vif1.0 -j ACCEPT
-A FORWARD -p udp -m physdev --physdev-in vif1.0 -m udp --sport 68 --dport 67 -j ACCEPT
-A FORWARD -s 172.16.2.16/32 -m physdev --physdev-in vif1.0 -j ACCEPT
-A FORWARD -d 172.16.2.16/32 -m physdev --physdev-out vif1.0 -j ACCEPT
-A FORWARD -s 172.16.2.15/32 -m physdev --physdev-in vif1.0 -j ACCEPT
-A FORWARD -d 172.16.2.15/32 -m physdev --physdev-out vif1.0 -j ACCEPT
-A FORWARD -s 172.16.2.14/32 -m physdev --physdev-in vif1.0 -j ACCEPT
-A FORWARD -d 172.16.2.14/32 -m physdev --physdev-out vif1.0 -j ACCEPT
-A FORWARD -s 172.16.2.22/32 -m physdev --physdev-in vif1.0 -j ACCEPT
-A FORWARD -d 172.16.2.22/32 -m physdev --physdev-out vif1.0 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -m physdev --physdev-out vif3.0 -j ACCEPT
-A FORWARD -p udp -m physdev --physdev-in vif3.0 -m udp --sport 68 --dport 67 -j ACCEPT
-A FORWARD -s 172.16.2.29/32 -m physdev --physdev-in vif3.0 -j ACCEPT
-A FORWARD -d 172.16.2.29/32 -m physdev --physdev-out vif3.0 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -m physdev --physdev-out vif2.0 -j ACCEPT
-A FORWARD -p udp -m physdev --physdev-in vif2.0 -m udp --sport 68 --dport 67 -j ACCEPT
-A FORWARD -s 172.16.2.10/32 -m physdev --physdev-in vif2.0 -j ACCEPT
-A FORWARD -d 172.16.2.10/32 -m physdev --physdev-out vif2.0 -j ACCEPT
-A FORWARD -p udp -m physdev --physdev-in vif4.0 -m udp --sport 68 --dport 67 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -m physdev --physdev-out vif4.0 -j ACCEPT
-A FORWARD -s 172.16.2.36/32 -m physdev --physdev-in vif4.0 -j ACCEPT
-A FORWARD -d 172.16.2.36/32 -m physdev --physdev-out vif4.0 -j ACCEPT
## If it's not an IP address allowed on that vif, log and drop it.
-A FORWARD -m limit --limit 15/min -j LOG --log-prefix "Dropped by firewall: " --log-level 7
-A FORWARD -j DROP
#
## Here are the rules which protect the dom0
-A INPUT -i lo -j ACCEPT 
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 
#
## Allow ICMP packets necessary for MTU path discovery
-A INPUT -p icmp --icmp-type fragmentation-needed -j ACCEPT
## Allow echo request
-A INPUT -p icmp --icmp-type 8 -j ACCEPT

## No connections to the gateway address for /24 subnet
-A INPUT -i eth0 -d 172.16.2.1 -j DROP
#
## Allow everything to the domUs so they can make their own decision
-A INPUT -i eth0 -d 172.16.2.0/24 -j ACCEPT
#
## Allow SSH for management
-A INPUT -i eth0 -d 172.16.0.10 -p tcp -m tcp --dport 22 -j ACCEPT 
#
## No connections from domUs to dom0 gateway address
-A INPUT -i dummy0 -d 172.16.2.1 -j DROP
#
## But allow domU to get to everything else
-A INPUT -i dummy0 -s 172.16.2.0/24 -j ACCEPT
#
## Otherwise, drop it
-A INPUT -j DROP 
COMMIT

As before, apply with iptables-restore

# iptables-restore < /etc/iptables.restore

Now don't take my word for it, nmap your dom0 from all angles!