Docker multi host networking with openvswitch

What's required for this?

For this little experiment we need:

1. Some basic network knowledge, or google skills
2. Two VM's with your favourite linux flavour, in my case it will be Ubuntu 14.04 and Centos 7.1
3. Working docker installation, something recent, so docker version > 1.6
4. Openvswitch installation on both virtual machines
5. Both VM's should be able to reach each other over some interface, however, they don't need to be on the same network.

Network diagram

I will be using openvswitch and vxlan to create connectivity between two hosts running docker containers. More information about openvswitch can be found here.

Let's get it up and running

Start by spinning up said VM's, and installing docker in both. I will be using latest version at the moment, which seems to be 1.7. To make thinks bit simpler, I will make Dan Walsh cry, and disable selinux on Centos 7, Ubuntu 14.04 comes without it anyways, and disable firewall on both, although later you can enable it and open required ports.So, lets start with Centos box, stop docker in case it is running, disable selinux and turn off firewall. I am assuming Centos VM can ping Ubuntu VM over eth0, and we will use eth0 to create vxlan.

Centos VM

First we take care of docker, selinux and firewall. This will disable selinux only temporarily, on reboot it would be enabled again. You can find here detailed instruction on how to install openvswitch on Centos 7.

sudo service docker stop
setenforce 0
sudo service firelwalld stop

I will assume that vm's can reach each other over eth0, you can set it up any way you like. First let's delete docker's default bridge docker0, and create new one:

ifconfig docker0 down
sudo brctl delbr docker0
sudo brctl addbr docker0
#lets bring it up and assign it address, set mtu to 1400
sudo ifconfig docker0 192.168.100.1/16 mtu 1400 up
#Let's create br0 for vxlan
sudo ovs-vsctl add-br br0
#Let's create vxlan using openvswitch, assuming VM2 has ip 159.107.166.230 
#and we can reach it from VM1 over eth0
sudo ovs-vsctl add-port br0 vx0 -- set interface vx0 type=vxlan options:remote_ip=159.107.166.230
#Bring it up and set mtu to 1400
sudo ifconfig br0 mtu 1400 up
#Add br0 as interface to docker0 bridge
sudo brctl addif docker0 br0
#Edit /etc/sysconfig/docker and set docker daemon option --mtu=1400
sudo service docker start

Ubuntu VM

Same process again, stop docker, firewall, remove default bridge and create it again, together with vxlan.

#Stop docker in case it's running
[deki@localhost ~]$ sudo service docker stop
#Stop ubuntu's firewall, just in case
[deki@localhost ~]$ sudo service ufw stop
#Bring down docker0 in case it's up
[deki@localhost ~]$ sudo ifconfig docker0 down
#Delete docker0 bridge
[deki@localhost ~]$ sudo brctl delbr docker0
#Create it again
[deki@localhost ~]$ sudo brctl addbr docker0
#Set docker0 ip and mtu, notice this one is in 192.168.200.x network
[deki@localhost ~]$ sudo ifconfig docker0 192.168.200.1/16 mtu 1400 up
[deki@localhost ~]$ sudo ovs-vsctl add-br br0
#Let's create vxlan using openvswitch, assuming VM1 has ip 159.107.133.130 and we can reach it from VM2 over eth0
[deki@localhost ~]$ sudo ovs-vsctl add-port br0 vx0 -- set interface vx0 type=vxlan options:remote_ip=159.107.133.130
#Bring it up and set mtu to 1400
[deki@localhost ~]$ sudo ifconfig br0 mtu 1400 up
#Add br0 as interface to docker0 bridge
[deki@localhost ~]$ sudo brctl addif docker0 br0
#Edit /etc/default/docker and set docker daemon option --mtu=1400
[deki@localhost ~]$ sudo service docker start

At this stage we have vxlan created between VM1 and VM2, and you can ping 192.168.100.1 and 192.168.200.1 from each other. Start docker container on each, for example docker run -it centos:6 and try to ping each other, or start wildfly in ha mode on both and see it clustered, fully working jgroups.