Kubernetes Dev-Stack

Background

Small proof of concept for running kubernetes cluster, specifically intended for development environment. Can create kubernetes cluster compromised of one master and arbitrary number of minions. Can run on Linux, Windows or Mac. Vagrant box is based on Centos 7.1 with latest stable kernel 4.3.0, docker 1.8.2 using overlay storage driver, backed by xfs file system, kubernetes is at the latest version 1.1.1. SELinux will be set to permissive mode, and firewall will be down.

This is little demo was created for the purpose of development, and should not be used in production, as kubernetes is configured without security.

Source can be found here: https://github.com/dekstroza/kubernetes-dev-stack

What's inside the tin can

• Centos 7.1 kernel 4.3.0, xfs
• Docker 1.8.2, overlay storage driver
• Kubernetes 1.1.1 with cluster-addons
• Saltstack 2015.5.5-1

Requirements

1. VirtualBox, recent version, you can get it here: Oracle Virtual box
2. Vagrant, recent version, you can get it from here: Vagrant

Getting started

In order to get started, first we need to start kube master, after which we can start multiple minions anywhere on the network as long as they can reach master. So clone the git repo above and go into vagrant/kube-master directory.

Starting kube master

To start kubernetes master (which will also be used to schedule docker containers), do:

cd vagrant/kube-master
## By default vagrant box will use 4G of RAM,
## to use less or more, export env variable MEM_SIZE,
## example: export MEM_SIZE=8096 for 8Gig VM

## Start the VM
vagrant up --provider=virtualbox

After initial download of vagrant box (once off download, 450Mb) from vagrant repository, box will be automatically configured, and depending on network setup on your machine, it might ask you which network interface you wish to use - normally choose one you use to connect to Internet (normally choice #1 is what you need), but can vary depending on the machine.

Since kubernetes operates on separate network, script to create route to your newly created kubernetes cloud will be generated in the same dir (for Windows, Linux and Mac), so run:

## Depending on your os, for example Linux:

./add-route-LIN.sh

## Which will create route
## using your VM as gateway.

You can now ssh into your kubernetes master with:

  vagrant ssh

Kubernetes master should be up and running for you:

## Bellow will show you all kube memebers
kubectl get nodes

## Bellow will show everything that currently Requirements
## in kube-system namespace (dns, ui, grafana etc..)
kubectl get po --namespace=kube-system

## Gives you cluster info, all cluster services running
kubectl cluster-info

## You can start kube-ui or grafana as example:
sudo kubectl create -f /etc/kubernetes/kube-ui/

## Or Graphana:
sudo kubectl create -f /etc/kubernetes/grafana/

## And monitor progress with:
watch -n 5 kubectl get po --namespace=kube-system

## Once up and running cluster-info will tell you where to go:

kubectl cluster-info

## and open up Grafana url shown in your browser.

Also there will be dns up and running - depending how fast your network is, it might take a few for docker to pull required images. DNS server will be at 10.0.0.10 and serve domain dekstroza.local

To verify dns is up and running, inside master or minions, run:

dig @10.0.0.10 kuberenetes.default.svc.dekstroza.local

If you have added route as described above, dns will be reachable not only from inside the VM, but also from your host OS. You can find configuration for it in salt/pillar/kube-global.sls and set different cluster CIDR, service CIDR, DNS domain or DNS IP address. After changing any of these, running salt-stack can reconfigure your already running VM, but I would recommend to restart your VMs (master and minions). Bellow is current content:

## cat kube-global.sls:
service_cluster_cidr: 10.0.0.0/16
kube_cluster_cidr: 10.244.0.0/16
dns_replicas: 1
dns_server: 10.0.0.10
dns_domain: dekstroza.local
cluster_registry_disk_size: 1G

Important bits are:

• service_cluster_cidr : Range from which kuberentes nodes will get address for internal communication
• kube_cluster_cidr : Range from which services in kubernetes will get address
• dns_replicas : Number of DNS server replicas
• dns_server : IP that will be assigned to DNS server
• dns_domain : Chosen DNS domain
• cluster_registry_disk_size : Internal docker registry disk size

Note cluster_registry_disk_size is not used and has not been tested

Starting kube minion(s)

Change directory to kube-minion:

cd kubernetes-dev-stack/vagrant/kube-minion
## Set the MASTER_IP to point to your kubeernetes master

export MASTER_IP= ip address of your master

## Set MEM_SIZE if you wish more or less then 4Gig for ## minion(s)
## Set NUM_MINIONS=n, where n is number of minions you wish to ## start

Vagrant will start up your minions and salt-stack will configure them correctly. Again, depending on your network setup, you might be asked to select network interface over which minions will communicate (normally one you use to access Internet, normally choice #1).

Master and minions on separate machines

Since master and minions will be bridged to your host interface they can be on different hosts, only thing required is for the minions to export MASTER_IP as shown above.

How it works

Packer template provided in the repo is used to create vagrant box, in case you wish to create your own. Code here will use one I have already created and deployed to vagrant repository.

Salt-stack is used to configure VM upon startup, you can find configuration in salt directory.

Adding files into running master or minion

Vagrant will mount directory where Vagrantfile is located inside the VM, under /vagrant path. You can use this to add more files into the box, ie pass in docker images instead of downloading them.

Happy hacking.... Dejan

Google kubernetes test drive

Setting up networking

Install openvswitch on both nodes (master and minion), good instruction for Centos 7 can be found here: openvswitch on centos 7

Assuming master node has IP:  159.107.152.121 and minion node has 159.107.152.161, and both can reach each other over these addresses.

Master node:

Edit /etc/sysconfig/docker-network and add option "--iptables=false --ip-masq=false --mtu=1400"

yum -y update && yum -y install net-tools bridge-utils
setenforce 0
service firewalld stop
service docker stop
ifconfig docker0 down
brctl delbr docker0
brctl addbr docker0
ifconfig docker0 10.244.1.1/16 mtu 1400 up
service docker start
ovs-vsctl add-br br0
ifconfig br0 mtu 1400
ovs-vsctl add-port br0 vx0 -- set interface vx0 \
type=vxlan options:remote_ip=159.107.152.161
ifconfig br0 up
brctl addif docker0 br0
route add -net 10.244.0.0/16 dev docker0

Minion node: 

Edit /etc/sysconfig/docker-network and add option "--iptables=false --ip-masq=false --mtu=1400"

setenforce 0
service firewalld stop
service docker stop
ifconfig docker0 down
brctl delbr docker0
brctl addbr docker0
ifconfig docker0 10.244.2.1/16 mtu 1400 up
service docker start
ovs-vsctl add-br br0
ifconfig br0 mtu 1400
ovs-vsctl add-port br0 vx0 -- set interface vx0 \
type=vxlan options:remote_ip=159.107.152.121
ifconfig br0 up
brctl addif docker0 br0
route add -net 10.244.0.0/16 dev docker0

Starting kubernetes on master node

On master node, run as root:

DOCKERIP="10.244.1.1"
service docker stop
docker -d -H \
unix:///var/run/docker-bootstrap.sock -p /var/run/docker-bootstrap.pid \
--iptables=false --ip-masq=false --bridge=none \
--graph=/var/lib/docker-bootstrap 2> \
/var/log/docker-bootstrap.log 1> /dev/null &

docker -H \
unix:///var/run/docker-bootstrap.sock run --net=host \
-d gcr.io/google_containers/etcd:2.0.9 /usr/local/bin/etcd \
--addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 \
--data-dir=/var/etcd/data

docker -H \
unix:///var/run/docker-bootstrap.sock run \
--net=host gcr.io/google_containers/etcd:2.0.9 \
etcdctl set /coreos.com/network/config \
'{ "Network": "10.244.0.0/16" }'

service docker start
docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock  \
gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube kubelet \
--api_servers=http://localhost:8080 --v=2 --address=$DOCKERIP \
--enable_server --hostname_override=$DOCKERIP \
--config=/etc/kubernetes/manifests-multi

docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v1.0.1 \
/hyperkube proxy --master=http://127.0.0.1:8080 --v=2

Verify master node is up and running

Download kubectl from here Verify with ./kubectl get nodes should show only one:

[root@localhost ~]# ./kubectl -s 10.244.1.1:8080 get nodes
NAME         LABELS                              STATUS
10.244.1.1   kubernetes.io/hostname=10.244.1.1   Ready

Setting up worker node(s)

Now, on worker nodes do:

MASTER_DOCKER_IP=10.244.1.1
WORKER_DOCKER_IP=10.244.2.1
docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock \
gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube kubelet \
--api_servers=http://$MASTER_DOCKER_IP:8080 --v=2 \
--address=$WORKER_DOCKER_IP --enable_server \
--hostname_override=$WORKER_DOCKER_IP

#and service proxy:
docker run -d --net=host --privileged \
gcr.io/google_containers/hyperkube:v1.0.1 \
/hyperkube proxy --master=http://$MASTER_DOCKER_IP:8080 --v=2

Verify master and minion working

Now we should have two nodes, verified from master:

[root@localhost ~]# ./kubectl  get nodes
NAME         LABELS                              STATUS
10.244.1.1   kubernetes.io/hostname=10.244.1.1   Ready
10.244.2.1   kubernetes.io/hostname=10.244.2.1   Ready

and verified from slave (note -s $MASTER_IP):

[root@localhost ~]# ./kubectl -s 10.244.1.1:8080 get nodes
NAME         LABELS                              STATUS
10.244.1.1   kubernetes.io/hostname=10.244.1.1   Ready
10.244.2.1   kubernetes.io/hostname=10.244.2.1   Ready

Create simple application using nginx

Run application, from master:

./kubectl -s http://localhost:8080 run nginx --image=nginx --port=80
./kubectl expose rc nginx --port=80
###few seconds later:
./kubectl get pods
###shows pods running with ip**

Scale application up and down:

./kubectl scale rc nginx --replicas=3
#after few seconds
./kubectl get pods will show new pods
#and downscale 
./kubectl scale rc nginx --replicas=1

Deleting nginx we created above:

./kubectl delete rc nginx
./kubectl delete svc nginx

Creating application and service using yaml files

[root@localhost ~]# cat nginxrc.yaml 
apiVersion: v1
kind: ReplicationController
metadata:
  name: my-nginx
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

##and run with: 
./kubectl -s $MASTER_DOCKER_IP:8080 create -f nginxrc.yaml
##then we can create service as well:
[root@localhost ~]# cat nginxsvc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: nginxsvc
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: nginx

##and run it with:
./kubectl -s $MASTER_DOCKER_IP:8080 create -f nginxsvc.yaml
##after a while results can be seen with:
./kubectl describe svc nginxsvc

Adding DNS support

I will be setting up dns service on 10.0.0.10 for dev.deki.local domain. This has three steps, creation of skydns service, creation of skydns rc and restarting our master and minions to use dns, since example above was without DNS add on.
Create skydns-svc.yaml as shown bellow:

apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP:  "10.0.0.10"
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP

and skydns-rc.yaml as shown bellow (make sure to replace $MASTER_IP with IP address of docker on your master node, in my example 10.244.1.1):

apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-dns-v8
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    version: v8
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-dns
    version: v8
  template:
    metadata:
      labels:
        k8s-app: kube-dns
        version: v8
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: etcd
        image: gcr.io/google_containers/etcd:2.0.9
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        command:
        - /usr/local/bin/etcd
        - -data-dir
        - /var/etcd/data
        - -listen-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -advertise-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -initial-cluster-token
        - skydns-etcd
        volumeMounts:
        - name: etcd-storage
          mountPath: /var/etcd/data
      - name: kube2sky
        image: gcr.io/google_containers/kube2sky:1.11
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/kube2sky"
        - -kube_master_url=http://$MASTER_IP:8080
        - -domain=dev.deki.local
      - name: skydns
        image: gcr.io/google_containers/skydns:2015-03-11-001
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/skydns"
        - -machines=http://localhost:4001
        - -addr=0.0.0.0:53
        - -domain=dev.deki.local
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
      - name: healthz
        image: gcr.io/google_containers/exechealthz:1.0
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
        args:
        - -cmd=nslookup kubernetes.default.svc.dev.deki.local localhost >/dev/null
        - -port=8080
        ports:
        - containerPort: 8080
          protocol: TCP
      volumes:
      - name: etcd-storage
        emptyDir: {}
      dnsPolicy: Default  # Don't use cluster DNS.

So now, lets kill master and minions and start them up again:

#Run this on master and all minions:
docker ps -a | egrep -v "STATUS" | \
awk '{print $1}' | xargs docker kill 
docker ps -a | grep "Exited" | \
awk '{print $1}' | xargs docker rm -v

This will clear master and minions, so lets start them up again, this time using dns domain and dns ip:

###On master####
docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock  \
gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube kubelet \
--api_servers=http://localhost:8080 --v=2 --address=$DOCKERIP \
--enable_server --cluster-dns=10.0.0.10 \
--cluster-domain=dev.deki.local \
--hostname_override=$DOCKERIP \
--config=/etc/kubernetes/manifests-multi

docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v1.0.1 \
/hyperkube proxy --master=http://127.0.0.1:8080 --v=2

####And on minion####
MASTER_DOCKER_IP=10.244.1.1
WORKER_DOCKER_IP=10.244.2.1
docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock \
gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube kubelet \
--api_servers=http://$MASTER_DOCKER_IP:8080 --v=2 \
--address=$WORKER_DOCKER_IP --enable_server \
--cluster-dns=10.0.0.10 \
--cluster-domain=dev.deki.local \
--hostname_override=$WORKER_DOCKER_IP

#and service proxy on minion:
docker run -d --net=host --privileged \
gcr.io/google_containers/hyperkube:v1.0.1 \
/hyperkube proxy --master=http://$MASTER_DOCKER_IP:8080 --v=2

### Finally start dns ###
./kubectl create -f skydns-svc.yaml
./kubectl create -f skydns-rc.yaml

Few moments later dns should be up and running. Using example of nginx service above, we can verify it's working by doing:

dig @10.0.0.10 nginxsvc.default.svc.dev.deki.local

Openstack kilo playing nicely with docker

This has been tested on Centos 7, Ubuntu should be absolutely same thing. Start up virtual machine, I am running Centos 7 (6 cores + 8Gigs of RAM assinged) through parallels on mac, smaller VM will work too.
To make things simpler, I will disable selinux on Centos 7 and make Dan Walsh cry, so either run setenforce 0, or edit /etc/sysconfig/selinux (set to disabled and reboot).

#Update system before we start
sudo yum install epel-release
sudo yum -y update
sudo yum -y install python-pip python-devel git openssh-server
sudo yum groupinstall "Development Tools"
sudo setenforce 0
sudo mkdir /opt/stack && chown deki:deki /opt/stack/ -R
git clone https://git.openstack.org/stackforge/nova-docker /opt/stack/nova-docker
git clone https://git.openstack.org/openstack-dev/devstack /opt/stack/devstack
#Switch to kilo stable branch
cd /opt/stack/nova-docker && git checkout -b kilo origin/stable/kilo
cd /opt/stack/devstack && git checkout -b kilo origin/stable/kilo
cd /opt/stack/nova-docker
./contrib/devstack/prepare_devstack.sh

Now the important bit, as stack.sh will not work without it:
Edit file: /opt/stack/nova-docker/novadocker/virt/docker/driver.py
Change:
from nova.openstack.common import fileutils to from oslo_utils import fileutils
To simplify further, I will not make any other modifications to devstack, so

cd /opt/stack/devstack 
#Run stack.sh and have a cup of coffee
./stack.sh

It should complete and show something like:

##This is your host IP address: 10.211.55.34
##2015-08-05 09:40:07.103 | Skip setting lvm filters for non Ubuntu systems
##This is your host IPv6 address: ::1
##Horizon is now available at http://10.211.55.34/
##Keystone is serving at http://10.211.55.34:5000/
##The default users are: admin and demo
##The password: mysecretpassword
##2015-08-05 09:40:07.108 | stack.sh completed in 278 seconds.

In my case I had to restart firewall, not sure yet why, so do:

sudo service firewalld status
#
#If it shows:
#Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled)
#   Active: inactive (dead)
#run:
sudo service firewalld start

So lets verify all is well:

docker version
# Lets try it out:
. openrc admin
docker pull larsks/thttpd
docker save larsks/thttpd | glance image-create \
--name larsks/thttpd \
--is-public true \
--container-format docker \
--disk-format raw

. openrc demo
nova boot --image larsks/thttpd \
--flavor m1.small test0

#Wait a bit and nova list will show:
[deki@localhost devstack]$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks         |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 83678e40-78f9-4d41-90d0-c8dbd75c54b9 | test0 | ACTIVE | -          | Running     | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

#then:
[deki@localhost devstack]$ nova floating-ip-create
+----+------------+-----------+----------+--------+
| Id | IP         | Server Id | Fixed IP | Pool   |
+----+------------+-----------+----------+--------+
| 1  | 172.24.4.1 | -         | -        | public |
+----+------------+-----------+----------+--------+

nova floating-ip-associate test0 172.24.4.1
#and finally try it out and behold:
curl http://172.24.4.1
#will return bellow:


             
  
    
 
 
  

  
  ____                            _         _       _   _                 
 / ___|___  _ __   __ _ _ __ __ _| |_ _   _| | __ _| |_(_) ___  _ __  ___ 
| |   / _ \| '_ \ / _` | '__/ _` | __| | | | |/ _` | __| |/ _ \| '_ \/ __|
| |__| (_) | | | | (_| | | | (_| | |_| |_| | | (_| | |_| | (_) | | | \__ \
 \____\___/|_| |_|\__, |_|  \__,_|\__|\__,_|_|\__,_|\__|_|\___/|_| |_|___/
                  |___/                                                   
  

  
You have a web server.

  
  

    
This is a statically compiled version of thttpd 
    put together to build a demonstration container for my
    Heat templates for Kubernetes.  But maybe 
    you'll find it useful for other things.
  
 

Multihost dns with skydns v1 and skydock

Skydns

Start skydns on one of the hosts, for example host1:

docker run -d --name skydns \
-p 53:53/udp \
-p 8080:8080/tcp \
crosbymichael/skydns:latest \
-nameserver="8.8.8.8:53" \
-domain="deki.local"

If running on Centos 7 use IP with port bind:

docker run -d --name skydns \
-p $HOSTIP:53:53/udp \
-p $HOSTIP:8080:8080/tcp \
crosbymichael/skydns:latest \
-nameserver="8.8.8.8:53" \
-domain="deki.local"

where $HOSTIP is the ip address of host1.

Skydock

Skydock will run on both hosts , host1 and host2. On run this on both hosts:

docker run -d -v /var/run/docker.sock:/docker.sock \
--name skydock crosbymichael/skydock \
-ttl 30 -environment dev -s /docker.sock \
-domain "deki.local" \
-skydns="http://$DNS_IP:8080/"

In this case since we run skydns on host1, $DNS_IP is ip address of host1. In this setup we have one DNS on host1, and skydock running on both. Each time container is started/stopped/destroyed skydock on that node will update DNS records for us. This setup can be further improved by running multiple DNS servers and multiple instances of skydock for higher availability.

Skydns2 and etcd playing nicely

Getting dns up and running

Setup:

I will be running docker on host with ip 10.37.129.6, docker bridge on 192.168.1.1/16, and setting up domain *.deki.local

On with it:

Start up by running etcd with something like this (10.37.129.6 is host address):

#!/bin/bash

HostIP="10.37.129.6"