Google kubernetes test drive

Setting up networking

Install openvswitch on both nodes (master and minion), good instruction for Centos 7 can be found here: openvswitch on centos 7

Assuming master node has IP:  159.107.152.121 and minion node has 159.107.152.161, and both can reach each other over these addresses.

Master node:

Edit /etc/sysconfig/docker-network and add option "--iptables=false --ip-masq=false --mtu=1400"

yum -y update && yum -y install net-tools bridge-utils
setenforce 0
service firewalld stop
service docker stop
ifconfig docker0 down
brctl delbr docker0
brctl addbr docker0
ifconfig docker0 10.244.1.1/16 mtu 1400 up
service docker start
ovs-vsctl add-br br0
ifconfig br0 mtu 1400
ovs-vsctl add-port br0 vx0 -- set interface vx0 \
type=vxlan options:remote_ip=159.107.152.161
ifconfig br0 up
brctl addif docker0 br0
route add -net 10.244.0.0/16 dev docker0

Minion node: 

Edit /etc/sysconfig/docker-network and add option "--iptables=false --ip-masq=false --mtu=1400"

setenforce 0
service firewalld stop
service docker stop
ifconfig docker0 down
brctl delbr docker0
brctl addbr docker0
ifconfig docker0 10.244.2.1/16 mtu 1400 up
service docker start
ovs-vsctl add-br br0
ifconfig br0 mtu 1400
ovs-vsctl add-port br0 vx0 -- set interface vx0 \
type=vxlan options:remote_ip=159.107.152.121
ifconfig br0 up
brctl addif docker0 br0
route add -net 10.244.0.0/16 dev docker0

Starting kubernetes on master node

On master node, run as root:

DOCKERIP="10.244.1.1"
service docker stop
docker -d -H \
unix:///var/run/docker-bootstrap.sock -p /var/run/docker-bootstrap.pid \
--iptables=false --ip-masq=false --bridge=none \
--graph=/var/lib/docker-bootstrap 2> \
/var/log/docker-bootstrap.log 1> /dev/null &

docker -H \
unix:///var/run/docker-bootstrap.sock run --net=host \
-d gcr.io/google_containers/etcd:2.0.9 /usr/local/bin/etcd \
--addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 \
--data-dir=/var/etcd/data

docker -H \
unix:///var/run/docker-bootstrap.sock run \
--net=host gcr.io/google_containers/etcd:2.0.9 \
etcdctl set /coreos.com/network/config \
'{ "Network": "10.244.0.0/16" }'

service docker start
docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock  \
gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube kubelet \
--api_servers=http://localhost:8080 --v=2 --address=$DOCKERIP \
--enable_server --hostname_override=$DOCKERIP \
--config=/etc/kubernetes/manifests-multi

docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v1.0.1 \
/hyperkube proxy --master=http://127.0.0.1:8080 --v=2

Verify master node is up and running

Download kubectl from here Verify with ./kubectl get nodes should show only one:

[root@localhost ~]# ./kubectl -s 10.244.1.1:8080 get nodes
NAME         LABELS                              STATUS
10.244.1.1   kubernetes.io/hostname=10.244.1.1   Ready

Setting up worker node(s)

Now, on worker nodes do:

MASTER_DOCKER_IP=10.244.1.1
WORKER_DOCKER_IP=10.244.2.1
docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock \
gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube kubelet \
--api_servers=http://$MASTER_DOCKER_IP:8080 --v=2 \
--address=$WORKER_DOCKER_IP --enable_server \
--hostname_override=$WORKER_DOCKER_IP

#and service proxy:
docker run -d --net=host --privileged \
gcr.io/google_containers/hyperkube:v1.0.1 \
/hyperkube proxy --master=http://$MASTER_DOCKER_IP:8080 --v=2

Verify master and minion working

Now we should have two nodes, verified from master:

[root@localhost ~]# ./kubectl  get nodes
NAME         LABELS                              STATUS
10.244.1.1   kubernetes.io/hostname=10.244.1.1   Ready
10.244.2.1   kubernetes.io/hostname=10.244.2.1   Ready

and verified from slave (note -s $MASTER_IP):

[root@localhost ~]# ./kubectl -s 10.244.1.1:8080 get nodes
NAME         LABELS                              STATUS
10.244.1.1   kubernetes.io/hostname=10.244.1.1   Ready
10.244.2.1   kubernetes.io/hostname=10.244.2.1   Ready

Create simple application using nginx

Run application, from master:

./kubectl -s http://localhost:8080 run nginx --image=nginx --port=80
./kubectl expose rc nginx --port=80
###few seconds later:
./kubectl get pods
###shows pods running with ip**

Scale application up and down:

./kubectl scale rc nginx --replicas=3
#after few seconds
./kubectl get pods will show new pods
#and downscale 
./kubectl scale rc nginx --replicas=1

Deleting nginx we created above:

./kubectl delete rc nginx
./kubectl delete svc nginx

Creating application and service using yaml files

[root@localhost ~]# cat nginxrc.yaml 
apiVersion: v1
kind: ReplicationController
metadata:
  name: my-nginx
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

##and run with: 
./kubectl -s $MASTER_DOCKER_IP:8080 create -f nginxrc.yaml
##then we can create service as well:
[root@localhost ~]# cat nginxsvc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: nginxsvc
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: nginx

##and run it with:
./kubectl -s $MASTER_DOCKER_IP:8080 create -f nginxsvc.yaml
##after a while results can be seen with:
./kubectl describe svc nginxsvc

Adding DNS support

I will be setting up dns service on 10.0.0.10 for dev.deki.local domain. This has three steps, creation of skydns service, creation of skydns rc and restarting our master and minions to use dns, since example above was without DNS add on.
Create skydns-svc.yaml as shown bellow:

apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP:  "10.0.0.10"
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP

and skydns-rc.yaml as shown bellow (make sure to replace $MASTER_IP with IP address of docker on your master node, in my example 10.244.1.1):

apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-dns-v8
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    version: v8
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-dns
    version: v8
  template:
    metadata:
      labels:
        k8s-app: kube-dns
        version: v8
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: etcd
        image: gcr.io/google_containers/etcd:2.0.9
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        command:
        - /usr/local/bin/etcd
        - -data-dir
        - /var/etcd/data
        - -listen-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -advertise-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -initial-cluster-token
        - skydns-etcd
        volumeMounts:
        - name: etcd-storage
          mountPath: /var/etcd/data
      - name: kube2sky
        image: gcr.io/google_containers/kube2sky:1.11
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/kube2sky"
        - -kube_master_url=http://$MASTER_IP:8080
        - -domain=dev.deki.local
      - name: skydns
        image: gcr.io/google_containers/skydns:2015-03-11-001
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/skydns"
        - -machines=http://localhost:4001
        - -addr=0.0.0.0:53
        - -domain=dev.deki.local
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
      - name: healthz
        image: gcr.io/google_containers/exechealthz:1.0
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
        args:
        - -cmd=nslookup kubernetes.default.svc.dev.deki.local localhost >/dev/null
        - -port=8080
        ports:
        - containerPort: 8080
          protocol: TCP
      volumes:
      - name: etcd-storage
        emptyDir: {}
      dnsPolicy: Default  # Don't use cluster DNS.

So now, lets kill master and minions and start them up again:

#Run this on master and all minions:
docker ps -a | egrep -v "STATUS" | \
awk '{print $1}' | xargs docker kill 
docker ps -a | grep "Exited" | \
awk '{print $1}' | xargs docker rm -v

This will clear master and minions, so lets start them up again, this time using dns domain and dns ip:

###On master####
docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock  \
gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube kubelet \
--api_servers=http://localhost:8080 --v=2 --address=$DOCKERIP \
--enable_server --cluster-dns=10.0.0.10 \
--cluster-domain=dev.deki.local \
--hostname_override=$DOCKERIP \
--config=/etc/kubernetes/manifests-multi

docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v1.0.1 \
/hyperkube proxy --master=http://127.0.0.1:8080 --v=2

####And on minion####
MASTER_DOCKER_IP=10.244.1.1
WORKER_DOCKER_IP=10.244.2.1
docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock \
gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube kubelet \
--api_servers=http://$MASTER_DOCKER_IP:8080 --v=2 \
--address=$WORKER_DOCKER_IP --enable_server \
--cluster-dns=10.0.0.10 \
--cluster-domain=dev.deki.local \
--hostname_override=$WORKER_DOCKER_IP

#and service proxy on minion:
docker run -d --net=host --privileged \
gcr.io/google_containers/hyperkube:v1.0.1 \
/hyperkube proxy --master=http://$MASTER_DOCKER_IP:8080 --v=2

### Finally start dns ###
./kubectl create -f skydns-svc.yaml
./kubectl create -f skydns-rc.yaml

Few moments later dns should be up and running. Using example of nginx service above, we can verify it's working by doing:

dig @10.0.0.10 nginxsvc.default.svc.dev.deki.local