Setting up networking
Install openvswitch on both nodes (master and minion), good instruction for Centos 7 can be found here: openvswitch on centos 7
Assuming master node has IP: 159.107.152.121 and minion node has 159.107.152.161, and both can reach each other over these addresses.
Master node:
Edit /etc/sysconfig/docker-network and add option "--iptables=false --ip-masq=false --mtu=1400"
yum -y update && yum -y install net-tools bridge-utils setenforce 0 service firewalld stop service docker stop ifconfig docker0 down brctl delbr docker0 brctl addbr docker0 ifconfig docker0 10.244.1.1/16 mtu 1400 up service docker start ovs-vsctl add-br br0 ifconfig br0 mtu 1400 ovs-vsctl add-port br0 vx0 -- set interface vx0 \ type=vxlan options:remote_ip=159.107.152.161 ifconfig br0 up brctl addif docker0 br0 route add -net 10.244.0.0/16 dev docker0
Minion node:
Edit /etc/sysconfig/docker-network and add option "--iptables=false --ip-masq=false --mtu=1400"setenforce 0 service firewalld stop service docker stop ifconfig docker0 down brctl delbr docker0 brctl addbr docker0 ifconfig docker0 10.244.2.1/16 mtu 1400 up service docker start ovs-vsctl add-br br0 ifconfig br0 mtu 1400 ovs-vsctl add-port br0 vx0 -- set interface vx0 \ type=vxlan options:remote_ip=159.107.152.121 ifconfig br0 up brctl addif docker0 br0 route add -net 10.244.0.0/16 dev docker0
Starting kubernetes on master node
On master node, run as root:DOCKERIP="10.244.1.1"
service docker stop
docker -d -H \
unix:///var/run/docker-bootstrap.sock -p /var/run/docker-bootstrap.pid \
--iptables=false --ip-masq=false --bridge=none \
--graph=/var/lib/docker-bootstrap 2> \
/var/log/docker-bootstrap.log 1> /dev/null &
docker -H \
unix:///var/run/docker-bootstrap.sock run --net=host \
-d gcr.io/google_containers/etcd:2.0.9 /usr/local/bin/etcd \
--addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 \
--data-dir=/var/etcd/data
docker -H \
unix:///var/run/docker-bootstrap.sock run \
--net=host gcr.io/google_containers/etcd:2.0.9 \
etcdctl set /coreos.com/network/config \
'{ "Network": "10.244.0.0/16" }'
service docker start
docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock \
gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube kubelet \
--api_servers=http://localhost:8080 --v=2 --address=$DOCKERIP \
--enable_server --hostname_override=$DOCKERIP \
--config=/etc/kubernetes/manifests-multi
docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v1.0.1 \
/hyperkube proxy --master=http://127.0.0.1:8080 --v=2
Verify master node is up and running
Download kubectl from here Verify with ./kubectl get nodes should show only one:[root@localhost ~]# ./kubectl -s 10.244.1.1:8080 get nodes NAME LABELS STATUS 10.244.1.1 kubernetes.io/hostname=10.244.1.1 Ready
Setting up worker node(s)
Now, on worker nodes do:MASTER_DOCKER_IP=10.244.1.1 WORKER_DOCKER_IP=10.244.2.1 docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock \ gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube kubelet \ --api_servers=http://$MASTER_DOCKER_IP:8080 --v=2 \ --address=$WORKER_DOCKER_IP --enable_server \ --hostname_override=$WORKER_DOCKER_IP #and service proxy: docker run -d --net=host --privileged \ gcr.io/google_containers/hyperkube:v1.0.1 \ /hyperkube proxy --master=http://$MASTER_DOCKER_IP:8080 --v=2
Verify master and minion working
Now we should have two nodes, verified from master:[root@localhost ~]# ./kubectl get nodes NAME LABELS STATUS 10.244.1.1 kubernetes.io/hostname=10.244.1.1 Ready 10.244.2.1 kubernetes.io/hostname=10.244.2.1 Readyand verified from slave (note -s $MASTER_IP):
[root@localhost ~]# ./kubectl -s 10.244.1.1:8080 get nodes NAME LABELS STATUS 10.244.1.1 kubernetes.io/hostname=10.244.1.1 Ready 10.244.2.1 kubernetes.io/hostname=10.244.2.1 Ready
Create simple application using nginx
Run application, from master:./kubectl -s http://localhost:8080 run nginx --image=nginx --port=80 ./kubectl expose rc nginx --port=80 ###few seconds later: ./kubectl get pods ###shows pods running with ip**
Scale application up and down:
./kubectl scale rc nginx --replicas=3 #after few seconds ./kubectl get pods will show new pods #and downscale ./kubectl scale rc nginx --replicas=1Deleting nginx we created above:
./kubectl delete rc nginx ./kubectl delete svc nginx
Creating application and service using yaml files
[root@localhost ~]# cat nginxrc.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: my-nginx
spec:
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
##and run with:
./kubectl -s $MASTER_DOCKER_IP:8080 create -f nginxrc.yaml
##then we can create service as well:
[root@localhost ~]# cat nginxsvc.yaml
apiVersion: v1
kind: Service
metadata:
name: nginxsvc
labels:
app: nginx
spec:
ports:
- port: 80
protocol: TCP
selector:
app: nginx
##and run it with:
./kubectl -s $MASTER_DOCKER_IP:8080 create -f nginxsvc.yaml
##after a while results can be seen with:
./kubectl describe svc nginxsvc
Adding DNS support
I will be setting up dns service on 10.0.0.10 for dev.deki.local domain. This has three steps, creation of skydns service, creation of skydns rc and restarting our master and minions to use dns, since example above was without DNS add on.
Create skydns-svc.yaml as shown bellow:
Create skydns-svc.yaml as shown bellow:
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "KubeDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: "10.0.0.10"
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
and skydns-rc.yaml as shown bellow (make sure to replace $MASTER_IP with IP address of docker on your master node, in my example 10.244.1.1):
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-dns-v8
namespace: kube-system
labels:
k8s-app: kube-dns
version: v8
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
k8s-app: kube-dns
version: v8
template:
metadata:
labels:
k8s-app: kube-dns
version: v8
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: etcd
image: gcr.io/google_containers/etcd:2.0.9
resources:
limits:
cpu: 100m
memory: 50Mi
command:
- /usr/local/bin/etcd
- -data-dir
- /var/etcd/data
- -listen-client-urls
- http://127.0.0.1:2379,http://127.0.0.1:4001
- -advertise-client-urls
- http://127.0.0.1:2379,http://127.0.0.1:4001
- -initial-cluster-token
- skydns-etcd
volumeMounts:
- name: etcd-storage
mountPath: /var/etcd/data
- name: kube2sky
image: gcr.io/google_containers/kube2sky:1.11
resources:
limits:
cpu: 100m
memory: 50Mi
args:
# command = "/kube2sky"
- -kube_master_url=http://$MASTER_IP:8080
- -domain=dev.deki.local
- name: skydns
image: gcr.io/google_containers/skydns:2015-03-11-001
resources:
limits:
cpu: 100m
memory: 50Mi
args:
# command = "/skydns"
- -machines=http://localhost:4001
- -addr=0.0.0.0:53
- -domain=dev.deki.local
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
- name: healthz
image: gcr.io/google_containers/exechealthz:1.0
resources:
limits:
cpu: 10m
memory: 20Mi
args:
- -cmd=nslookup kubernetes.default.svc.dev.deki.local localhost >/dev/null
- -port=8080
ports:
- containerPort: 8080
protocol: TCP
volumes:
- name: etcd-storage
emptyDir: {}
dnsPolicy: Default # Don't use cluster DNS.
So now, lets kill master and minions and start them up again:
#Run this on master and all minions:
docker ps -a | egrep -v "STATUS" | \
awk '{print $1}' | xargs docker kill
docker ps -a | grep "Exited" | \
awk '{print $1}' | xargs docker rm -v
This will clear master and minions, so lets start them up again, this time using dns domain and dns ip:
###On master#### docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock \ gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube kubelet \ --api_servers=http://localhost:8080 --v=2 --address=$DOCKERIP \ --enable_server --cluster-dns=10.0.0.10 \ --cluster-domain=dev.deki.local \ --hostname_override=$DOCKERIP \ --config=/etc/kubernetes/manifests-multi docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v1.0.1 \ /hyperkube proxy --master=http://127.0.0.1:8080 --v=2 ####And on minion#### MASTER_DOCKER_IP=10.244.1.1 WORKER_DOCKER_IP=10.244.2.1 docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock \ gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube kubelet \ --api_servers=http://$MASTER_DOCKER_IP:8080 --v=2 \ --address=$WORKER_DOCKER_IP --enable_server \ --cluster-dns=10.0.0.10 \ --cluster-domain=dev.deki.local \ --hostname_override=$WORKER_DOCKER_IP #and service proxy on minion: docker run -d --net=host --privileged \ gcr.io/google_containers/hyperkube:v1.0.1 \ /hyperkube proxy --master=http://$MASTER_DOCKER_IP:8080 --v=2 ### Finally start dns ### ./kubectl create -f skydns-svc.yaml ./kubectl create -f skydns-rc.yamlFew moments later dns should be up and running. Using example of nginx service above, we can verify it's working by doing:
dig @10.0.0.10 nginxsvc.default.svc.dev.deki.local
