Upgrading Your Cluster

One of the things I noticed while studying for my CKAD exam is that my test cluster was a bit behind. It’s been up and running for over 200 days at this point and I’m several versions behind. Version 1.17 is out, the exam was based on 1.16, and I’m running 1.13.

NAME                         STATUS   ROLES    AGE    VERSION
runlevl41c.mylabserver.com   Ready    master   207d   v1.13.5
runlevl42c.mylabserver.com   Ready    <none>   207d   v1.13.5
runlevl43c.mylabserver.com   Ready    <none>   207d   v1.13.5

At this point, I haven’t updated anything. So let’s see what my options are with sudo kubeadm upgrade plan

I0112 00:57:07.499711   16540 version.go:237] remote version is much newer: v1.17.0; falling back to: stable-1.13
 
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
Kubelet     3 x v1.13.5   v1.13.12
 
Upgrade to the latest version in the v1.13 series:
 
COMPONENT            CURRENT   AVAILABLE
API Server           v1.13.7   v1.13.12
Controller Manager   v1.13.7   v1.13.12
Scheduler            v1.13.7   v1.13.12
Kube Proxy           v1.13.7   v1.13.12
CoreDNS              1.2.6     1.2.6
Etcd                 3.2.24    3.2.24

I’ve culled out the boring bits. The key takeaways are that I can see that I’m four versions behind and I can currently only upgrade to another 1.13 dot release. So let’s update kubeadm and see what happens.

Now for the downside. You really need to keep your cluster(s) current since you can’t jump multiple versions. Per the official documentation, you can only jump between single minor versions or patches. For example, in my case, I can only go to the latest patch version or from 1.13 to 1.14, but not from 1.13 to 1.17.

Let’s start by upgrading kubeadm. First we’ll find the latest version to upgrade to.

$ apt update
$ apt-cache policy kubeadm
...
     1.15.0-00 500
        500 https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.14.10-00 500
        500 https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
     1.14.9-00 500
        500 https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
...

So 1.14.10 is the latest 1.14 version.

sudo apt-mark unhold kubeadm kubelet && \
sudo apt-get update && sudo apt-get install -y kubeadm=1.14.10-00 && \
sudo apt-mark hold kubeadm

Now if we check our upgrade plan again, we’ll see 1.14 is available.

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
Kubelet     3 x v1.13.5   v1.14.10
 
Upgrade to the latest stable version:
 
COMPONENT            CURRENT   AVAILABLE
API Server           v1.13.7   v1.14.10
Controller Manager   v1.13.7   v1.14.10
Scheduler            v1.13.7   v1.14.10
Kube Proxy           v1.13.7   v1.14.10
CoreDNS              1.2.6     1.3.1
Etcd                 3.2.24    3.3.10

We’ll use the command provided by the plan. sudo kubeadm upgrade apply v1.14.10

Hopefully, you’ll get the following message: [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.14.10". Enjoy!

Next we’ll upgrade the kubelet.

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet=1.14.10-00 kubectl=1.14.10-00 && \
sudo apt-mark hold kubelet kubectl

Now we’ll update kubeadm on our worker nodes using the same command we used above.

sudo apt-mark unhold kubeadm kubelet && \
sudo apt-get update && sudo apt-get install -y kubeadm=1.14.10-00 && \
sudo apt-mark hold kubeadm

From a master node, we’ll start draining the pods which aren’t needed (e.g. app pods). To preserve kube pods, we’ll ignore DaemonSets.
kubectl drain worker_node --ignore-daemonsets

Before we drain the nodes, let’s see what pods we have deployed and where.

$ k get po -o custom-columns=NAME:{.metadata.name},NODE:{.spec.nodeName}
NAME                     NODE
bash                     runlevl42c.mylabserver.com
nginx                    runlevl42c.mylabserver.com
nginx-7cdbd8cdc9-qhhh2   runlevl42c.mylabserver.com
nginx-7cdbd8cdc9-s6dvp   runlevl42c.mylabserver.com
nginx-7cdbd8cdc9-xbdb2   runlevl43c.mylabserver.com
pod-calc                 runlevl43c.mylabserver.com
pod-calc-47chz           runlevl43c.mylabserver.com
pod-calc-4pff2           runlevl42c.mylabserver.com
pod1                     runlevl42c.mylabserver.com
repel-pod                runlevl42c.mylabserver.com
secrets                  runlevl43c.mylabserver.com
xmas                     runlevl42c.mylabserver.com

Once we’ve drained the node, we only have pods on runlevl43c.

NAME                     NODE
nginx-7cdbd8cdc9-7wbnq   runlevl43c.mylabserver.com
nginx-7cdbd8cdc9-ghggf   runlevl43c.mylabserver.com
nginx-7cdbd8cdc9-xbdb2   runlevl43c.mylabserver.com
pod-calc                 runlevl43c.mylabserver.com
pod-calc-47chz           runlevl43c.mylabserver.com
pod-calc-r7ct5           runlevl43c.mylabserver.com
secrets                  runlevl43c.mylabserver.com

From each worker node, let’s update the kubelet config.
sudo kubeadm upgrade node config --kubelet-version v1.14.10

And the kubelet itself.

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet=1.14.10-00 kubectl=1.14.10-00 && \
sudo apt-mark hold kubelet kubectl

Restart the kubelet.
sudo systemctl restart kubelet

Uncordon each worker.
kubectl uncordon node_name

Once you’re done, check the nodes.

$ k get no
NAME                         STATUS   ROLES    AGE    VERSION
runlevl41c.mylabserver.com   Ready    master   207d   v1.14.10
runlevl42c.mylabserver.com   Ready    <none>   207d   v1.14.10
runlevl43c.mylabserver.com   Ready    <none>   207d   v1.14.10

Now to keep upgrading until I’m current. 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.