In this article, we will explore a step-by-step guide on how to upgrade your Kubernetes cluster. If you are new to Kubernetes, consider reading my previous blog post about its origins, the concept of a cluster, and its various components, including the kube-apiserver and kube-controller-manager. Familiarizing yourself with these topics will help you follow this guide more effectively.

How versioning is done in Kubernetes?

When installing Kubernetes, a specific version is selected. To check the current version of Kubernetes, run the command: kubectl get nodes.

The current version of Kubernetes is expressed in the format x.y.z, where x represents the major version, y represents the minor version, and z represents the patch version. Kubernetes adheres to semantic versioning conventions.

Major version changes occur when there are significant alterations to the software, which are typically not backward compatible. These changes may introduce new features, modify the API, or deprecate an API.

Minor versions are released every few months, bringing new features and improvements. Patch versions are released more frequently, addressing bug fixes and security patches.

Apart from stable releases, Kubernetes also provides alpha and beta releases. The alpha release represents the initial release of a new feature, and it is not recommended for production use. The beta release follows the alpha release, and the feature is well-tested and documented, but it is still not considered stable enough for production use. The stable release represents the final release of a feature and is recommended for production use.

Version skew policy

To ensure that all Kubernetes components work together seamlessly, it is crucial to adhere to the version skew policy. This policy outlines the maximum allowable difference in versions between the control plane components and the components operating in worker nodes.

All Kubernetes releases can be found on the releases page. After downloading the tar.gz file, extract it to obtain all the Kubernetes components in the same version. However, keep in mind that etcd and CoreDNS are separate projects and may have different version numbers.

It is important to note that while all components do not need to be in the same version, the kube-apiserver is the primary control plane component that other components communicate with. As a result, no other component can have a higher version than the kube-apiserver.

For example, if the kube-apiserver is at version 1.24, the kube-controller-manager, kube-scheduler, and cloud-controller-manager can be at the same version or one minor version older. This means these components can be at v1.24 or v1.23.

Similarly, kube-proxy must not be newer than the kube-apiserver and can be up to two minor versions older. So, it can be at v1.24, v1.23, or v1.22. Kube-proxy may be up to two minor versions older or newer than the kubelet instance.

As for the kubelet, it must not be newer than the kube-apiserver and can be up to two minor versions older. So, it can be at v1.24, v1.23, or v1.22.

Finally, kubectl can be one minor version older or newer than the kube-apiserver, which means it can be at v1.23, v1.24, or v1.25.

Supported versions by Kubernetes

Kubernetes supports only three minor versions at any given time. For example, if your cluster is running version v1.15, the supported versions would be v1.15, v1.14, and v1.13. To ensure that your cluster remains supported, it is recommended to upgrade to a newer version before the current version reaches its end of life.

When a new version of Kubernetes is released, only the current minor version and the two previous minor versions will be supported. For instance, when v1.18 is released, only versions v1.16, v1.17, and v1.18 will be supported.

It is best practice to upgrade your cluster one minor version at a time. For example, if you want to upgrade your cluster from v1.18 to v1.20, it is recommended to first upgrade to v1.19 and then to v1.20.

When upgrading between minor versions, it is recommended to:

Upgrade the components to the most recent patch version of the current minor version.
Upgrade the components to the most recent patch version of the target minor version.

Various ways to install Kubernetes

There are several ways to install and set up a Kubernetes cluster:

Managed services: Managed services like GKE (Google Kubernetes Engine), EKS (Elastic Kubernetes Service), and AKS (Azure Kubernetes Service) offer a quick and easy way to set up a Kubernetes cluster with minimal effort. In this approach, the service provider manages the infrastructure, and you only have to manage the applications. Upgrading a managed Kubernetes cluster is also very easy.
Kubeadm tool: This tool is managed by the Kubernetes community itself and can be used to bootstrap your cluster into your infrastructure. Kubeadm simplifies the process of planning and upgrading your cluster, making it a popular choice for many users.
The hard way: This method involves manually installing and configuring each component of the cluster, including etcd, kube-apiserver, kube-controller-manager, kube-scheduler, and kubelet. This approach is recommended only if you want complete control over your Kubernetes cluster and its infrastructure. However, if you deploy your cluster manually, you will need to upgrade each component manually as well.

Regardless of which method you choose, it's important to select the one that best suits your needs and skills.

Current setup

Our current setup consists of a Kubernetes cluster running version v1.24.0, and we want to upgrade it to version v1.25.0.

Upgrade Procedure:

Before upgrading any components or nodes, we need to ensure that we have the appropriate version of kubeadm installed. We will use this tool to upgrade our cluster.

Next, we will apply the upgrade steps to the nodes in the following order:

Control plane nodes
Worker nodes

Once the upgrade procedure is completed on a particular node, we will manually upgrade the following components on that node:

kubelet
kubectl

Before proceeding with the upgrade, we need to identify the names of the control plane and worker nodes present in our cluster by running the following command:

kubectl get nodes

The output of the command shows the names of the nodes, such as controlplane and node01, both running version v1.24.0.

Upgrade control plane nodes

Upgrade kubeadm first👈

It is important to note that before upgrading the cluster to a specific version, we must first upgrade the kubeadm tool to that particular version. Kubeadm is responsible for initializing the cluster, including the control plane and worker nodes, and it is version-specific, meaning it is designed to work with a specific version of Kubernetes. This helps ensure compatibility. The kubeadm tool is in charge of bootstrapping the cluster and upgrading its control plane components, so it needs to run the same version as the version of Kubernetes you want to upgrade to.

By executing the kubeadm upgrade plan command, we can view all the necessary details required to perform the cluster upgrade.

The current version of the kubeadm tool and the Kubernetes cluster.
The remote version refers to the latest release of Kubernetes.
The latest release in the v1.24 series of Kubernetes.
After upgrading the control plane components, manually upgrade the kubelet on each master and worker node.
The current version of the control plane components and the version to which they can be upgraded.
Command to upgrade the cluster to version v1.24.13.
Upgrade the kubeadm tool to the desired version of the cluster.

When we ran the command it clearly stated that before upgrading the cluster, we have to upgrade the kubeadm to that version.

👉Check all available versions of kubeadm

To check all available versions of kubeadm, run the following commands:
```
  apt update
  apt-cache madison kubeadm
```
In the current setup, the cluster is running on version v1.24.0 and the latest version in the v1.24 series is v1.24.13.
However, the goal is to upgrade the cluster to version v1.25.0, which requires upgrading kubeadm to the same version first.

👉Upgrade kubeadm

Upgrade kubeadm to version v1.25.0.

  apt-mark unhold kubeadm && \
  apt-get update && apt-get install -y kubeadm=1.25.0-00 && \
  apt-mark hold kubeadm

Unholding kubeadm removes any existing version constraints on the package, allowing it to be upgraded to the desired version. On the other hand, holding kubeadm allows us to prevent accidental upgrades or downgrades of the package in the future.
However, it's important to note that this command should be executed on all nodes that will be involved in the cluster upgrade, including the control plane and worker nodes.

👉Verify the download

After upgrading kubeadm to version v1.25.0, we can check whether the download was successful and kubeadm has the desired version by running the following command:
```
  kubeadm version
```
This command will output the version of kubeadm currently running in the cluster. If the output shows version v1.25.0, then it means the upgrade was successful and kubeadm is now running the desired version.

👉Plan your cluster upgrade

This command checks the current version of Kubernetes and outputs the possible versions to which it can be upgraded. It also lists the available upgrade versions for control plane components and tells you which components can be upgraded automatically and which have to be done manually. Running this command before upgrading the cluster helps to ensure a smooth and successful upgrade process.
```
  kubeadm upgrade plan
```

👉Upgrade control plane node

After upgrading the kubeadm tool, the next step is to upgrade the control plane node components. The control plane components include the API server, etcd, kube-scheduler, and kube-controller-manager. Upgrading these components ensures that the control plane node is running the desired version of Kubernetes and incorporates the latest bug fixes and security patches.
To upgrade the control plane node components, you can select a specific version to upgrade to and run the appropriate command. For example, if you want to upgrade from version v1.24.0 to v1.25.0, you can run the following command:
```
  sudo kubeadm upgrade apply v1.25.0
```
When it prompts to confirm type 'y'
After you have successfully upgraded, you should see this message:

Note: After successfully upgrading, when checking the version of the control plane node by running the command: kubectl get nodes it will still show the previous version of the cluster. This version specifies the version of the kubelet running on the node not the version of the cluster. After upgrading the kubelet, you should see the desired version.

👉Upgrade additional control plane nodes.

After upgrading the first control plane node, you need to upgrade the additional control plane nodes. You don't need to upgrade the kubeadm tool again, you can simply run the following command on each additional control plane node:
```
  sudo kubeadm upgrade node
```
This will upgrade the node to the same version as the first control plane node. Once the upgrade is complete, you should see a message confirming the success of the upgrade.

👉Drain control plane node

It is important to note that before upgrading kubelet and kubectl, you should drain the control plane node to ensure that the workloads or pods are shifted to another node. To drain the control plane node, run the following command:
```
  kubectl drain controlplane --ignore-daemonsets
```
This will evict all the pods out of the node and will cordon off the node, i.e.; no pods can be scheduled on it. Now we can safely upgrade the kubelet and kubectl.

👉Upgrade kubelet and kubectl

Run these commands:

  apt-mark unhold kubelet kubectl && \
  apt-get update && apt-get install -y kubelet=1.25.0-00 kubectl=1.25.0-00 && \
  apt-mark hold kubelet kubectl

It is also important to note the Kubernetes version skew policy, which states that you can only upgrade one minor version at a time. For example, if you are currently running version v1.25.1, you cannot upgrade directly to v1.27.1. You would need to first upgrade to any available version in the v1.26 series, and then you can upgrade to any available version in the v1.27 series.

👉Restart the kubelet service

The kubelet service is responsible for managing the state of each node in the Kubernetes cluster. After upgrading kubelet to a new version, you need to restart the kubelet service to apply the changes. To do this, run the following commands:
```
  sudo systemctl daemon-reload
  sudo systemctl restart kubelet
```

👉Uncordon the node

When you drained the control plane node before upgrading kubelet, you made sure that no new pods were scheduled on the node during the upgrade process. Now that you have upgraded kubelet and restarted the kubelet service, you can uncordon the control plane node to allow pods to be scheduled on it again. To do this, run the following command:
```
  kubectl uncordon controlplane
```
This will make the control plane node available for scheduling new pods.

Upgrade worker nodes

After successfully upgrading the control plane nodes, it's time to upgrade the worker nodes. The process is similar to that of the control plane nodes, where we first upgrade kubeadm to a specific version and then upgrade kubelet and kubectl tools.

To upgrade the worker nodes, first SSH into each node, As we have only one worker node in our setup, we will ssh into that node. Before we SSH into 'node01' to upgrade components we have to drain the node first so that all the pods running on the node are rescheduled on some other nodes and this process has to be done from a control plane node.

👉Drain the node to evict pods (workloads) [ Run in the control plane node ]

Run this command. The --ignore-daemonsets flag will ignore all the daemon sets running on the node.
```
  kubectl drain node01 --ignore-daemonsets
```

👉SSH into the node

After moving all pods from the node, we can now SSH into the node. To SSH into 'node01' run the following command:
```
  ssh node01
```
Press 'Ctrl + d' any time to exit out of the node.

👉Upgrade kubeadm [ Run in the worker node ]

Upgrade the kubeadm tool to the version v1.25.0.

  apt-mark unhold kubeadm && \
  apt-get update && apt-get install -y kubeadm=1.25.0-00 && \
  apt-mark hold kubeadm

👉Upgrade local kubelet configuration [ Run in the worker node ]

```
  sudo kubeadm upgrade node
```

👉Upgrade kubelet and kubectl [ Run in the worker node ]

Upgrade kubelet and kubectl to version v1.25.0.

  apt-mark unhold kubelet kubectl && \
  apt-get update && apt-get install -y kubelet=1.25.0-00 kubectl=1.25.0-00 && \
  apt-mark hold kubelet kubectl

👉Restart the kubelet service [ Run in the worker node ]

  sudo systemctl daemon-reload
  sudo systemctl restart kubelet

👉Uncordon the node [ Run in the control plane node ]

After upgrading the kubectl and kubelet on the worker node, we then exit out of that node. When we drained the node, it moved all the pods to a different node and also cordoned the node. Now that we have upgraded everything, we have to again uncordon it so that new pods can be scheduled on it.
```
  kubectl uncordon node01
```

Check Upgrade

After upgrading the control plane and worker nodes, we can check the status of the nodes by running the command: kubectl get nodes

We can see that both the control plane node 'controlplane' and the worker node 'node01' are upgraded to version v1.25.0. Hence our upgrade process was successful.

Conclusion

In conclusion, updating your Kubernetes cluster is essential for maintaining its stability, security, and performance. By following the step-by-step guide outlined in this article, you can successfully upgrade your cluster's control plane and worker nodes, ensuring compatibility with the latest Kubernetes features and enhancements. The kubeadm tool must be upgraded to the desired version of the cluster before upgrading the control plane components and worker node components. Remember to adhere to the Kubernetes version skew policy and upgrade one minor version at a time. By upgrading your cluster, you can ensure your cluster remains updated, secure, and efficient.

Sambit's blog