Kubernetes is an open-source tool for managing and orchestrating containers. These containers can be built with Docker or other equivallent tool.
From a IaaS and SaaS perspective, most major providers have solutions for this. For example OpenShift by RedHat, AKS by Azure, EKS by Amazon etc.
These solutions shields the grunt work and makes this easier to use and adopt. This is great but also comes at a cost.
In this post we will be looking at setting up a Kubernetes Cluster from the ground up. This option gives us the most control and flexibility.
Server Requirements
We will be building a 3 node cluster comprised of one master and 2 worker nodes:
| Hostname |
Ip Address |
Host Specifications |
Operating System |
| kube-master |
192.168.1.197 |
2 CPU, 8GB RAM, 20GB HDD |
Rocky Linux |
| kube-node-01 |
192.168.1.239 |
2 CPU, 4GB RAM, 20GB HDD |
Rocky Linux |
| kube-node-02 |
192.168.1.109 |
2 CPU, 4GB RAM, 20GB HDD |
Rocky Linux |
Package Installation and System Configuration
Perform these tasks on all 3 hosts. This assumes you are logged in as a user with sudo privilleges.
Update Hostname, Hosts file and install base packages:
sudo dnf update -y && sudo dnf install -y sysstat unzip jq gcc kernel-devel kernel-headers \
dnf-plugins-core python3-pip iproute-tc rpm-ostree dnf-plugins-core net-tools vim wget curl \
telnet nfs-utils python3-dnf-plugin-versionlock chrony bind-utils binutils-devel libxml2-devel \
zlib-devel openssl-devel systemd-devel kernel-modules-extra
Install Docker & Containerd:
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf update -y && sudo dnf install -y docker-ce docker-ce-cli containerd.io
Setup entry for private registry
sudo bash -c 'cat <<EOF> /etc/docker/daemon.json
{
"insecure-registries" : ["sonarnexus.infracl.com:5000"]
}
EOF'
Configure Containerd and setup Private Registry:
sudo systemctl enable containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl status containerd
sudo vim /etc/containerd/config.toml
>>
[plugins.'io.containerd.cri.v1.images'.registry]
config_path = '/etc/containerd/certs.d'
sudo mkdir -p /etc/containerd/certs.d/sonarnexus.infracl.com:5000
sudo vim /etc/containerd/certs.d/sonarnexus.infracl.com:5000/hosts.toml
>>
server = "http://sonarnexus.infracl.com:5000"
[host."http://sonarnexus.infracl.com:5000"]
capabilities = ["pull", "resolve", "push"]
[host."http://sonarnexus.infracl.com:5000".header]
authorization = "Basic YWRtaW46cGFzc3dvcmQ="
Update Firewall & Selinux:
sudo sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
sudo systemctl disable firewalld
NOTE: The above should not be done in Production. See below and update based on your needs
firewall-cmd --add-masquerade --permanent
firewall-cmd --permanent --add-port=6443/tcp
firewall-cmd --permanent --add-port=2379-2380/tcp
firewall-cmd --permanent --add-port=10250/tcp
firewall-cmd --permanent --add-port=10251/tcp
firewall-cmd --permanent --add-port=10252/tcp
firewall-cmd --permanent --add-port=10255/tcp
firewall-cmd --permanent --add-port=6783/tcp
firewall-cmd --permanent --add-port=10255/tcp
firewall-cmd --permanent --add-port=30000-32767/tcp
firewall-cmd --reload
Load Modules:
sudo modprobe overlay && sudo modprobe br_netfilter
sudo bash -c 'cat <<EOF > /etc/modules-load.d/overlay.conf
overlay
br_netfilter
EOF'
Create Custom sysctl:
sudo bash -c 'cat <<EOF>> /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF'
Create Kubernetes Repository:
sudo bash -c cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.36/rpm/
enabled=0
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.36/rpm/repodata/repomd.xml.key
EOF
Install Kubernetes on master node:
sudo dnf --enablerepo=kubernetes -y install kubelet kubeadm kubectl
sudo systemctl enable kubelet && sudo systemctl start kubelet
sudo systemctl status kubelet
Install Kubernetes on worker node:
sudo dnf --enablerepo=kubernetes -y install kubelet kubeadm
sudo systemctl enable kubelet && sudo systemctl start kubelet
sudo systemctl status kubelet
Disable SWAP:
sudo sed -i '/swap/d' /etc/fstab
sudo swapoff -a
Reboot all nodes when this is completed
Initialize Cluster:
Log into the kubernetes Master and initialize the cluster.
sudo kubeadm init --cri-socket unix:///var/run/containerd/containerd.sock
This will initialize the cluster and output the connection string. This section is what you want to focus on
....
....
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.1.197:6443 --token zfuhxi.hcvzal3hkkoctw10 \
--discovery-token-ca-cert-hash sha256:ad04b7e4d27d5020339a29da45879baf0318988d98626955dd03f887430e1bfa
Test your installation by typeing this command kubectl get nodes. You should see
NAME STATUS ROLES AGE VERSION
NAME STATUS ROLES AGE VERSION
kube-master NotReady control-plane 3m19s v1.36.2
Install Calico Network:
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.0/manifests/calico.yaml
Test Master Node
NAME STATUS ROLES AGE VERSION
kube-master Ready control-plane 6m31s v1.36.2
Now, login to the other nodes and type this command:
sudo kubeadm join 192.168.1.197:6443 --token zfuhxi.hcvzal3hkkoctw10 \
--discovery-token-ca-cert-hash sha256:ad04b7e4d27d5020339a29da45879baf0318988d98626955dd03f887430e1bfa \
--cri-socket unix:///run/containerd/containerd.sock
Now log back to the master node and run this command kubectl get nodes. If all goes well, you should see this
NAME STATUS ROLES AGE VERSION
kube-master Ready control-plane 54m v1.36.2
kube-node-01 Ready <none> 18m v1.36.2
kube-node-02 Ready <none> 72s v1.36.2
For detailed node information, type kubectl get nodes -o wide. You should see
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kube-master Ready control-plane 54m v1.36.2 192.168.1.197 <none> Rocky Linux 10.2 (Red Quartz) 6.12.0-211.26.1.el10_2.x86_64 (amd64) containerd://2.2.5
kube-node-01 Ready <none> 18m v1.36.2 192.168.1.239 <none> Rocky Linux 10.2 (Red Quartz) 6.12.0-211.26.1.el10_2.x86_64 (amd64) containerd://2.2.5
kube-node-02 Ready <none> 111s v1.36.2 192.168.1.109 <none> Rocky Linux 10.2 (Red Quartz) 6.12.0-211.26.1.el10_2.x86_64 (amd64) containerd://2.2.5
Your cluster is now ready.
Kubernetes Headlamp
To enable the Kubernetes Headlamp, See Kubernetes Headlamp Web UI. In summary, type the following:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/headlamp/main/kubernetes-headlamp.yaml
To see new dashboard, use port forwarding kubectl port-forward -n kube-system service/headlamp 8080:80.
Then open your browser and type http://localhost:8080
Reference Links