[Cloud] 1. Kubernetes Installation

In this post, I will post about the installation procedure of Kubernetes, a cloud automation tool. The OS is based on Ubuntu 20.04.

​

As shown in the figure above, Kubernetes is a tool that automatically configures and manages infrastructure, and even deploys and recovers from failures. It is largely classified into Master Node and Worker Node, and Master Node is composed of 4 letters as shown in the figure above.

• API Server
• Scheduler
• Controller-Manager
• etcd

And Worker Nodes are mainly composed of 4 types.

• Pod
• Docker
• Kubelet
• Kube-proxy

반응형

Details and operation method will be posted in a later post, and in this post, we will post based on the installation. First of all, if you look at the practice environment, install 1 Master Node VM and 2 Worker Node VMs in a VirtualBox environment and in a NAT network environment as shown below.

​

The procedure below requires installation in Master Node and Worker Node mode. (No. 1 to No. 8) Install Docker as follows.

root@master-VirtualBox:~#apt-get update root@master-VirtualBox:~#apt-get install docker.io

1. Update apt package and install https, curl, ca-certi as follows.

oot@master-VirtualBox:~# apt-get update && apt-get install -y apt-transport-https curl ca-certificates Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease Hit:2 http://kr.archive.ubuntu.com/ubuntu focal InRelease Hit:3 http://kr.archive.ubuntu.com/ubuntu focal-updates InRelease Hit:4 http://kr.archive.ubuntu.com/ubuntu focal-backports InRelease Reading package lists... Done Reading package lists... Done Building dependency tree Reading state information... Done ca-certificates is already the newest version (20211016~20.04.1). ca-certificates set to manually installed. The following package was automatically installed and is no longer required: libfwupdplugin1 Use 'apt autoremove' to remove it. The following NEW packages will be installed: apt-transport-https curl 0 upgraded, 2 newly installed, 0 to remove and 1 not upgraded. Need to get 163 kB of archives. After this operation, 573 kB of additional disk space will be used. Get:1 http://kr.archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B] Get:2 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 curl amd64 7.68.0-1ubuntu2.12 [161 kB] Fetched 163 kB in 1s (113 kB/s) Selecting previously unselected package apt-transport-https. (Reading database ... 178564 files and directories currently installed.) Preparing to unpack .../apt-transport-https_2.0.9_all.deb ... Unpacking apt-transport-https (2.0.9) ... Selecting previously unselected package curl. Preparing to unpack .../curl_7.68.0-1ubuntu2.12_amd64.deb ... Unpacking curl (7.68.0-1ubuntu2.12) ... Setting up apt-transport-https (2.0.9) ... Setting up curl (7.68.0-1ubuntu2.12) ... Processing triggers for man-db (2.9.1-1) ... root@master-VirtualBox:~# ​

2. Download Google's public key and add it to your local key.

root@master-VirtualBox:~# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - OK root@master-VirtualBox:~#

​

3. Add the Kubernetes repository to your local repository.

oot@master-VirtualBox:~# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - OK root@master-VirtualBox:~# cat <<EOF >/etc/apt/sources.list.d/kubernetes.list > deb https://apt.kubernetes.io/ kubernetes-xenial main > EOF root@master-VirtualBox:~#

​

4. Update apt and install Kubernetes to download the added Kubernetes package.

root@master-VirtualBox:~# apt-get update Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] Hit:2 http://kr.archive.ubuntu.com/ubuntu focal InRelease Get:4 http://kr.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [57.6 kB] Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 DEP-11 Metadata [40.6 kB] Get:7 http://kr.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] Get:8 http://security.ubuntu.com/ubuntu focal-security/universe amd64 DEP-11 Metadata [66.7 kB] Get:9 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 DEP-11 Metadata [2,464 B] Get:10 http://kr.archive.ubuntu.com/ubuntu focal-updates/main amd64 DEP-11 Metadata [278 kB] Get:11 http://kr.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [926 kB] Get:12 http://kr.archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [678 kB] Get:13 http://kr.archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] Get:14 http://kr.archive.ubuntu.com/ubuntu focal-updates/universe amd64 DEP-11 Metadata [391 kB] Get:15 http://kr.archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 DEP-11 Metadata [944 B] Get:16 http://kr.archive.ubuntu.com/ubuntu focal-backports/main amd64 DEP-11 Metadata [8,004 B] Get:17 http://kr.archive.ubuntu.com/ubuntu focal-backports/universe amd64 DEP-11 Metadata [30.5 kB] Fetched 3,033 kB in 14s (221 kB/s) Reading package lists... Done oot@master-VirtualBox:~# apt-get install -y kubelet kubeadm kubectl Reading package lists... Done Building dependency tree Reading state information... Done The following package was automatically installed and is no longer required: libfwupdplugin1 Use 'apt autoremove' to remove it. The following additional packages will be installed: conntrack cri-tools ebtables kubernetes-cni socat Suggested packages: nftables The following NEW packages will be installed: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 0 upgraded, 8 newly installed, 0 to remove and 1 not upgraded. Need to get 75.2 MB of archives. After this operation, 313 MB of additional disk space will be used. Get:3 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.24.2-00 [12.3 MB] Get:7 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] Get:8 http://kr.archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.24.3-00 [19.2 MB] Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.24.3-00 [9,316 kB] Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.24.3-00 [9,002 kB] Fetched 75.2 MB in 14s (5,514 kB/s) Selecting previously unselected package conntrack. (Reading database ... 178575 files and directories currently installed.) Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... Unpacking conntrack (1:1.4.5-2) ... Selecting previously unselected package cri-tools. Preparing to unpack .../1-cri-tools_1.24.2-00_amd64.deb ... Unpacking cri-tools (1.24.2-00) ... Selecting previously unselected package ebtables. Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... Unpacking ebtables (2.0.11-3build1) ... Selecting previously unselected package kubernetes-cni. Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... Unpacking kubernetes-cni (0.8.7-00) ... Selecting previously unselected package socat. Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... Unpacking socat (1.7.3.3-2) ... Selecting previously unselected package kubelet. Preparing to unpack .../5-kubelet_1.24.3-00_amd64.deb ... Unpacking kubelet (1.24.3-00) ... Selecting previously unselected package kubectl. Preparing to unpack .../6-kubectl_1.24.3-00_amd64.deb ... Unpacking kubectl (1.24.3-00) ... Selecting previously unselected package kubeadm. Preparing to unpack .../7-kubeadm_1.24.3-00_amd64.deb ... Unpacking kubeadm (1.24.3-00) ... Setting up conntrack (1:1.4.5-2) ... Setting up kubectl (1.24.3-00) ... Setting up ebtables (2.0.11-3build1) ... Setting up socat (1.7.3.3-2) ... Setting up cri-tools (1.24.2-00) ... Setting up kubernetes-cni (0.8.7-00) ... Setting up kubelet (1.24.3-00) ... Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /lib/systemd/system/kubelet.service. Setting up kubeadm (1.24.3-00) ... Processing triggers for man-db (2.9.1-1) ... root@master-VirtualBox:~#

5. Run the below command to prevent Kubernetes from auto-updating.

root@master-VirtualBox:~# apt-mark hold kubelet kubeadm kubectl kubelet set on hold. kubeadm set on hold. kubectl set on hold. root@master-VirtualBox:~#

6. Change and reflect the settings for Kubernetes iptable as follows.

root@master-VirtualBox:~# cat <<EOF > /etc/sysctl.d/k8s.conf > net.bridge.bridge-nf-call-ip6tables = 1 > net.bridge.bridge-nf-call-iptables = 1 > net.ipv4.ip_forward = 1 > EOF root@master-VirtualBox:~# root@master-VirtualBox:~# sysctl --system * Applying /etc/sysctl.d/10-console-messages.conf ... kernel.printk = 4 4 1 7 * Applying /etc/sysctl.d/10-ipv6-privacy.conf ... net.ipv6.conf.all.use_tempaddr = 2 net.ipv6.conf.default.use_tempaddr = 2 * Applying /etc/sysctl.d/10-kernel-hardening.conf ... kernel.kptr_restrict = 1 * Applying /etc/sysctl.d/10-link-restrictions.conf ... fs.protected_hardlinks = 1 fs.protected_symlinks = 1 * Applying /etc/sysctl.d/10-magic-sysrq.conf ... kernel.sysrq = 176 * Applying /etc/sysctl.d/10-network-security.conf ... net.ipv4.conf.default.rp_filter = 2 net.ipv4.conf.all.rp_filter = 2 * Applying /etc/sysctl.d/10-ptrace.conf ... kernel.yama.ptrace_scope = 1 * Applying /etc/sysctl.d/10-zeropage.conf ... vm.mmap_min_addr = 65536 * Applying /usr/lib/sysctl.d/30-tracker.conf ... fs.inotify.max_user_watches = 65536 * Applying /usr/lib/sysctl.d/50-default.conf ... net.ipv4.conf.default.promote_secondaries = 1 sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument net.ipv4.ping_group_range = 0 2147483647 net.core.default_qdisc = fq_codel fs.protected_regular = 1 fs.protected_fifos = 1 * Applying /usr/lib/sysctl.d/50-pid-max.conf ... kernel.pid_max = 4194304 * Applying /etc/sysctl.d/99-sysctl.conf ... * Applying /etc/sysctl.d/k8s.conf ... net.ipv4.ip_forward = 1 * Applying /usr/lib/sysctl.d/protect-links.conf ... fs.protected_fifos = 1 fs.protected_hardlinks = 1 fs.protected_regular = 2 fs.protected_symlinks = 1 * Applying /etc/sysctl.conf ... root@master-VirtualBox:~# ​

​

7. Turn off the host's Hot Swap function and change the setting so that it turns off even when rebooting.

root@master-VirtualBox:~# swapoff -a root@master-VirtualBox:~# sed -i '/swap/s/^/#/' /etc/fstab

 

8. Restart Kubernetes.

root@master-VirtualBox:~# systemctl daemon-reload root@master-VirtualBox:~# systemctl restart kubelet

9. Start Kubernetes Clustering (Only on Master Node) Save the token value at the bottom of the notepad (red)

root@master-VirtualBox:~# kubeadm init [init] Using Kubernetes version: v1.24.3 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master-virtualbox] and IPs [10.96.0.1 192.168.1.4] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [localhost master-virtualbox] and IPs [192.168.1.4 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [localhost master-virtualbox] and IPs [192.168.1.4 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 21.003363 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node master-virtualbox as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node master-virtualbox as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: vdtbt8.c2xi7o64kq1u6ocw [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy ​ Your Kubernetes control-plane has initialized successfully! ​ To start using your cluster, you need to run the following as a regular user: ​ mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ​ Alternatively, if you are the root user, you can run: ​ export KUBECONFIG=/etc/kubernetes/admin.conf ​ You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ ​ Then you can join any number of worker nodes by running the following on each as root: ​ kubeadm join 192.168.1.4:6443 --token vdtbt8.c2xi7o64kq1u6ocw \ --discovery-token-ca-cert-hash sha256:da78c6cf3a148f958474f003d42643cc64a9efe1aa7e4000d3dd5db10270ac85 root@master-VirtualBox:~# ​

​

10. Execute the command below on the Master Node to start using the Kubernetes cluster. Note that it depends on which user you are running as. In this case, run as root user. (Master Node)

root@master-VirtualBox:~# mkdir -p $HOME/.kube root@master-VirtualBox:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config root@master-VirtualBox:~# chown $(id -u):$(id -g) $HOME/.kube/config

​

11. Execute the following command as the root account to set the certificate used when connecting to the cluster. (Master Node)

root@master-VirtualBox:~# export KUBECONFIG=/etc/kubernetes/admin.conf

12. Download the manifest file of the Pod network add-on module (calico) that enables communication between Pods (Master Node)

oot@master-VirtualBox:~# curl https://docs.projectcalico.org/archive/v3.17/manifests/calico.yaml -O % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 183k 100 183k 0 0 84984 0 0:00:02 0:00:02 --:--:-- 84984 root@master-VirtualBox:~#

13. Apply the Meniffest file. (Master Node)

root@master-VirtualBox:~# kubectl apply -f calico.yaml configmap/calico-config created customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created clusterrole.rbac.authorization.k8s.io/calico-node created clusterrolebinding.rbac.authorization.k8s.io/calico-node created daemonset.apps/calico-node created serviceaccount/calico-node created deployment.apps/calico-kube-controllers created serviceaccount/calico-kube-controllers created Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget poddisruptionbudget.policy/calico-kube-controllers created root@master-VirtualBox:~# ​

14. Now on the Worker Node, run the below command to include it in the Kubernetes node cluster.

root@worknode1-VirtualBox:~# kubeadm join 192.168.1.4:6443 --token vdtbt8.c2xi7o64kq1u6ocw \ > --discovery-token-ca-cert-hash sha256:da78c6cf3a148f958474f003d42643cc64a9efe1aa7e4000d3dd5db10270ac85 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... ​ This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. ​ Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

15. Now, in the Master node, check the status of the nodes included in the cluster.

root@master-VirtualBox:~# kubectl get nodes NAME STATUS ROLES AGE VERSION master-virtualbox Ready control-plane 48m v1.24.3 worknode1-virtualbox Ready <none> 30m v1.24.3 worknode2-virtualbox Ready <none> 65s v1.24.3

​

16. Now try to query the status of Control Pods in the cluster.

root@master-VirtualBox:~# kubectl get pods -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-kube-controllers-56bfc4c5c7-cv5j7 1/1 Running 0 43m 172.16.168.193 worknode1-virtualbox <none> <none> calico-node-5q4xb 1/1 Running 0 43m 192.168.1.4 master-virtualbox <none> <none> calico-node-l8m94 1/1 Running 0 35m 192.168.1.5 worknode1-virtualbox <none> <none> calico-node-tw6xw 1/1 Running 0 6m14s 192.168.1.6 worknode2-virtualbox <none> <none> coredns-6d4b75cb6d-224hd 1/1 Running 0 53m 172.16.29.66 master-virtualbox <none> <none> coredns-6d4b75cb6d-qgscs 1/1 Running 0 53m 172.16.29.65 master-virtualbox <none> <none> etcd-master-virtualbox 1/1 Running 0 53m 192.168.1.4 master-virtualbox <none> <none> kube-apiserver-master-virtualbox 1/1 Running 0 53m 192.168.1.4 master-virtualbox <none> <none> kube-controller-manager-master-virtualbox 1/1 Running 0 53m 192.168.1.4 master-virtualbox <none> <none> kube-proxy-hh7fv 1/1 Running 0 35m 192.168.1.5 worknode1-virtualbox <none> <none> kube-proxy-wwm2h 1/1 Running 0 53m 192.168.1.4 master-virtualbox <none> <none> kube-proxy-xztdj 1/1 Running 0 6m14s 192.168.1.6 worknode2-virtualbox <none> <none> kube-scheduler-master-virtualbox 1/1 Running 0 53m 192.168.1.4 master-virtualbox <none> <none> root@master-VirtualBox:~#

​

The Kubernetes installation is now complete. In the next post, I will post sequentially about how to operate pods and services through Kubernetes.

​

 

​