Table of Contents

실험 환경

...

설치 Version

CRI-O, kube-apiserver, kubelet, kube-controller-manager, kube-scheduler, cloud-controller-manager, kubectl 등의 Kubernetes 구성 요소들의 버전은 1.25로 설치한다.

Note

단, 구성 요소들간의 version skew policy를 참고하여 반드시 허용 범위내의 버전으로 설치 및 업그레이드를 진행하여야 한다.

버전 차이(skew) 정책

Install Container Runtime

설치 가이드
위 linked page의 지침 中, CRI-O로 설치.

Short installation guide:

Code Block

language	bash

OS=CentOS_8 VERSION=1.25
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
yum install -y cri-o
systemctl enable crio & systemctl start crio

Installing K8s with kubeadm

Ref: https://v1-25.docs.kubernetes.io/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#kubeadm-kubelet-%EB%B0%8F-kubectl-%EC%84%A4%EC%B9%98

K8s의 환경을 구성하는 방법에는 다양한 방법이 있으며,
여기에서는 kubedam이라는 tool을 사용하여 구성한다.

Requirements & Prechecking

2 GB or more of RAM per machine
2 CPUs or more
Unique hostname, MAC address, and product_uuid for every node.
- You can find the following instructions.
  - Host name: hostname
  - MAC address: ip link or ip a or ifconfig -a
  - Product uuid: sudo cat /sys/class/dmi/id/product_uuid
Full network connectivity between all machines in the cluster (public or private network is fine)
Certain ports are open on your machines.
- See here for more details.
Swap disabled.
- 일시적: swapoff -a
- 영구적: /etc/fstab 파일의 swap mount 부분 제거.

Forwarding IPv4 and letting iptables see bridged traffic

Setting:

Code Block

language	py

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

Verify that the br_netfilter, overlay modules are loaded by running the following commands:
Code Block
lsmod | grep br_netfilter lsmod | grep overlay
Verify that the net.bridge.bridge-nf-call-iptables, net.bridge.bridge-nf-call-ip6tables, and net.ipv4.ip_forward system variables are set to 1 in your sysctl config by running the following command:
Code Block
sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

Check and open required ports

기본으로 요구되는 port들은 다음과 같다.

Control-plane node(s)

Protocol	Direction	Port Range	Purpose	Used By
TCP	Inbound	6443*	Kubernetes API server	All
TCP	Inbound	2379-2380	etcd server client API	kube-apiserver, etcd
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	10251	kube-scheduler	Self
TCP	Inbound	10252	kube-controller-manager	Self

Worker node(s)

Protocol	Direction	Port Range	Purpose	Used By
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	30000-32767	NodePort Services†	All

Open ports via Firewalld

Note

위 표에서 나열한 port들을 하나씩 오픈해도 되지만, etcd와 같이 내부 클러스터 통신을 하는 또 다른 pod들을 생성할 수도 있다.
그때마다 따로 지정된 port를 오픈해야 하는 번거로움 때문에 여기서는 Source IP 대역으로 오픈하여 신뢰되는 네트워크 간(Master nodes ↔︎ Worker nodes) 통신은 허용하도록 하였다.

참고로 개별 포트를 오픈하는 방법도 아래 펼침 목록에 넣어두었다.

단, 어떤 방식으로 하든 간에, masquerade설정은 반드시 적용하여야 한다.
설정하지 않을 시, 서로 다른 node의 pod간 routing이 되지 않는다.

On every k8s nodes

Code Block

language	xml

/etc/firewalld/zones/public.xml
<?xml version="1.0" encoding="utf-8"?>
<zone>
  <short>Public</short>
  <description>For use in public areas. You do not trust the other computers on networks to not harm your computer. Only selected incoming connections are accepted.</description>
  <service name="https"/>
  <port protocol="tcp" port="6443"/>
  <port protocol="tcp" port="10250"/>
  <masquerade/>
  <rule family="ipv4">
    <source address="10.0.0.0/8"/>
    <accept/>
  </rule>
</zone>

Expand

title	Open by each port

On Master nodes

Code Block

language	py

$ firewall-cmd --permanent --add-port=6443/tcp
$ firewall-cmd --permanent --add-port=2379-2380/tcp
$ firewall-cmd --permanent --add-port=10250/tcp
$ firewall-cmd --permanent --add-port=10251/tcp
$ firewall-cmd --permanent --add-port=10252/tcp
$ firewall-cmd --permanent --add-port=10255/tcp
$ firewall-cmd --permanent --add-port=8472/udp
$ firewall-cmd --permanent --add-masquerade

# only if you want NodePorts exposed on control plane IP as well
$ firewall-cmd --permanent --add-port=30000-32767/tcp
systemctl restart firewalld

$ firewall-cmd --reload

On Worker nodes

Code Block

$ firewall-cmd --permanent --add-port=10250/tcp
$ firewall-cmd --permanent --add-port=10255/tcp
$ firewall-cmd --permanent --add-port=8472/udp
$ firewall-cmd --permanent --add-port=30000-32767/tcp
$ firewall-cmd --permanent --add-masquerade

$ firewall-cmd --reload

Installing kubeadm, kubelet and kubectl

You will install these packages on all of your machines:

kubeadm: the command to bootstrap the cluster.
kubelet: the component that runs on all of the machines in your cluster and does things like starting pods and containers.
kubectl: the command line util to talk to your cluster.

Installation instructions on CentOS, RHEL or Fedora

Code Block

language	py

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

yum install -y kubelet-1.25.4 kubeadm-1.25.4 kubectl-1.25.4 --disableexcludes=kubernetes

systemctl enable --now kubelet

Note

위 SELinux 세팅은, 컨테이너가 호스트 파일시스템에 엑세스 하는 것을 허용한다.(예를 들어, pod networks 같은…)

적어도 kubelet에서 보다 향상된 SELinux 지원이 이루어 질 때까지 유지하는 것이 좋다.

Start kubelet

Note
Worker node에서는 kubelet 서비스를 아래처럼 실행하지 않아도 된다.(하더라도 실패할 것이다.) 다음 장에서 살펴볼 `kubeadm join`을 하면 자동 실행된다.

sudo systemctl start kubelet 명령으로 실행하고 나서 , journalctl -n1000 -f|grep -i kubelet 를 통해 로그를 보면 /var/lib/kubelet/config.yaml이 없다고 에러가 난다.
주기(10초)적으로 systemd에 의해 실행되며, 또한 설정 파일에 변경 사항이 발견되면 systemctl daemon-reload && systemctl restart kubelet이 실행된다.

...

Version	Old Version 1	New Version 2
Changes made by	Jack kim	Jack kim
Saved on	May 31, 2023	Jun 01, 2023

Versions Compared

Key

실험 환경

설치 Version

Install Container Runtime

Installing K8s with kubeadm

Requirements & Prechecking

Check and open required ports

Control-plane node(s)

Worker node(s)

Open ports via Firewalld

Installing kubeadm, kubelet and kubectl

Installation instructions on CentOS, RHEL or Fedora

Start kubelet

Content Comparison

Versions Compared

Key

실험 환경

설치 Version

Install Container Runtime

Installing K8s with kubeadm

Requirements & Prechecking

Check and open required ports

Control-plane node(s)

Worker node(s)

Open ports via Firewalld

Installing kubeadm, kubelet and kubectl

Installation instructions on CentOS, RHEL or Fedora

Start kubelet