{"type":"doc","content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{},"macroMetadata":{"macroId":{"value":"b39800ed-17b9-42dd-9d06-397040715adf"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"f46180e9-2e1f-4210-a1f5-f41575ea2391"}},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"본 페이지에서 쓰이는 Custom K8s object 정의 파일(yaml)은 아래 git repo에서 다운로드 할 수 있다.","type":"text"}]},{"type":"paragraph","content":[{"text":"Github repo: ","type":"text"},{"text":"snetsystems","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/snetsystems"}}]},{"text":"/","type":"text"},{"text":"K8s-Objects","type":"text","marks":[{"type":"strong"},{"type":"link","attrs":{"href":"https://github.com/snetsystems/K8s-Objects"}}]}]}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"아래 실행 명령어들에는 편의상 sudo와 같은 privilege를 필수 표기하지는 않는다.","type":"text"},{"type":"hardBreak"},{"text":"실습[운영] 환경에 맞추어 알맞게 실행하도록 한다.","type":"text"}]}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"type":"inlineCard","attrs":{"url":"https://v1-25.docs.kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/"}},{"text":" ","type":"text"}]},{"type":"paragraph","content":[{"type":"inlineCard","attrs":{"url":"https://waspro.tistory.com/516"}},{"text":" ","type":"text"}]},{"type":"paragraph","content":[{"type":"inlineCard","attrs":{"url":"https://may9noy.tistory.com/343"}},{"text":" ","type":"text"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Initialize the control-plane node","type":"text"}]},{"type":"codeBlock","content":[{"text":"kubeadm init [flags]","type":"text"}]},{"type":"paragraph","content":[{"text":"--apiserver-advertise-address","type":"text","marks":[{"type":"code"}]},{"text":"와 ","type":"text"},{"text":"--control-plane-endpoint","type":"text","marks":[{"type":"code"}]},{"text":"옵션(flag)에 대해서는 ","type":"text"},{"text":"여기","type":"text","marks":[{"type":"strong"},{"type":"link","attrs":{"href":"https://v1-25.docs.kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#considerations-about-apiserver-advertise-address-and-controlplaneendpoint"}}]},{"text":"를 참고하며,","type":"text"},{"type":"hardBreak"},{"text":"이 환경에서는 Flannel network addon을 사용할 것이다.","type":"text"},{"type":"hardBreak"},{"text":"따라서, ","type":"text"},{"text":"--control-plane-endpoint=","type":"text","marks":[{"type":"code"}]},{"text":"을 지정하며, ","type":"text"},{"text":"--apiserver-advertise-address","type":"text","marks":[{"type":"code"}]},{"text":"에 Pod cluster networking을 위해 NAT Network IP로 API server를 지정한다.","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"참고로, flannel network 대역은 ","type":"text"},{"text":"https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml"}}]},{"text":"에 지정되어 있으며, 디폴트는 ","type":"text"},{"text":"10.244.0.0/16","type":"text","marks":[{"type":"code"}]},{"text":"이다. 만일 호스트에서 사용하는 네트워크 대역과 겹친다면, 겹치지 않도록 지정해야 한다.","type":"text"}]},{"type":"paragraph","content":[{"text":"위 ","type":"text"},{"text":"kube-flannel.yml","type":"text","marks":[{"type":"code"}]},{"text":"파일은 아래 ","type":"text"},{"text":"Installing a Pod network add-on","type":"text","marks":[{"type":"link","attrs":{"href":"https://seversky.atlassian.net/wiki/spaces/CM/pages/527925259/Creating+a+single+control-plane+cluster+with+kubeadm#Installing-a-Pod-network-add-on"}}]},{"text":" 에서 사용된다.","type":"text"}]}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"$ kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.20.2.231\n...\nYour Kubernetes control-plane has initialized successfully!\n\nTo start using your cluster, you need to run the following as a regular user:\n\n mkdir -p $HOME/.kube\n sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\n sudo chown $(id -u):$(id -g) $HOME/.kube/config\n\nYou should now deploy a pod network to the cluster.\nRun \"kubectl apply -f [podnetwork].yaml\" with one of the options listed at:\n https://kubernetes.io/docs/concepts/cluster-administration/addons/\n\nThen you can join any number of worker nodes by running the following on each as root:\n\nkubeadm join 10.20.2.231:6443 --token yii49o.0h5mud78mc6n8x4l \\\n --discovery-token-ca-cert-hash sha256:88b9c2b032e2cebd057ce7af20e9cb8f1ef283ca87b97f27b56a71f285e4c545","type":"text"}]},{"type":"paragraph","content":[{"text":"성공하면, 위와 같이 출력되며, 마지막 출력된 지시를 아래와 같이 따르면 된다.","type":"text"}]},{"type":"paragraph","content":[{"text":"root가 아닌 사용자의 kubectl 사용을 위해 위 출력된 명령을 실행한다.","type":"text"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"mkdir -p $HOME/.kube\nsudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\nsudo chown $(id -u):$(id -g) $HOME/.kube/config","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Alternative Tip","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"root만 사용하길 원한다면, ","type":"text"},{"text":"export KUBECONFIG=/etc/kubernetes/admin.conf","type":"text","marks":[{"type":"code"}]},{"text":"환경 변수를 등록하면 된다.","type":"text"}]}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"kubeadm init","type":"text","marks":[{"type":"code"}]},{"text":"결과 출력의 맨 아래 부분은 추후 worker nodes join을 위해 copy해 두면 편하다.","type":"text"},{"type":"hardBreak"},{"text":"$ kubeadm join 10.20.2.231:6443 --token yii49o.0h5mud78mc6n8x4l \\ --discovery-token-ca-cert-hash sha256:88b9c2b032e2cebd057ce7af20e9cb8f1ef283ca87b97f27b56a71f285e4c545","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"위 token은 디폴트로 24h 후면 만료되며, 추후 필요 시, 아래 명령으로 삭제/생성/조회할 수 있다.","type":"text"},{"type":"hardBreak"},{"text":"--discovery-token-ca-cert-hash","type":"text","marks":[{"type":"code"}]},{"text":"값도 아래와 같이 조회할 수 있다.","type":"text"}]}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"$ kubeadm token delete yii49o.0h5mud78mc6n8x4l\nbootstrap token \"yii49o\" deleted\n\n$ kubeadm token create\nW0728 12:35:47.461305 11391 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]\n6h4vx6.50srly4ep94kfv9\n\n$ kubeadm token list\nTOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS\n6h4vx6.50srly4ep94kfv93 23h 2020-07-29T12:35:47+09:00 authentication,signing system:bootstrappers:kubeadm:default-node-token\n\n# --discovery-token-ca-cert-hash 조회\n$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'\ndd49e1b79dde1b3068d4c794082b8eed0a6e9821c298d6b1d11591bcb75539b2","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Installing a Pod network add-on","type":"text"}]},{"type":"paragraph","content":[{"text":"Pod CNI 네트워크을 위한 Addon을 설치해야 하는데, 여기서는 ","type":"text"},{"text":"Flannel","type":"text","marks":[{"type":"strong"}]},{"text":"를 사용하도록 한다.","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"[중요]","type":"text","marks":[{"type":"strong"}]},{"type":"hardBreak"},{"text":"K8s는 local 혹은 remote pod/node 간 통신을 위해 overlay networking을 사용하는데,","type":"text"},{"type":"hardBreak"},{"text":"이 중 Flannel 네트워킹을 이해하기 위해 ","type":"text"},{"text":"쿠버네티스 파드 네트워크 정리","type":"text","marks":[{"type":"link","attrs":{"href":"https://jonnung.dev/kubernetes/2020/02/24/kubernetes-pod-networking/"}}]},{"text":"를 참고 한다.","type":"text"}]},{"type":"paragraph","content":[{"text":"또한, 기타 k8s 호환 Addons 및 Pod cluster networking 개념에 대해서는 ","type":"text"},{"text":"Networking and Network Policy","type":"text","marks":[{"type":"link","attrs":{"href":"https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy"}}]},{"text":"와 ","type":"text"},{"text":"Cluster Networking","type":"text","marks":[{"type":"link","attrs":{"href":"https://kubernetes.io/docs/concepts/cluster-administration/networking/"}}]},{"text":"를 참조한다.","type":"text"}]}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"$ kubectl create -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml\nnamespace/kube-flannel created\nserviceaccount/flannel created\nclusterrole.rbac.authorization.k8s.io/flannel created\nclusterrolebinding.rbac.authorization.k8s.io/flannel created\nconfigmap/kube-flannel-cfg created\ndaemonset.apps/kube-flannel-ds created\n\n$ kubectl get ds -A -o wide\nNAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR\nkube-flannel kube-flannel-ds 1 1 1 1 1 9m35s kube-flannel docker.io/flannel/flannel:v0.22.0 app=flannel,k8s-app=flannel\nkube-system kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 57m kube-proxy registry.k8s.io/kube-proxy:v1.25.10 k8s-app=kube-proxy\n\n# coredns pods가 READY에 정상 count로 바뀐다.\n$ kubectl get po -A -o wide\nNAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES\nkube-flannel kube-flannel-ds-9xn4n 1/1 Running 0 34m 10.20.2.231 k8s-master-centos8.snetsystems.com \nkube-flannel kube-flannel-ds-l5jn2 1/1 Running 0 34m 10.20.2.232 k8s-worker01-centos8.snetsystems.com \nkube-flannel kube-flannel-ds-wk2kw 1/1 Running 0 34m 10.20.2.233 k8s-worker02-centos8.snetsystems.com \nkube-system coredns-565d847f94-f8bcw 1/1 Running 2 5d5h 10.244.0.5 k8s-master-centos8.snetsystems.com \nkube-system coredns-565d847f94-x42xq 1/1 Running 2 5d5h 10.244.0.4 k8s-master-centos8.snetsystems.com \nkube-system etcd-k8s-master-centos8.snetsystems.com 1/1 Running 2 5d5h 10.20.2.231 k8s-master-centos8.snetsystems.com \nkube-system kube-apiserver-k8s-master-centos8.snetsystems.com 1/1 Running 2 5d5h 10.20.2.231 k8s-master-centos8.snetsystems.com \nkube-system kube-controller-manager-k8s-master-centos8.snetsystems.com 1/1 Running 2 5d5h 10.20.2.231 k8s-master-centos8.snetsystems.com \nkube-system kube-proxy-hjs56 1/1 Running 2 5d4h 10.20.2.233 k8s-worker02-centos8.snetsystems.com \nkube-system kube-proxy-rmz2d 1/1 Running 2 5d4h 10.20.2.232 k8s-worker01-centos8.snetsystems.com \nkube-system kube-proxy-xszfg 1/1 Running 2 5d5h 10.20.2.231 k8s-master-centos8.snetsystems.com \nkube-system kube-scheduler-k8s-master-centos8.snetsystems.com 1/1 Running 2 5d5h 10.20.2.231 k8s-master-centos8.snetsystems.com ","type":"text"}]},{"type":"paragraph","content":[{"text":"위에서 Flannel 및 coredns pod들이 정상적으로 실행(Ready 상태)되는 것을 확인할 수 있다.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Joining the worker node","type":"text"}]},{"type":"paragraph","content":[{"text":"추가할 노드에 ssh 접속 후, 위에 copy해 둔 명령을 실행한다.","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"kubeadm join","type":"text","marks":[{"type":"code"}]},{"text":" 후 아래와 같은 에러가 발생할 경우,","type":"text"},{"type":"hardBreak"},{"text":"error execution phase preflight: [preflight] Some fatal errors occurred:","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":" [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"modprobe br_netfilter","type":"text","marks":[{"type":"code"}]},{"text":"명령을 실행한 후 ","type":"text"},{"text":"kubeadm join","type":"text","marks":[{"type":"code"}]},{"text":"을 수행한다.","type":"text"}]}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"$ modprobe br_netfilter # Optional\n\n$ sudo kubeadm join 10.20.2.231:6443 --token yii49o.0h5mud78mc6n8x4l \\\n> --discovery-token-ca-cert-hash sha256:88b9c2b032e2cebd057ce7af20e9cb8f1ef283ca87b97f27b56a71f285e4c545","type":"text"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"$ kubectl get no -o wide\nNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME\nk8s-master-centos8.snetsystems.com Ready control-plane 155m v1.25.4 10.20.2.231 CentOS Stream 8 4.18.0-492.el8.x86_64 cri-o://1.25.3\nk8s-worker01-centos8.snetsystems.com Ready 71m v1.25.4 10.20.2.232 CentOS Stream 8 4.18.0-492.el8.x86_64 cri-o://1.25.3\nk8s-worker02-centos8.snetsystems.com Ready 64m v1.25.4 10.20.2.233 CentOS Stream 8 4.18.0-492.el8.x86_64 cri-o://1.25.3\n\n# 아래 kube-flannel-ds-amd64의 count들이 추가한 노드 수만큼 증가한다.\nkubectl get ds -A -o wide\nNAMESPACE NAME \n DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR\nkube-flannel kube-flannel-ds 3 3 3 3 3 106m kube-flannel docker.io/flannel/flannel:v0.22.0 app=flannel,k8s-app=flannel\nkube-system kube-proxy 3 3 3 3 3 kubernetes.io/os=linux 154m kube-proxy registry.k8s.io/kube-proxy:v1.25.10 k8s-app=kube-proxy","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Proxying API Server to localhost","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"별도의 SA에 대한 토큰 발급없이 K8s API를 호출할 수 있게 하는 프록시이므로,","type":"text"},{"type":"hardBreak"},{"text":"특별히 테스트를 위한 경우를 제외하고는 쓸 일이 없을 것이다.","type":"text"}]},{"type":"paragraph","content":[{"text":"그냥 넘어가도 무관하며,","type":"text"},{"type":"hardBreak"},{"text":"다음 장에 나오는 적절한 Role를 부여받은 SA-Secret를 생성한 후, Bearer token 헤더를 사용하여 호출하는 것이 정석이다.","type":"text"}]}]},{"type":"codeBlock","content":[{"text":"$ kubectl proxy --address='0.0.0.0' --disable-filter=true\nW0728 13:12:18.967640 24461 proxy.go:167] Request filter disabled, your proxy is vulnerable to XSRF attacks, please be cautious\nStarting to serve on [::]:8001","type":"text"}]},{"type":"expand","attrs":{"title":"사용법 참고"},"content":[{"type":"paragraph","content":[{"text":"If you want to connect to the API Server from outside the cluster you can use ","type":"text"},{"text":"kubectl proxy","type":"text","marks":[{"type":"code"}]},{"text":":","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ scp root@:/etc/kubernetes/admin.conf .\n$ kubectl --kubeconfig ./admin.conf proxy --address='0.0.0.0' --disable-filter=true\nW0728 13:12:18.967640 24461 proxy.go:167] Request filter disabled, your proxy is vulnerable to XSRF attacks, please be cautious\nStarting to serve on [::]:8001","type":"text"}]}]},{"type":"paragraph","content":[{"text":"You can now access the API Server locally at ","type":"text"},{"text":"curl http://10.20.2.231:8001/api/v1","type":"text","marks":[{"type":"code"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Installing Dashboard","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Create Dashboard","type":"text"}]},{"type":"paragraph","content":[{"text":"https://github.com/kubernetes/dashboard/releases","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/kubernetes/dashboard/releases"}}]},{"text":" 에서 원하는 혹은 설치된 k8s control plane 과 호환되는 버전으로 골라서 설치한다. (여기서는 k8s 1.25와 호환되는 ","type":"text"},{"text":"v2.7.0","type":"text","marks":[{"type":"code"}]},{"text":"으로 설치한다.)","type":"text"}]},{"type":"codeBlock","attrs":{"language":"yaml"},"content":[{"text":"$ curl https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml -o dashboard.yml\n\n$ vi dashboard.yaml\n...\nkind: Service\napiVersion: v1\nmetadata:\n labels:\n k8s-app: kubernetes-dashboard\n name: kubernetes-dashboard\n namespace: kubernetes-dashboard\nspec:\n ports:\n - port: 443\n targetPort: 8443\n # 대시보드 접속 port 이므로 기억하기 쉽게 직접 node port 지정\n # 지정하지 않을 경우 임의값 할당됨(Default: 30000-32767)\n nodePort: 30000\n selector:\n k8s-app: kubernetes-dashboard\n type: NodePort # ClusterIP를 NodePort로 변경\n...\nkind: Deployment\n...\n args:\n - --auto-generate-certificates\n - --namespace=kubernetes-dashboard\n - --token-ttl=7776000 # 90일동안 browser no activity session 유지\n\n$ kubectl create -f dashboard.yaml","type":"text"}]},{"type":"paragraph","content":[{"text":"위 dashboard를 생성하고 나면, k8s cluster 內 모든 노드의 30000번 포트로 접속이 가능할 것이다.","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center"},"content":[{"type":"media","attrs":{"width":1450,"id":"5e6b4610-0a1c-412d-93e1-f25c07bfb7b6","collection":"contentId-527925259","type":"file","height":319}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"권한 부여","type":"text"}]},{"type":"paragraph","content":[{"text":"Dashboard의 모든 기능을 사용하기 위해서 적절한 이름의 ServiceAccount를 만들고 cluster admin 권한을 부여하도록 한다.","type":"text"},{"type":"hardBreak"},{"type":"status","attrs":{"color":"red","style":"bold","text":"주의","localId":"f9a65563-a0ea-403d-a3d2-ec7910079344"}},{"text":" ","type":"text"},{"text":"K8s 1.24부터 SA 생성과 동시에 디폴트 token secret이 자동 생성되지 않음에 유의 바람.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}}]}]},{"type":"paragraph","content":[{"text":"아래와 같이 SA, Token, ClusterRoleBinding을 생성.","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"type":"inlineCard","attrs":{"url":"https://github.com/snetsystems/K8s-Objects/blob/master/addons/admin-user.yaml"}},{"text":" ","type":"text"}]}]},{"type":"codeBlock","attrs":{"language":"yaml"},"content":[{"text":"apiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: admin-user\n namespace: kube-system\n\n---\n\napiVersion: v1\nkind: Secret\nmetadata:\n name: admin-user-secret\n namespace: kube-system\n annotations:\n kubernetes.io/service-account.name: admin-user\ntype: kubernetes.io/service-account-token\n\n---\n\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: admin-user\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: cluster-admin\nsubjects:\n- kind: ServiceAccount\n name: admin-user\n namespace: kube-system","type":"text"}]},{"type":"codeBlock","attrs":{"language":"yaml"},"content":[{"text":"$ kubectl describe secrets -n kube-system admin-user-secret\nName: admin-user-secret\nNamespace: kube-system\nLabels: \nAnnotations: kubernetes.io/service-account.name: admin-user\n kubernetes.io/service-account.uid: a4ffa4e6-55ad-4c41-83c8-8a585137ff75\n\nType: kubernetes.io/service-account-token\n\nData\n====\nca.crt: 1099 bytes\nnamespace: 11 bytes\ntoken: eyJhbGciOiJSUzI1NiIsIm...blah blah ","type":"text"}]},{"type":"paragraph","content":[{"text":"위의 token으로 로그인하면, 모든 기능에 접근 가능할 것이다.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Testing a simple Nginx deployment","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"이하 글은 ","type":"text"},{"text":"k8s-01","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"k8s-02","type":"text","marks":[{"type":"code"}]},{"text":"에 test한 이전 예제로서, 차차 업데이트 할 예정이다.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"테스트를 위해 간단한 Nginx deployment를 해보자.","type":"text"},{"type":"hardBreak"},{"text":"먼저 아래와 같이 스펙을 작성한다.","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"type":"inlineCard","attrs":{"url":"https://github.com/snetsystems/K8s-Objects/blob/master/tests/nginx-deployment.yaml"}},{"text":" ","type":"text"}]}]},{"type":"codeBlock","attrs":{"language":"yaml"},"content":[{"text":"$ nginx-deployment.yaml\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: nginx\n namespace: cloudhub\n labels:\n app: nginx\nspec:\n replicas: 2\n selector:\n matchLabels:\n app: nginx\n template:\n metadata:\n labels:\n app: nginx\n spec:\n containers:\n - name: nginx\n image: nginx:1.14.2\n ports:\n - containerPort: 80\n---\napiVersion: v1\nkind: Service\nmetadata:\n name: nginx-svc-lb\n namespace: cloudhub\nspec:\n selector:\n app: nginx\n type: LoadBalancer\n ports:\n - name: http\n port: 8080\n protocol: TCP\n targetPort: 80\n nodePort: 30037\n externalIPs:\n - 10.20.2.231\n---\napiVersion: v1\nkind: Service\nmetadata:\n name: nginx-svc-np\n namespace: cloudhub\nspec:\n selector:\n app: nginx\n type: NodePort\n ports:\n - name: http\n port: 8080\n protocol: TCP\n targetPort: 80\n nodePort: 30036","type":"text"}]},{"type":"paragraph","content":[{"text":"kubectl create -f nginx-deployment.yaml","type":"text","marks":[{"type":"code"}]},{"text":"을 실행한 후, 아래와 같이 ","type":"text"},{"text":"k8s-worker01-centos8","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"k8s-worker02-centos8","type":"text","marks":[{"type":"code"}]},{"text":"에 nginx가 생성된 것을 확인할 수 있다.","type":"text"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"$ kubectl get pod -A -o wide\nNAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES\ncloudhub nginx-7fb96c846b-8thr4 1/1 Running 0 5m38s 10.244.2.3 k8s-worker02-centos8.snetsystems.com \ncloudhub nginx-7fb96c846b-k2zhb 1/1 Running 0 5m38s 10.244.1.3 k8s-worker01-centos8.snetsystems.com \nkube-flannel kube-flannel-ds-9xn4n 1/1 Running 0 43h 10.20.2.231 k8s-master-centos8.snetsystems.com \nkube-flannel kube-flannel-ds-l5jn2 1/1 Running 0 43h 10.20.2.232 k8s-worker01-centos8.snetsystems.com \nkube-flannel kube-flannel-ds-wk2kw 1/1 Running 0 43h 10.20.2.233 k8s-worker02-centos8.snetsystems.com \nkube-system coredns-565d847f94-f8bcw 1/1 Running 2 7d 10.244.0.5 k8s-master-centos8.snetsystems.com \nkube-system coredns-565d847f94-x42xq 1/1 Running 2 7d 10.244.0.4 k8s-master-centos8.snetsystems.com \nkube-system etcd-k8s-master-centos8.snetsystems.com 1/1 Running 2 7d 10.20.2.231 k8s-master-centos8.snetsystems.com \nkube-system kube-apiserver-k8s-master-centos8.snetsystems.com 1/1 Running 2 7d 10.20.2.231 k8s-master-centos8.snetsystems.com \nkube-system kube-controller-manager-k8s-master-centos8.snetsystems.com 1/1 Running 2 7d 10.20.2.231 k8s-master-centos8.snetsystems.com \nkube-system kube-proxy-hjs56 1/1 Running 2 6d22h 10.20.2.233 k8s-worker02-centos8.snetsystems.com \nkube-system kube-proxy-rmz2d 1/1 Running 2 6d22h 10.20.2.232 k8s-worker01-centos8.snetsystems.com \nkube-system kube-proxy-xszfg 1/1 Running 2 7d 10.20.2.231 k8s-master-centos8.snetsystems.com \nkube-system kube-scheduler-k8s-master-centos8.snetsystems.com 1/1 Running 2 7d 10.20.2.231 k8s-master-centos8.snetsystems.com \nkubernetes-dashboard dashboard-metrics-scraper-664484847b-5497m 1/1 Running 0 41h 10.244.0.7 k8s-master-centos8.snetsystems.com \nkubernetes-dashboard kubernetes-dashboard-7d66467dc7-bv9fn 1/1 Running 0 41h 10.244.0.6 k8s-master-centos8.snetsystems.com ","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"K8s Control plane","type":"text","marks":[{"type":"strong"}]},{"text":"에는 디폴트로 일반 pod가 배치될 수 없도록 ","type":"text"},{"text":"taint처리","type":"text","marks":[{"type":"strong"}]},{"text":"되어 있다.","type":"text"}]},{"type":"paragraph","content":[{"text":"kubectl describe no k8s-master-centos8","type":"text","marks":[{"type":"code"}]},{"text":"를 통해 ","type":"text"},{"text":"Taints: node-role.kubernetes.io/control-plane:NoSchedule","type":"text","marks":[{"type":"code"}]},{"text":"를 확인할 수 있다.","type":"text"}]},{"type":"paragraph","content":[{"text":"따라서, 위 예제에서 nginx는 각각 ","type":"text"},{"text":"k8s-worker01-centos8","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"k8s-worker02-centos8","type":"text","marks":[{"type":"code"}]},{"text":"에 배치되며, IP 또한 ","type":"text"},{"text":"k8s-worker01[02]-centos8","type":"text","marks":[{"type":"code"}]},{"text":"의 ","type":"text"},{"text":"flannel.1","type":"text","marks":[{"type":"code"}]},{"text":"을 통한 ","type":"text"},{"text":"cni0","type":"text","marks":[{"type":"code"}]},{"text":" 인터페이스에 연결되어 있음을 알 수 있다.","type":"text"},{"type":"hardBreak"},{"text":"ip a","type":"text","marks":[{"type":"code"}]},{"text":"혹은 ","type":"text"},{"text":"ifconfig","type":"text","marks":[{"type":"code"}]},{"text":" 명령을 통해 확인할 수 있다.","type":"text"}]},{"type":"paragraph","content":[{"text":"taint","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"nodeSelector","type":"text","marks":[{"type":"code"}]},{"text":" 및 ","type":"text"},{"text":"tolerations","type":"text","marks":[{"type":"code"}]},{"text":" 등 target node 지정 방법과 관련해서는 다음 장인 ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://seversky.atlassian.net/wiki/spaces/CHT/pages/549847065"}},{"text":" 에 간략한 예시가 있다.","type":"text"},{"type":"hardBreak"},{"text":"크게 어려운 개념은 아니지만 알고 있어야 하는 내용이라, detail은 ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://v1-25.docs.kubernetes.io/ko/docs/concepts/scheduling-eviction/taint-and-toleration/"}},{"text":" 를 참고 하길 바란다.","type":"text"}]}]},{"type":"paragraph"},{"type":"panel","attrs":{"panelIconId":"atlassian-note","panelIcon":":note:","panelColor":"#FFF0B3","panelType":"custom"},"content":[{"type":"paragraph","content":[{"text":"서비스 및 서비스 디스커버리에 대한 자세한 내용은 ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://seversky.atlassian.net/wiki/spaces/CHT/pages/2114289730"}},{"text":" 에서 자세히 다룬다.","type":"text"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Clean up Node & Control Plane","type":"text"}]},{"type":"paragraph","content":[{"text":"Ref: ","type":"text"},{"text":"Creating a single control-plane cluster with kubeadm: Clean up","type":"text","marks":[{"type":"link","attrs":{"href":"https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#tear-down"}}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Reset worker node","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"On worker node:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"$ sudo kubeadm reset\n[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.\n[reset] Are you sure you want to proceed? [y/N]: y\n[preflight] Running pre-flight checks\nW0728 09:51:00.050546 6570 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory\n[reset] No etcd config found. Assuming external etcd\n[reset] Please, manually reset etcd to prevent further issues\n[reset] Stopping the kubelet service\n[reset] Unmounting mounted directories in \"/var/lib/kubelet\"\n[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]\n[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]\n[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]\n\nThe reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d\n\nThe reset process does not reset or clean up iptables rules or IPVS tables.\nIf you wish to reset iptables, you must do so manually by using the \"iptables\" command.\n\nIf your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)\nto reset your system's IPVS tables.\n\nThe reset process does not clean your kubeconfig files and you must remove them manually.\nPlease, check the contents of the $HOME/.kube/config file.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"On control plane:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"$ kubectl delete node k8s-worker01-centos8.snetsystems.com\nnode \"k8s-worker01-centos8.snetsystems.com\" deleted","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Reset Control Plane","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"On control plane:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"python"},"content":[{"text":"$ sudo kubeadm reset\nkubeadm reset\n[reset] Reading configuration from the cluster...\n[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\n[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.\n[reset] Are you sure you want to proceed? [y/N]: y\n[preflight] Running pre-flight checks\n[reset] Removing info for node \"k8s-worker01-centos8.snetsystems.com\" from the ConfigMap \"kubeadm-config\" in the \"kube-system\" Namespace\n{\"level\":\"warn\",\"ts\":\"2020-07-28T10:00:47.620+0900\",\"caller\":\"clientv3/retry_interceptor.go:61\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"endpoint://client-cf1f1707-af99-4f83-b813-139c1cc6130f/10.0.2.5:2379\",\"attempt\":0,\"error\":\"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members\"}\n{\"level\":\"warn\",\"ts\":\"2020-07-28T10:00:47.675+0900\",\"caller\":\"clientv3/retry_interceptor.go:61\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"endpoint://client-cf1f1707-af99-4f83-b813-139c1cc6130f/10.0.2.5:2379\",\"attempt\":0,\"error\":\"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members\"}\n{\"level\":\"warn\",\"ts\":\"2020-07-28T10:00:47.789+0900\",\"caller\":\"clientv3/retry_interceptor.go:61\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"endpoint://client-cf1f1707-af99-4f83-b813-139c1cc6130f/10.0.2.5:2379\",\"attempt\":0,\"error\":\"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members\"}\n{\"level\":\"warn\",\"ts\":\"2020-07-28T10:00:48.005+0900\",\"caller\":\"clientv3/retry_interceptor.go:61\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"endpoint://client-cf1f1707-af99-4f83-b813-139c1cc6130f/10.0.2.5:2379\",\"attempt\":0,\"error\":\"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members\"}\n{\"level\":\"warn\",\"ts\":\"2020-07-28T10:00:48.426+0900\",\"caller\":\"clientv3/retry_interceptor.go:61\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"endpoint://client-cf1f1707-af99-4f83-b813-139c1cc6130f/10.0.2.5:2379\",\"attempt\":0,\"error\":\"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members\"}\n{\"level\":\"warn\",\"ts\":\"2020-07-28T10:00:49.261+0900\",\"caller\":\"clientv3/retry_interceptor.go:61\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"endpoint://client-cf1f1707-af99-4f83-b813-139c1cc6130f/10.0.2.5:2379\",\"attempt\":0,\"error\":\"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members\"}\n{\"level\":\"warn\",\"ts\":\"2020-07-28T10:00:50.974+0900\",\"caller\":\"clientv3/retry_interceptor.go:61\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"endpoint://client-cf1f1707-af99-4f83-b813-139c1cc6130f/10.0.2.5:2379\",\"attempt\":0,\"error\":\"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members\"}\n{\"level\":\"warn\",\"ts\":\"2020-07-28T10:00:54.197+0900\",\"caller\":\"clientv3/retry_interceptor.go:61\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"endpoint://client-cf1f1707-af99-4f83-b813-139c1cc6130f/10.0.2.5:2379\",\"attempt\":0,\"error\":\"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members\"}\n{\"level\":\"warn\",\"ts\":\"2020-07-28T10:01:00.701+0900\",\"caller\":\"clientv3/retry_interceptor.go:61\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"endpoint://client-cf1f1707-af99-4f83-b813-139c1cc6130f/10.0.2.5:2379\",\"attempt\":0,\"error\":\"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members\"}\n{\"level\":\"warn\",\"ts\":\"2020-07-28T10:01:13.628+0900\",\"caller\":\"clientv3/retry_interceptor.go:61\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"endpoint://client-cf1f1707-af99-4f83-b813-139c1cc6130f/10.0.2.5:2379\",\"attempt\":0,\"error\":\"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members\"}\n{\"level\":\"warn\",\"ts\":\"2020-07-28T10:01:39.999+0900\",\"caller\":\"clientv3/retry_interceptor.go:61\",\"msg\":\"retrying of unary invoker failed\",\"target\":\"endpoint://client-cf1f1707-af99-4f83-b813-139c1cc6130f/10.0.2.5:2379\",\"attempt\":0,\"error\":\"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members\"}\nW0728 10:01:39.999707 29189 removeetcdmember.go:61] [reset] failed to remove etcd member: etcdserver: re-configuration failed due to not enough started members\n.Please manually remove this etcd member using etcdctl\n[reset] Stopping the kubelet service\n[reset] Unmounting mounted directories in \"/var/lib/kubelet\"\n[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]\n[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]\n[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]\n\nThe reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d\n\nThe reset process does not reset or clean up iptables rules or IPVS tables.\nIf you wish to reset iptables, you must do so manually by using the \"iptables\" command.\n\nIf your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)\nto reset your system's IPVS tables.\n\nThe reset process does not clean your kubeconfig files and you must remove them manually.\nPlease, check the contents of the $HOME/.kube/config file.","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Reset command 이후, ","type":"text"},{"text":"$HOME/.kube/config","type":"text","marks":[{"type":"code"}]},{"text":"과 같이 수동으로 정리해야 할 내용들이 출력 된다.","type":"text"},{"type":"hardBreak"},{"text":"따라서, 필요할 경우 출력된 지시대로 수행해야 한다.","type":"text"}]}]}],"version":1}