| Table of Contents |
|---|
개요
Kubernetes의 Namespace, Node, Pod 등의 각종 정보를 모니터링 합니다.
...
화면 설명
...
“ⓐ” “ⓐ” K8s Filter
“ⓑ” “ⓑ” K8s API 호출 정보
“ⓒ” “ⓒ” K8s Object UI
“ⓓ” “ⓓ” K8s Object Detail 정보
“ⓔ” “ⓔ” K8s Object 시계열 데이터 사각화 정보
Kubernetes 조회
| Info |
|---|
CloudHub 1.4.0 버전 이상에서는 Minion ID를 "ch-collector"로 설정해야 Kubernetes의 SaltStack Rest API를 호출할 Target Minion으로 사용할 수 있습니다. |
K8s를 호출할 Target 정보를 선택합니다.
K8s 호출 주기를 선택합니다.
...
Kubernetes Object 시계열 데이터 조회
K8s UI 화면에서 Object 클릭시 화면 하단에 해당 Object의 시계열 데이터가 시각화되어 조회되고 해당 Object에 마우스 오버시 Tooltip 형태로 화면에 CPU, Memory의 사용량을 확인 할 수 있습니다.
Object 마우스 클릭
...
Object 마우스 오버
...
Telegraf input plugin modules
Via k8s API server on Control plane node.
Daemonsets, deployments, nodes, pods 등 k8s를 구성하고 있는 요소들에 대한 실시간 상태 정보 및 설정 정보(resource_requests_cpu_units, resource_limits_cpu_units 등과 같이 metric data only) 수집.
Via kubelet API on each node.
Nodes, pods 등의 CPU, Memory, Traffic 등의 metric data 수집.
Configurations of Telegraf
Monitoring 권한 생성
먼저, k8s에 이미 설정된 혹은 추후 설정될 다른 account 및 role들과 혼동되는 것을 대비하기 위해,
별도의 namespace, service account, cluster role 등을 추가한다.
| Code Block |
|---|
$ vim cloudhub_role.yaml
---
# 별도의 Namespace를 만든다.
apiVersion: v1
kind: Namespace
metadata:
name: cloudhub
---
# cloudhub ServiceAccount를 만든다.
apiVersion: v1
kind: ServiceAccount
metadata:
name: cloudhub
namespace: cloudhub
---
# API에 대한 접근 권한을 만든다.
# 대충 resources endpoint에 대해 verbs 권한을 주겠다..라는 뜻이다.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: snetsystems:cluster:viewer
labels:
rbac.authorization.k8s.io/aggregate-view-cloudhub: "true"
rules:
- verbs:
- get
- list
apiGroups:
- ''
resources:
- persistentvolumes
- nodes
- nodes/stats
- nodes/proxy
---
# 위의 권한에 system:aggregate-to-view의 내용을 합한다.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: snetsystems:cloudhub
aggregationRule:
clusterRoleSelectors:
- matchLabels:
rbac.authorization.k8s.io/aggregate-view-cloudhub: "true"
- matchLabels:
rbac.authorization.k8s.io/aggregate-to-view: "true"
rules: [] # Rules are automatically filled in by the controller manager.
---
# 위에서 최종 합한 권한을 cloudhub 계정에 바인딩 한다.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: snetsystems:cloudhub:viewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: snetsystems:cloudhub
subjects:
- kind: ServiceAccount
name: cloudhub
namespace: cloudhub
# 위 내용들을 k8s에 적용/생성한다.
$ kubectl apply -f cloudhub_role.yaml |
Configure telegraf
위의 account를 생성하면, secret도 같이 생성되며, 그 안에 토큰이 발급되어 있습니다.
| Code Block |
|---|
$ kubectl get secrets -n cloudhub
NAME TYPE DATA AGE
cloudhub-token-wdkkx kubernetes.io/service-account-token 3 22h
default-token-lqcc8 kubernetes.io/service-account-token 3 22h
$ kubectl describe secrets -n cloudhub cloudhub-token-wdkkx
Name: cloudhub-token-wdkkx
Namespace: cloudhub
Labels: <none>
Annotations: kubernetes.io/service-account.name: cloudhub
kubernetes.io/service-account.uid: 9ec8dfc4-3129-43ec-ab67-baf955a00842
Type: kubernetes.io/service-account-token
Data
====
namespace: 8 bytes
token: GciOiJSUzI1NiIsImtpZCI6IjNqSm8xaXJ0MDZsaGxjdzVndWozY1A5VXBGbTdwX3VDUzBpd0J2a3ItR0EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbG91ZGh1Yi10b2tlbi13ZGtreCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjllYzhkZmM0LTMxMjktNDNlYy1hYjY3LWJhZjk1NWEwMDg0MiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjbG91ZGh1YjpjbG91ZGh1YiJ9.OXLsPxs2h_frtDPdJcB2XXl-GZa4UuEcRs5DHBqbJRzWPOoMTlq_loApX2iyeT-TzWekdnpHSTlEMpNztNkTlOPBfGqg-_ouXG_s9EiDhMxpOtK-bmnXP2FvLBqcf0J_rX4aiSkt7CitVktCCVh8m6CIMtrk3nPbE0k8qX87NC_8UtLp67-25wQ9DkQnpkLxRhbnFmhR0VNTPPuHwxOz5xqwTIq4uc0AHxvelCoHS2ebA9mcXovf0UT97ajotd0NXMz8N96JcWOyVCODvZkIXmS_sGiP6jq8TnTkgvat3ZeyaKeQTqjW7kM0yHLbEgcF8fRHZzyoxBDT2ovxVGxbFQ
ca.crt: 1025 bytes |
위 토큰을 k8s api 사용 권한 인증을 위해 아래 telegraf.conf에서 사용합니다.
kube_inventory plugin
| Code Block |
|---|
[global_tags]
dc = "snet-s2f" # will tag all metrics with dc=us-east-1
rack = "VMs"
...
[agent]
## 수집되는 항목/내용이 많으므로, 일단 1분으로 하였다.
interval = "1m"
...
# Read metrics from the Kubernetes api
[[inputs.kube_inventory]]
## URL for the Kubernetes API
## 일단, k8s master node에 설치하였기 때문에 localhost로 한다.
url = "https://localhost:6443"
## Namespace to use. Set to "" to use all namespaces.
namespace = ""
## Use bearer token for authorization. ('bearer_token' takes priority)
## If both of these are empty, we'll use the default serviceaccount:
## at: /run/secrets/kubernetes.io/serviceaccount/token
# bearer_token = "/path/to/bearer/token"
## OR
bearer_token_string = "GciOiJSUzI1NiIsImtpZCI6IjNqSm8xaXJ0MDZsaGxjdzVndWozY1A5VXBGbTdwX3VDUzBpd0J2a3ItR0EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbG91ZGh1Yi10b2tlbi13ZGtreCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjllYzhkZmM0LTMxMjktNDNlYy1hYjY3LWJhZjk1NWEwMDg0MiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjbG91ZGh1YjpjbG91ZGh1YiJ9.OXLsPxs2h_frtDPdJcB2XXl-GZa4UuEcRs5DHBqbJRzWPOoMTlq_loApX2iyeT-TzWekdnpHSTlEMpNztNkTlOPBfGqg-_ouXG_s9EiDhMxpOtK-bmnXP2FvLBqcf0J_rX4aiSkt7CitVktCCVh8m6CIMtrk3nPbE0k8qX87NC_8UtLp67-25wQ9DkQnpkLxRhbnFmhR0VNTPPuHwxOz5xqwTIq4uc0AHxvelCoHS2ebA9mcXovf0UT97ajotd0NXMz8N96JcWOyVCODvZkIXmS_sGiP6jq8TnTkgvat3ZeyaKeQTqjW7kM0yHLbEgcF8fRHZzyoxBDT2ovxVGxbFQ"
## Set response_timeout (default 5 seconds)
response_timeout = "5s"
## Optional Resources to exclude from gathering
## Leave them with blank with try to gather everything available.
## Values can be - "daemonsets", deployments", "endpoints", "ingress", "nodes",
## "persistentvolumes", "persistentvolumeclaims", "pods", "services", "statefulsets"
#resource_exclude = [ "deployments" ]
## Optional Resources to include when gathering
## Overrides resource_exclude if both set.
#resource_include = [ "daemonsets", "deployments", "endpoints", "ingress", "persistentvolumes", "persistentvolumeclaims", "nodes", "pods", "services", "statefulsets" ]
## Optional TLS Config
# tls_ca = "/path/to/cafile"
# tls_cert = "/path/to/certfile"
# tls_key = "/path/to/keyfile"
## Use TLS but skip chain & host verification
insecure_skip_verify = true
# fielddrop = ["created"] |
kubernetes plugin
| Code Block |
|---|
[global_tags]
dc = "snet-s2f" # will tag all metrics with dc=us-east-1
rack = "VMs"
...
[agent]
## 수집되는 항목/내용이 많으므로, 일단 1분으로 하였다.
interval = "1m"
...
[[inputs.kubernetes]]
## URL and port for the kubelet
url = "https://localhost:10250"
## Use bearer token for authorization. ('bearer_token' takes priority)
## If both of these are empty, we'll use the default serviceaccount:
## at: /run/secrets/kubernetes.io/serviceaccount/token
# bearer_token = "/path/to/bearer/token"
## OR
bearer_token_string = "GciOiJSUzI1NiIsImtpZCI6IjNqSm8xaXJ0MDZsaGxjdzVndWozY1A5VXBGbTdwX3VDUzBpd0J2a3ItR0EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbG91ZGh1Yi10b2tlbi13ZGtreCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjllYzhkZmM0LTMxMjktNDNlYy1hYjY3LWJhZjk1NWEwMDg0MiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjbG91ZGh1YjpjbG91ZGh1YiJ9.OXLsPxs2h_frtDPdJcB2XXl-GZa4UuEcRs5DHBqbJRzWPOoMTlq_loApX2iyeT-TzWekdnpHSTlEMpNztNkTlOPBfGqg-_ouXG_s9EiDhMxpOtK-bmnXP2FvLBqcf0J_rX4aiSkt7CitVktCCVh8m6CIMtrk3nPbE0k8qX87NC_8UtLp67-25wQ9DkQnpkLxRhbnFmhR0VNTPPuHwxOz5xqwTIq4uc0AHxvelCoHS2ebA9mcXovf0UT97ajotd0NXMz8N96JcWOyVCODvZkIXmS_sGiP6jq8TnTkgvat3ZeyaKeQTqjW7kM0yHLbEgcF8fRHZzyoxBDT2ovxVGxbFQ"
## Pod labels to be added as tags. An empty array for both include and
## exclude will include all labels.
# label_include = []
# label_exclude = ["*"]
## Set response_timeout (default 5 seconds)
# response_timeout = "5s"
## Optional TLS Config
# tls_ca = /path/to/cafile
#tls_cert = "/var/lib/minikube/certs/apiserver-kubelet-client.crt"
#tls_key = "/var/lib/minikube/certs/apiserver-kubelet-client.key"
## Use TLS but skip chain & host verification
insecure_skip_verify = true |