개요

Kubernetes의 Namespace, Node, Pod 등의 각종 정보를 모니터링 합니다.

화면 설명

“ⓐ” K8s Filter
“ⓑ” K8s API 호출 정보
“ⓒ” K8s Object UI
“ⓓ” K8s Object Detail 정보
“ⓔ” K8s Object 시계열 데이터 사각화 정보

Kubernetes 조회

K8s를 호출할 Target 정보를 선택합니다.
K8s 호출 주기를 선택합니다.

아래와 같이 K8s 정보가 조회 됩니다.

Kubernetes Filtering

Namespace, Node, Label, Limit Count를 선택해서 필터링 하여 조회 할 수 있습니다.

조회 결과

Kubernetes Object 간 포함 관계 조회

K8s Object 더블 클릭시 연관된 Object를 찾아 노란색 테두리로 표시해 줍니다.
연관된 Object 표시 해제는 hexagon Object 아닌 Circle Object 더블 클릭하거나 노란색 테두리로 표시된 Object를 더블 클릭하여 해제 할 수 있습니다.

(Deployment) > RS > Pod
[RS or RC or StatefulSet or DaemonSet] > Pod
(CronJob) > Job > Pod
Ingress > SVC > Pod
SVC > Pod

Kubernetes Object Details 정보 조회

K8s UI 화면에서 Object 클릭시 화면 오른쪽 Details에 해당 Object의 Detail 정보가 조회 됩니다.

Kubernetes Object 시계열 데이터 조회

K8s UI 화면에서 Object 클릭시 화면 하단에 해당 Object의 시계열 데이터가 시각화되어 조회되고 해당 Object에 마우스 오버시 Tooltip 형태로 화면에 CPU, Memory의 사용량을 확인 할 수 있습니다.

Object 마우스 클릭

Object 마우스 오버

Telegraf input plugin modules

kube_inventory
- Via k8s API server on Control plane node.
- Daemonsets, deployments, nodes, pods 등 k8s를 구성하고 있는 요소들에 대한 실시간 상태 정보 및 설정 정보(resource_requests_cpu_units, resource_limits_cpu_units 등과 같이 metric data only) 수집.
kubernetes
- Via kubelet API on each node.
- Nodes, pods 등의 CPU, Memory, Traffic 등의 metric data 수집.

Configurations of Telegraf

Monitoring 권한 생성

먼저, k8s에 이미 설정된 혹은 추후 설정될 다른 account 및 role들과 혼동되는 것을 대비하기 위해,
별도의 namespace, service account, cluster role 등을 추가한다.

$ vim cloudhub_role.yaml
---
# 별도의 Namespace를 만든다.
apiVersion: v1
kind: Namespace
metadata:
  name: cloudhub
---
# cloudhub ServiceAccount를 만든다.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cloudhub
  namespace: cloudhub
---
# API에 대한 접근 권한을 만든다.
# 대충 resources endpoint에 대해 verbs 권한을 주겠다..라는 뜻이다.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: snetsystems:cluster:viewer
  labels:
    rbac.authorization.k8s.io/aggregate-view-cloudhub: "true"
rules:
  - verbs:
      - get
      - list
    apiGroups:
      - ''
    resources:
      - persistentvolumes
      - nodes
      - nodes/stats
      - nodes/proxy
---
# 위의 권한에 system:aggregate-to-view의 내용을 합한다.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: snetsystems:cloudhub
aggregationRule:
  clusterRoleSelectors:
    - matchLabels:
        rbac.authorization.k8s.io/aggregate-view-cloudhub: "true"
    - matchLabels:
        rbac.authorization.k8s.io/aggregate-to-view: "true"
rules: [] # Rules are automatically filled in by the controller manager.
---
# 위에서 최종 합한 권한을 cloudhub 계정에 바인딩 한다.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: snetsystems:cloudhub:viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: snetsystems:cloudhub
subjects:
  - kind: ServiceAccount
    name: cloudhub 
    namespace: cloudhub

# 위 내용들을 k8s에 적용/생성한다.
$ kubectl apply -f cloudhub_role.yaml

Configure telegraf

위의 account를 생성하면, secret도 같이 생성되며, 그 안에 토큰이 발급되어 있습니다.

$ kubectl get secrets -n cloudhub
NAME                   TYPE                                  DATA   AGE
cloudhub-token-wdkkx   kubernetes.io/service-account-token   3      22h
default-token-lqcc8    kubernetes.io/service-account-token   3      22h

$ kubectl describe secrets -n cloudhub cloudhub-token-wdkkx
Name:         cloudhub-token-wdkkx
Namespace:    cloudhub
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: cloudhub
              kubernetes.io/service-account.uid: 9ec8dfc4-3129-43ec-ab67-baf955a00842

Type:  kubernetes.io/service-account-token

Data
====
namespace:  8 bytes
token:      GciOiJSUzI1NiIsImtpZCI6IjNqSm8xaXJ0MDZsaGxjdzVndWozY1A5VXBGbTdwX3VDUzBpd0J2a3ItR0EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbG91ZGh1Yi10b2tlbi13ZGtreCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjllYzhkZmM0LTMxMjktNDNlYy1hYjY3LWJhZjk1NWEwMDg0MiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjbG91ZGh1YjpjbG91ZGh1YiJ9.OXLsPxs2h_frtDPdJcB2XXl-GZa4UuEcRs5DHBqbJRzWPOoMTlq_loApX2iyeT-TzWekdnpHSTlEMpNztNkTlOPBfGqg-_ouXG_s9EiDhMxpOtK-bmnXP2FvLBqcf0J_rX4aiSkt7CitVktCCVh8m6CIMtrk3nPbE0k8qX87NC_8UtLp67-25wQ9DkQnpkLxRhbnFmhR0VNTPPuHwxOz5xqwTIq4uc0AHxvelCoHS2ebA9mcXovf0UT97ajotd0NXMz8N96JcWOyVCODvZkIXmS_sGiP6jq8TnTkgvat3ZeyaKeQTqjW7kM0yHLbEgcF8fRHZzyoxBDT2ovxVGxbFQ
ca.crt:     1025 bytes

위 토큰을 k8s api 사용 권한 인증을 위해 아래 telegraf.conf에서 사용합니다.

kube_inventory plugin

[global_tags]
  dc = "snet-s2f" # will tag all metrics with dc=us-east-1
  rack = "VMs"
...
[agent]
  ## 수집되는 항목/내용이 많으므로, 일단 1분으로 하였다.
  interval = "1m"
...
# Read metrics from the Kubernetes api
[[inputs.kube_inventory]]
  ## URL for the Kubernetes API
  ## 일단, k8s master node에 설치하였기 때문에 localhost로 한다.
  url = "https://localhost:6443"

  ## Namespace to use. Set to "" to use all namespaces.
  namespace = ""

  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## If both of these are empty, we'll use the default serviceaccount:
  ## at: /run/secrets/kubernetes.io/serviceaccount/token
  # bearer_token = "/path/to/bearer/token"
  ## OR
  bearer_token_string = "GciOiJSUzI1NiIsImtpZCI6IjNqSm8xaXJ0MDZsaGxjdzVndWozY1A5VXBGbTdwX3VDUzBpd0J2a3ItR0EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbG91ZGh1Yi10b2tlbi13ZGtreCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjllYzhkZmM0LTMxMjktNDNlYy1hYjY3LWJhZjk1NWEwMDg0MiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjbG91ZGh1YjpjbG91ZGh1YiJ9.OXLsPxs2h_frtDPdJcB2XXl-GZa4UuEcRs5DHBqbJRzWPOoMTlq_loApX2iyeT-TzWekdnpHSTlEMpNztNkTlOPBfGqg-_ouXG_s9EiDhMxpOtK-bmnXP2FvLBqcf0J_rX4aiSkt7CitVktCCVh8m6CIMtrk3nPbE0k8qX87NC_8UtLp67-25wQ9DkQnpkLxRhbnFmhR0VNTPPuHwxOz5xqwTIq4uc0AHxvelCoHS2ebA9mcXovf0UT97ajotd0NXMz8N96JcWOyVCODvZkIXmS_sGiP6jq8TnTkgvat3ZeyaKeQTqjW7kM0yHLbEgcF8fRHZzyoxBDT2ovxVGxbFQ"

  ## Set response_timeout (default 5 seconds)
  response_timeout = "5s"

  ## Optional Resources to exclude from gathering
  ## Leave them with blank with try to gather everything available.
  ## Values can be - "daemonsets", deployments", "endpoints", "ingress", "nodes",
  ## "persistentvolumes", "persistentvolumeclaims", "pods", "services", "statefulsets"
  #resource_exclude = [ "deployments" ]

  ## Optional Resources to include when gathering
  ## Overrides resource_exclude if both set.
  #resource_include = [ "daemonsets", "deployments", "endpoints", "ingress", "persistentvolumes", "persistentvolumeclaims", "nodes", "pods", "services", "statefulsets" ]

  ## Optional TLS Config
  # tls_ca = "/path/to/cafile"
  # tls_cert = "/path/to/certfile"
  # tls_key = "/path/to/keyfile"
  ## Use TLS but skip chain & host verification
  insecure_skip_verify = true

  # fielddrop = ["created"]

kubernetes plugin

[global_tags]
  dc = "snet-s2f" # will tag all metrics with dc=us-east-1
  rack = "VMs"
...
[agent]
  ## 수집되는 항목/내용이 많으므로, 일단 1분으로 하였다.
  interval = "1m"
...

[[inputs.kubernetes]]
  ## URL and port for the kubelet
  url = "https://localhost:10250"

  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## If both of these are empty, we'll use the default serviceaccount:
  ## at: /run/secrets/kubernetes.io/serviceaccount/token
  # bearer_token = "/path/to/bearer/token"
  ## OR
  bearer_token_string = "GciOiJSUzI1NiIsImtpZCI6IjNqSm8xaXJ0MDZsaGxjdzVndWozY1A5VXBGbTdwX3VDUzBpd0J2a3ItR0EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbG91ZGh1Yi10b2tlbi13ZGtreCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjllYzhkZmM0LTMxMjktNDNlYy1hYjY3LWJhZjk1NWEwMDg0MiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjbG91ZGh1YjpjbG91ZGh1YiJ9.OXLsPxs2h_frtDPdJcB2XXl-GZa4UuEcRs5DHBqbJRzWPOoMTlq_loApX2iyeT-TzWekdnpHSTlEMpNztNkTlOPBfGqg-_ouXG_s9EiDhMxpOtK-bmnXP2FvLBqcf0J_rX4aiSkt7CitVktCCVh8m6CIMtrk3nPbE0k8qX87NC_8UtLp67-25wQ9DkQnpkLxRhbnFmhR0VNTPPuHwxOz5xqwTIq4uc0AHxvelCoHS2ebA9mcXovf0UT97ajotd0NXMz8N96JcWOyVCODvZkIXmS_sGiP6jq8TnTkgvat3ZeyaKeQTqjW7kM0yHLbEgcF8fRHZzyoxBDT2ovxVGxbFQ"

  ## Pod labels to be added as tags.  An empty array for both include and
  ## exclude will include all labels.
  # label_include = []
  # label_exclude = ["*"]

  ## Set response_timeout (default 5 seconds)
  # response_timeout = "5s"

  ## Optional TLS Config
  # tls_ca = /path/to/cafile
  #tls_cert = "/var/lib/minikube/certs/apiserver-kubelet-client.crt"
  #tls_key = "/var/lib/minikube/certs/apiserver-kubelet-client.key"
  ## Use TLS but skip chain & host verification
  insecure_skip_verify = true

Kubernetes Monitoring

개요