Versions Compared

Old Version 1

changes.mady.by.user 이용식

Saved on

compared with

New Version Current

changes.mady.by.user 하재근

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Table of Contents

개요

  • Kubernetes의 Namespace, Node, Pod 등의 각종 정보를 모니터링 합니다.

...

화면 설명

...

  • “ⓐ” “ⓐ” K8s Filter

  • “ⓑ” “ⓑ” K8s API 호출 정보

  • “ⓒ” “ⓒ” K8s Object UI

  • “ⓓ” “ⓓ” K8s Object Detail 정보

  • “ⓔ” “ⓔ” K8s Object 시계열 데이터 사각화 정보

Kubernetes 조회

Info

CloudHub 1.4.0 버전 이상에서는 Minion ID를 "ch-collector"로 설정해야 Kubernetes의 SaltStack Rest API를 호출할 Target Minion으로 사용할 수 있습니다.

  • K8s를 호출할 Target 정보를 선택합니다.

  • K8s 호출 주기를 선택합니다.

...

  • K8s UI 화면에서 Object 클릭시 화면 오른쪽 Details에 해당 Object의 Detail 정보가 조회 됩니다.

...

Kubernetes Object 시계열 데이터 조회

  • K8s UI 화면에서 Object 클릭시 화면 하단에 해당 Object의 시계열 데이터가 시각화되어 조회되고 해당 Object에 마우스 오버시 Tooltip 형태로 화면에 CPU, Memory의 사용량을 확인 할 수 있습니다.

Object 마우스 클릭

...

Object 마우스 오버

...

Telegraf input plugin modules

  • kube_inventory

    • Via k8s API server on Control plane node.

    • Daemonsets, deployments, nodes, pods 등 k8s를 구성하고 있는 요소들에 대한 실시간 상태 정보 및 설정 정보(resource_requests_cpu_units, resource_limits_cpu_units 등과 같이 metric data only) 수집.

  • kubernetes

    • Via kubelet API on each node.

    • Nodes, pods 등의 CPU, Memory, Traffic 등의 metric data 수집.

Configurations of Telegraf

Monitoring 권한 생성

먼저, k8s에 이미 설정된 혹은 추후 설정될 다른 account 및 role들과 혼동되는 것을 대비하기 위해,
별도의 namespace, service account, cluster role 등을 추가한다.

Code Block
$ vim cloudhub_role.yaml
---
# 별도의 Namespace를 만든다.
apiVersion: v1
kind: Namespace
metadata:
  name: cloudhub
---
# cloudhub ServiceAccount를 만든다.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cloudhub
  namespace: cloudhub
---
# API에 대한 접근 권한을 만든다.
# 대충 resources endpoint에 대해 verbs 권한을 주겠다..라는 뜻이다.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: snetsystems:cluster:viewer
  labels:
    rbac.authorization.k8s.io/aggregate-view-cloudhub: "true"
rules:
  - verbs:
      - get
      - list
    apiGroups:
      - ''
    resources:
      - persistentvolumes
      - nodes
      - nodes/stats
      - nodes/proxy
---
# 위의 권한에 system:aggregate-to-view의 내용을 합한다.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: snetsystems:cloudhub
aggregationRule:
  clusterRoleSelectors:
    - matchLabels:
        rbac.authorization.k8s.io/aggregate-view-cloudhub: "true"
    - matchLabels:
        rbac.authorization.k8s.io/aggregate-to-view: "true"
rules: [] # Rules are automatically filled in by the controller manager.
---
# 위에서 최종 합한 권한을 cloudhub 계정에 바인딩 한다.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: snetsystems:cloudhub:viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: snetsystems:cloudhub
subjects:
  - kind: ServiceAccount
    name: cloudhub 
    namespace: cloudhub

# 위 내용들을 k8s에 적용/생성한다.
$ kubectl apply -f cloudhub_role.yaml

Configure telegraf

위의 account를 생성하면, secret도 같이 생성되며, 그 안에 토큰이 발급되어 있습니다.

Code Block
$ kubectl get secrets -n cloudhub
NAME                   TYPE                                  DATA   AGE
cloudhub-token-wdkkx   kubernetes.io/service-account-token   3      22h
default-token-lqcc8    kubernetes.io/service-account-token   3      22h

$ kubectl describe secrets -n cloudhub cloudhub-token-wdkkx
Name:         cloudhub-token-wdkkx
Namespace:    cloudhub
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: cloudhub
              kubernetes.io/service-account.uid: 9ec8dfc4-3129-43ec-ab67-baf955a00842

Type:  kubernetes.io/service-account-token

Data
====
namespace:  8 bytes
token:      GciOiJSUzI1NiIsImtpZCI6IjNqSm8xaXJ0MDZsaGxjdzVndWozY1A5VXBGbTdwX3VDUzBpd0J2a3ItR0EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbG91ZGh1Yi10b2tlbi13ZGtreCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjllYzhkZmM0LTMxMjktNDNlYy1hYjY3LWJhZjk1NWEwMDg0MiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjbG91ZGh1YjpjbG91ZGh1YiJ9.OXLsPxs2h_frtDPdJcB2XXl-GZa4UuEcRs5DHBqbJRzWPOoMTlq_loApX2iyeT-TzWekdnpHSTlEMpNztNkTlOPBfGqg-_ouXG_s9EiDhMxpOtK-bmnXP2FvLBqcf0J_rX4aiSkt7CitVktCCVh8m6CIMtrk3nPbE0k8qX87NC_8UtLp67-25wQ9DkQnpkLxRhbnFmhR0VNTPPuHwxOz5xqwTIq4uc0AHxvelCoHS2ebA9mcXovf0UT97ajotd0NXMz8N96JcWOyVCODvZkIXmS_sGiP6jq8TnTkgvat3ZeyaKeQTqjW7kM0yHLbEgcF8fRHZzyoxBDT2ovxVGxbFQ
ca.crt:     1025 bytes

위 토큰을 k8s api 사용 권한 인증을 위해 아래 telegraf.conf에서 사용합니다.

kube_inventory plugin

Code Block
[global_tags]
  dc = "snet-s2f" # will tag all metrics with dc=us-east-1
  rack = "VMs"
...
[agent]
  ## 수집되는 항목/내용이 많으므로, 일단 1분으로 하였다.
  interval = "1m"
...
# Read metrics from the Kubernetes api
[[inputs.kube_inventory]]
  ## URL for the Kubernetes API
  ## 일단, k8s master node에 설치하였기 때문에 localhost로 한다.
  url = "https://localhost:6443"

  ## Namespace to use. Set to "" to use all namespaces.
  namespace = ""

  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## If both of these are empty, we'll use the default serviceaccount:
  ## at: /run/secrets/kubernetes.io/serviceaccount/token
  # bearer_token = "/path/to/bearer/token"
  ## OR
  bearer_token_string = "GciOiJSUzI1NiIsImtpZCI6IjNqSm8xaXJ0MDZsaGxjdzVndWozY1A5VXBGbTdwX3VDUzBpd0J2a3ItR0EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbG91ZGh1Yi10b2tlbi13ZGtreCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjllYzhkZmM0LTMxMjktNDNlYy1hYjY3LWJhZjk1NWEwMDg0MiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjbG91ZGh1YjpjbG91ZGh1YiJ9.OXLsPxs2h_frtDPdJcB2XXl-GZa4UuEcRs5DHBqbJRzWPOoMTlq_loApX2iyeT-TzWekdnpHSTlEMpNztNkTlOPBfGqg-_ouXG_s9EiDhMxpOtK-bmnXP2FvLBqcf0J_rX4aiSkt7CitVktCCVh8m6CIMtrk3nPbE0k8qX87NC_8UtLp67-25wQ9DkQnpkLxRhbnFmhR0VNTPPuHwxOz5xqwTIq4uc0AHxvelCoHS2ebA9mcXovf0UT97ajotd0NXMz8N96JcWOyVCODvZkIXmS_sGiP6jq8TnTkgvat3ZeyaKeQTqjW7kM0yHLbEgcF8fRHZzyoxBDT2ovxVGxbFQ"

  ## Set response_timeout (default 5 seconds)
  response_timeout = "5s"

  ## Optional Resources to exclude from gathering
  ## Leave them with blank with try to gather everything available.
  ## Values can be - "daemonsets", deployments", "endpoints", "ingress", "nodes",
  ## "persistentvolumes", "persistentvolumeclaims", "pods", "services", "statefulsets"
  #resource_exclude = [ "deployments" ]

  ## Optional Resources to include when gathering
  ## Overrides resource_exclude if both set.
  #resource_include = [ "daemonsets", "deployments", "endpoints", "ingress", "persistentvolumes", "persistentvolumeclaims", "nodes", "pods", "services", "statefulsets" ]

  ## Optional TLS Config
  # tls_ca = "/path/to/cafile"
  # tls_cert = "/path/to/certfile"
  # tls_key = "/path/to/keyfile"
  ## Use TLS but skip chain & host verification
  insecure_skip_verify = true

  # fielddrop = ["created"]

kubernetes plugin

Code Block
[global_tags]
  dc = "snet-s2f" # will tag all metrics with dc=us-east-1
  rack = "VMs"
...
[agent]
  ## 수집되는 항목/내용이 많으므로, 일단 1분으로 하였다.
  interval = "1m"
...

[[inputs.kubernetes]]
  ## URL and port for the kubelet
  url = "https://localhost:10250"

  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## If both of these are empty, we'll use the default serviceaccount:
  ## at: /run/secrets/kubernetes.io/serviceaccount/token
  # bearer_token = "/path/to/bearer/token"
  ## OR
  bearer_token_string = "GciOiJSUzI1NiIsImtpZCI6IjNqSm8xaXJ0MDZsaGxjdzVndWozY1A5VXBGbTdwX3VDUzBpd0J2a3ItR0EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbG91ZGh1Yi10b2tlbi13ZGtreCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbG91ZGh1YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjllYzhkZmM0LTMxMjktNDNlYy1hYjY3LWJhZjk1NWEwMDg0MiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjbG91ZGh1YjpjbG91ZGh1YiJ9.OXLsPxs2h_frtDPdJcB2XXl-GZa4UuEcRs5DHBqbJRzWPOoMTlq_loApX2iyeT-TzWekdnpHSTlEMpNztNkTlOPBfGqg-_ouXG_s9EiDhMxpOtK-bmnXP2FvLBqcf0J_rX4aiSkt7CitVktCCVh8m6CIMtrk3nPbE0k8qX87NC_8UtLp67-25wQ9DkQnpkLxRhbnFmhR0VNTPPuHwxOz5xqwTIq4uc0AHxvelCoHS2ebA9mcXovf0UT97ajotd0NXMz8N96JcWOyVCODvZkIXmS_sGiP6jq8TnTkgvat3ZeyaKeQTqjW7kM0yHLbEgcF8fRHZzyoxBDT2ovxVGxbFQ"

  ## Pod labels to be added as tags.  An empty array for both include and
  ## exclude will include all labels.
  # label_include = []
  # label_exclude = ["*"]

  ## Set response_timeout (default 5 seconds)
  # response_timeout = "5s"

  ## Optional TLS Config
  # tls_ca = /path/to/cafile
  #tls_cert = "/var/lib/minikube/certs/apiserver-kubelet-client.crt"
  #tls_key = "/var/lib/minikube/certs/apiserver-kubelet-client.key"
  ## Use TLS but skip chain & host verification
  insecure_skip_verify = true