Cloud nyam-nyam
Monitoring Solutions for K8s Cluster & Autoscaling

Monitoring Solutions for K8s Cluster & Autoscaling

Tsania Khilyatin Nada's photo
Tsania Khilyatin Nada
·

11 min read

In today’s cloud-native landscape, Kubernetes has revolutionized container management, providing organizations with powerful tools to deploy, scale, and manage applications efficiently. With so many components working together like pods, nodes, and services - it’s crucial to keep an eye on performance, resource usage, and potential issues.

Monitoring helps us to detect problems early, optimize workloads and ensure everything runs smoothly. In this article we will take a quick look how to monitor your Kubernetes cluster using Prometheus Stack (Prometheus, Grafana, Alert Manager) and we will also implement Horizontal Pod Autoscaler (HPA) using custom metrics from Prometheus Adapter.

But before diving into the steps, it’s important to understand the fundamental terms.

What is Kubernetes?

Kubernetes is a platform that helps you to run and manage application in containers. Containers are like mini virtual machines, but they are faster and use fewer resources. Kubernetes acts as coordinator, ensuring that your containers are running where they need to, scaling them up and down depending on the demand, and keeping everything running smoothly.

What is Metric?

A metric is basically a way to measure something. In tech, it’s used to track how well something is performing. For example, we might measure how much CPU or memory our app is using, or how long it takes to respond. These numbers help us keep an eye on things, find problems, and make sure everything runs smoothly.

What is Prometheus?

Prometheus is a tool that helps us collect and store metrics from our applications or systems. It constantly checks how things are performing like tracking how much CPU or memory our app is using and saves that data. we can then use Prometheus to analyze this data, spot any issues, and make sure everything is running as it should.

What is Grafana?

Grafana is a tool that helps us visualize data in a clear and interactive way. It takes the metrics collected by tools like Prometheus and turns them into easy-to-understand charts and graphs. This makes it simple to see how your applications or systems are performing at a glance.

What is Alertmanager?

Alertmanager is a tool that works with Prometheus to manage alerts. When something goes wrong in your system—like high CPU usage or an app going down—Prometheus triggers an alert. Alertmanager helps organize these alerts by grouping, silencing, or routing them to the right people or channels (like email, Slack, etc.).

What is HPA?

HPA (Horizontal Pod Autoscaler) is a feature in Kubernetes that automatically adjusts the number of pods running in our application based on demand.If the app is getting more traffic and needs more resources, HPA will increase the number of pods. If the traffic drops, HPA will reduce the number of pods to save resources.

What is Prometheus Adapter?

Prometheus Adapter acts as a bridge between Prometheus and Kubernetes, letting Kubernetes use the metrics (such as CPU usage, memory, etc.) from Prometheus to make decisions about scaling resources.

Tools

ToolsVersions
Kubeletv1.32.1
Kubectlv1.32.1
Kubeadm1.32
Containerd1.7.25
Prometheus3.1.0
Grafana11.5.1
Alertmanager0.28.0
Prometheus Adapterv0.12.0
Helmv3.17.0

Topology

Workflow

Preflight

*Execute on all nodes

  1. Change hostname

     sudo hostnamectl set-hostname master
 sudo hostnamectl set-hostname worker1 
 sudo hostnamectl set-hostname worker2

  2. Map hostname to /etc/hosts

     192.168.3.10 master
 192.168.3.20 worker1
 192.168.3.30 worker2

  3. Create & distribute SSH Key

     #Generate key
 ssh-keygen -t rsa
 #Distribute key
 ssh-copy-id student@master
 ssh-copy-id student@worker1
 ssh-copy-id student@worker2

Kubernetes Cluster Provisioning

*Execute on all nodes

  1. Upgrade Package and install the dependencies

     sudo apt update
 sudo apt upgrade -y
 sudo apt autoremove -y
 #install dependencies
 sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates

  2. Install Containerd as a container runtime

    • Add gpg key and Docker repository to your local machine

        #Add gpg key
  sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
  #Add repository
  sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

    • Update repository and install the constainerd

        sudo apt update
  sudo apt install -y containerd.io

    • Configure containerd to use systemd as cgroup driver

        # Generate default configuration for containerd
  containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
  # Configure systemd cgroup driver in configuration
  sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

    • Restart & enable containerd service

        sudo systemctl restart containerd
  sudo systemctl enable containerd

  3. Add Kernel configuration

     # enable overlay and br_netfilter modules
 cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
 overlay
 br_netfilter
 EOF
 # reload kernel modules
 sudo modprobe overlay
 sudo modprobe br_netfilter

  4. Add IPtables configuration

     cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
 net.bridge.bridge-nf-call-iptables  = 1
 net.bridge.bridge-nf-call-ip6tables = 1
 net.ipv4.ip_forward = 1
 EOF
 # apply sysctl without reboot
 sudo sysctl --system

  5. Install Kubectl, Kubelet, and Kubeadm

    • Add gpg key and Kubernetes repository

        # add gpg key
  curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
  sudo chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg
  # add repository
  echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
  sudo chmod 644 /etc/apt/sources.list.d/kubernetes.list

    • Install kubectl, kubelet & kubeadm

        sudo apt-get update
  sudo apt-get install -y kubelet kubeadm kubectl

*Execute only in master node

  1. Initialize Cluster

     # stop swap first
 sudo swapoff -a
 # initialize cluster
 sudo kubeadm init --pod-network-cidr=192.168.0.0/16

  2. Copy admin configuration so that we can manage cluster without “sudo”

     mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

  3. Install calico pod network

     # deploy calico operator
 kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
 # apply custom resource
 kubectl create -f  https://docs.projectcalico.org/manifests/custom-resources.yaml

    If your cidr network is 192.168.0.0/16 then you don't need to change anything in the custom resource file but if it is different then you need to adjust the configuration by changing the cidr value in the ipPools section.

  4. Verify config & cluster info

     kubectl config view
 kubectl cluster-info

  5. Show token and token ca-cert-hash

    sudo kubeadm token list
sudo openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

*Execute on worker nodes

Join Node Worker to Cluster

swapon -s
sudo swapoff -a
sudo kubeadm join --token [TOKEN] [NODE-MASTER]:6443 --discovery-token-ca-cert-hash sha256:[TOKEN-CA-CERT-HASH]

Install Prometheus Stack

*Execute on Master node

  1. Create namespace monitoring

     # create namespace monitoring
 kubectl create namespace monitoring
 # navigate to monitoring namespace
 kubectl config set-context --current --namespace=monitoring

  2. Install Helm

     # Add gpg key
 curl -s https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
 # install dependency
 sudo apt-get install apt-transport-https --yes
 # Add repository
 echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
 # Update repository & install Helm
 sudo apt update
 sudo apt install helm

  3. Add Repository for helm charts

     helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
 # Update repository
 helm repo update

  4. Install prometheus stack

     helm install prometheus-stack prometheus-community/kube-prometheus-stack

    prometheus-stack : release name

    prometheus-community : repository name

    kube-prometheus-stack : name of the chart

  5. List resources that has been created by operator

     kubectl get all
 ... 

 NAME                                                         READY   STATUS      RESTARTS       AGE
 pod/alertmanager-prometheus-stack-kube-prom-alertmanager-0   2/2     Running     0              7d19h
 pod/prometheus-prometheus-stack-kube-prom-prometheus-0       2/2     Running     0              7d19h
 pod/prometheus-stack-grafana-695cdf5b57-kmg5j                3/3     Running     0              7d19h
 pod/prometheus-stack-kube-prom-operator-5f6bff8545-g2kj9     1/1     Running     1 (2d ago)     7d19h
 pod/prometheus-stack-kube-state-metrics-8587b679d9-nwjqg     1/1     Running     11 (47h ago)   7d19h
 pod/prometheus-stack-prometheus-node-exporter-b75xw          1/1     Running     1 (2d ago)     7d19h
 pod/prometheus-stack-prometheus-node-exporter-nx94g          1/1     Running     1 (2d ago)     7d19h
 pod/prometheus-stack-prometheus-node-exporter-qhzpb          1/1     Running     0              7d19h

 NAME                                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
 service/alertmanager-operated                       ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP      7d19h
 service/prometheus-operated                         ClusterIP   None             <none>        9090/TCP                        7d19h
 service/prometheus-stack-grafana                    ClusterIP   10.107.67.74     <none>        80/TCP                          7d19h
 service/prometheus-stack-kube-prom-alertmanager     ClusterIP   10.110.179.13    <none>        9093/TCP,8080/TCP               7d19h
 service/prometheus-stack-kube-prom-operator         ClusterIP   10.105.35.42     <none>        443/TCP                         7d19h
 service/prometheus-stack-kube-prom-prometheus       ClusterIP   10.108.203.207   <none>        9090/TCP,8080/TCP               7d19h
 service/prometheus-stack-kube-state-metrics         ClusterIP   10.108.48.148    <none>        8080/TCP                        7d19h
 service/prometheus-stack-prometheus-node-exporter   ClusterIP   10.96.64.162     <none>        9100/TCP                        7d19h

 NAME                                                       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
 daemonset.apps/prometheus-stack-prometheus-node-exporter   3         3         3       3            3           kubernetes.io/os=linux   7d19h

 NAME                                                  READY   UP-TO-DATE   AVAILABLE   AGE
 deployment.apps/prometheus-stack-grafana              1/1     1            1           7d19h
 deployment.apps/prometheus-stack-kube-prom-operator   1/1     1            1           7d19h
 deployment.apps/prometheus-stack-kube-state-metrics   1/1     1            1           7d19h

 NAME                                                             DESIRED   CURRENT   READY   AGE
 replicaset.apps/prometheus-stack-grafana-695cdf5b57              1         1         1       7d19h
 replicaset.apps/prometheus-stack-kube-prom-operator-5f6bff8545   1         1         1       7d19h
 replicaset.apps/prometheus-stack-kube-state-metrics-8587b679d9   1         1         1       7d19h

 NAME                                                                    READY   AGE
 statefulset.apps/alertmanager-prometheus-stack-kube-prom-alertmanager   1/1     7d19h
 statefulset.apps/prometheus-prometheus-stack-kube-prom-prometheus       1/1     7d19h

  6. Expose Service Prometheus & Grafana

     kubectl expose service prometheus-stack-grafana   --name grafana-access-external --type=NodePort
 kubectl expose service prometheus-stack-kube-prom-prometheus --name prometheus-access-external --type=NodePort

  7. Access your Prometheus and Grafana using port that has been created from the service (e.g. 192.168.3.10:32548)

    Prometheus

  8. Execute this command to get the default user and password for grafana

     kubectl get secret --namespace monitoring prometheus-stack-grafana -o jsonpath='{.data.admin-user}' | base64 --decode
 kubectl get secret --namespace monitoring prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 --decode

Create Visualization

  1. Create new folder named ‘Simulasi‘ to group the dashboard

  2. Create first new dashboard and add visualization

  3. Add visualization named ‘Total CPU Usage in percent’

    Add your query metrics in query textbox based on your needs. here I will create visualization to monitor the total cpu usage in the entire cluster using node_cpu_seconds_total{mode=”idle”} divided by node_cpu_seconds_total in 5m range. Choose the visualization type as Stat and add the title. To look exactly like this, you can navigate to “Stat styles” > “Color mode” and choose Background Gradient, “Stat styles” > “Graph mode” and choose None . For the percent sign you have to change the Unit, go to “Standard options” > “Unit and choose” percent (0-100)

  4. Add visualization named Total Memory Usage in percent

  5. Add visualization named CPU Usage per Node [5m]

    Legend is used to overrides the display name based on label or custom text.

  6. Add visualization named Memory Usage per Node

  7. Add visualization named Nodes Status

    by default if a node is on the value is 1, while off the value is 0. To change to text UP and DOWN, it is necessary to add value mappings

  8. Add visualization named All Pod’s Status

    Repeat this query for each phase (Pending, Failed, etc.)

  9. Add visualization named Pod per Namespace

  10. Add visualization named Total PV & PVC

  11. Add visualization named

  12. Save Dashboard named Monitor-Cluster-Wide

  13. Overall visualization of the Monitor-Cluster-Wide dashboard

  14. Create second Dashboard named Monitor-Namespace-Pod

  15. Create a variable to group panels by labels such as namespace, instance etc.

    Result:

  16. Add visualization named CPU Usage per Pod

  17. Add visualization named Memory Usage per Pod

  18. Add visualization named Pod Status

  19. Add visualization named Total Service

  20. Add visualization named Total DaemonSet & StatefulSet

  21. Add visualization named Total Deployment

  22. Add visualization named Total ConfigMap & Secret

  23. Add visualization named Total Network Policy & Rules

    Repeat query for the egres rule

  24. Add visualization named Job Status

    Repeat query for the succeeded status

  25. Add visualization named HPA: Current Replicas vs Desired Replicas

  26. Create Contact Points (Which platforms that alerts will be sent)

    Here we will use 2 platforms (Discord & WhatsApp)

    Test alerts

    • Discord

    • WhatsApp

  27. Create alert for HPA panel

    go to “Alert” > “New alert rule”

    Select query A (current replicas status - match with query on panel ) as alert condition

    Configure Thresholds to evaluate the data that comes from query A so that if the current replica is above 2 then the alerts will be sent

    Select the Contact Point and save the alerts rule

Deploy Prometheus Adapter

* Execute only on master node

  1. Create values.yaml file to configure the custom metrics for prometheus adapter

     prometheus:
   url: "http://prometheus-stack-kube-prom-prometheus.monitoring.svc.cluster.local"
   port: 9090

 rules:
   default: false
   custom:
     - seriesQuery: '{__name__=~"^container_cpu_usage_seconds_total$"}'
       resources:
         overrides:
           namespace: {resource: "namespace"}
           pod: {resource: "pod"}
       name:
         matches: "^(.*)"
         as: "custom_cpu_usage"
       metricsQuery: "sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>)"

  2. Install Prometheus Adapater using Helm Charts

     helm install prometheus-adapater prometheus-community/prometheus-adapter -f values.yaml

  3. Verify the custom metrics

    We can check the custom metrics that has been exposed at /apis/custom.metrics.k8s.io

     kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .
 ...
 {
   "kind": "APIResourceList",
   "apiVersion": "v1",
   "groupVersion": "custom.metrics.k8s.io/v1beta1",
   "resources": [
     {
       "name": "namespaces/custom_cpu_usage",
       "singularName": "",
       "namespaced": false,
       "kind": "MetricValueList",
       "verbs": [
         "get"
       ]
     },
     {
       "name": "pods/custom_cpu_usage",
       "singularName": "",
       "namespaced": true,
       "kind": "MetricValueList",
       "verbs": [
         "get"
       ]
     }
   ]
 }

Testing

  1. Create deployment named uji-hpa-deploy with nginx as image container

     # save this configuration as uji-deploy.yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: uji-hpa-deploy
 spec:
   selector:
     matchLabels:
       app: nginx
   template:
     metadata:
       labels:
         app: nginx
     spec:
       containers:
       - name: container-nginx
         image: nginx:latest
         ports:
         - containerPort: 80
         resources:
           requests:
             cpu: "100m"
           limits:
             cpu: "300m"
 ...
 # apply configuration
 kubectl apply -f uji-deploy.yaml

  2. Create service for the deployment

     kubectl expose deployment uji-hpa-deploy --type=clusterip --port=80

  3. Create HPA with custom metrics

    This HPA will create replica from the deployment if the cpu usage is more than 300m (0,3 core)

     # save this configuration as uji-hpa.yaml
 apiVersion: autoscaling/v2
 kind: HorizontalPodAutoscaler
 metadata:
   name: uji-hpa
   namespace: default
 spec:
   scaleTargetRef:
     apiVersion: apps/v1
     kind: Deployment
     name: uji-hpa-deploy
   minReplicas: 2
   maxReplicas: 20
   metrics:
   - type: Pods
     pods:
       metric:
         name: custom_cpu_usage
       target:
         type: AverageValue
         averageValue: 300m
 ...
 # apply configuration
 kubectl apply -f uji-hpa.yaml

  4. Check the HPA

     kubectl get hpa -n default
 ...
 NAME              REFERENCE                   TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
 my-monitor-hpa    Deployment/my-deploy-app    cpu: 0%/50%   2         20        2          2d19h
 my-monitor-hpa2   Deployment/my-deploy-app2   cpu: 0%/70%   2         20        2          2d17h
 uji-hpa           Deployment/uji-hpa-deploy   0/300m        2         20        2          2d

  5. Test the scaling process using hey command

     hey -z 2m -c 5000 -m GET http://<IP SERVICE uji-hpa-deploy>

  6. Monitor the scaling process from the dashboard

    Here we can see that the desired replica is 3 and the alert rules is firing now (heart sign)

  7. Check the alerts on your contact points

    • Discord

    • You will get Resolved notification if the alert rules is no longer active (not firing)

Reference

Kubernetes#prometheusGrafanamonitoring