Service Monitoring using Prometheus with Kube-Prometheus-Stack

Introduction

In modern cloud-based environments, keeping an eye on the health and performance of your applications is important. Through Monitoring, users can spot problems early so they can be fixed before causing issues in production. This blog will guide you through setting up a Service Monitor using Prometheus with the help of the kube-prometheus-stack Helm chart.

What is a Service Monitor?

A Service Monitor is a custom resource used in Kubernetes with the help of Prometheus Operator to specify how services should be monitored. Before diving into Service Monitors, it’s important to understand what the Prometheus Operator is.

The Prometheus Operator uses Kubernetes Custom Resource Definitions (CRDs) to simplify the deployment and configuration of Prometheus, Alertmanager, and other monitoring components.

With the Prometheus Operator, you can easily deploy Prometheus instances, configure scrape targets, and collect custom metrics from endpoints.

What are Scrape targets?

 

Scrape targets are the data sources from which Prometheus collects (or “scrapes”) metrics at regular intervals

Service Monitors are used to set up these scrape targets. Note that a single Prometheus instance can select and use multiple Service Monitors. Service Monitors specify what Prometheus should scrape. Instead of manually configuring the Prometheus file, you can write a YAML file for the Service Monitor.

To understand more, imagine you have a deployment with two pods that expose /metrics endpoints. Traffic to these pods is managed by a Kubernetes service. To get Prometheus to scrape this service, you deploy a Service Monitor. Here, selectors and labels play a crucial role. Prometheus needs a Service Monitor selector to discover the Service Monitor, and the Service Monitor needs a selector to target the correct Kubernetes service. This ensures that the scrape targets appear in the Prometheus dashboard.

It’s essential to have a Prometheus instance correctly configured to select the appropriate Service Monitors. The Service Monitor must exist in the correct namespace, and label selectors must be accurate. This is why selectors and labels are so important.

Connection Diagram

 

Before diving into setup, let’s look at the connection diagram of Prometheus within a Kubernetes environment.

This diagram describes the connection between Prometheus and a microservice (myapp-svc) through a Service Monitor in a Kubernetes environment. The architecture is divided into two main namespaces: svc-monitoring and app.

The components consist of:

1. Prometheus Operator: It manages and automates the deployment and configuration of the Prometheus server. It handles the management of the Prometheus server.
2. Prometheus Server: This component is responsible for scraping and storing metrics from monitored targets and is directly managed by the Prometheus Operator.
3. Prometheus: The service responsible for executing the scraping of metrics.
4. Service Monitor: A custom resource definition (CRD) that specifies to scrape metrics from a service and configures it to monitor specific endpoints. In this case, it is configured to monitor ‘myapp-svc’.
5. myapp-svc: The Kubernetes service that represents a microservice (myapp-pod) exposing metrics, serving as the target for the Service Monitor.
6. myapp-pod: This application pod running within the Kubernetes cluster, associated with ‘myapp-svc’ to handle incoming traffic.

This diagram shows how the Prometheus Operator and Service Monitor work together to enable Prometheus to scrape metrics from a specific service within a Kubernetes cluster.

Setup

1. Prerequisite: A Kubernetes cluster should be spinned up.
2. Install kubectl (https://kubernetes.io/docs/tasks/tools/) to communicate with your cluster.
3. Install helm (https://helm.sh/docs/intro/install/)
4. Connect to your cluster context and add the prometheus-community hem repo using below commands.
   

   $ helm repo add prometheus-community https://github.com/prometheus-community/helm-charts
   $ helm repo update

5. Install Kube-Prometheus-Stack, created a namespace “svc-monitoring” before executing below command.
   

   $ helm install prom-svc-monitor prometheus-community/kube-prometheus-stack -n svc-monitoring

6. Verify the kube prometheus stack got installed.

      7. Configuring a service monitor.

           a. Before deploying service monitor, deploy sample application pod and service. Here’s how to configure one for an example application called “myapp” using deployment.yaml and service.yaml file.

             b. Create Service Monitor, ensure below configuration while deploying.

• Under metadata, Lables should configure with a release name which matches the release name of helm chart, we used “prom-svc-monitor” above.
• Selector match label should match with the label in service.yaml of the application.
• Metric path should be configured correctly.

            c. Service monitor should communicate with the service matching the labels and selector from service-monitor and service of the application.

           d. Prometheus should now begin collecting metrics. You can confirm this in the Prometheus dashboard. Prometheus dashboard can be exposed by port-forwarding the service “prom-svc-monitor-kube-prom-prometheus”.

Conclusion

In this blog, we’ve demonstrated how to set up service monitoring in a Kubernetes environment using Prometheus and the kube-prometheus-stack Helm chart. We configured a Service Monitor for an application. Using these tools and techniques, you can efficiently monitor your services, ensuring they remain highly available and perform efficiently.
Ready to revolutionize your organization with DevOps? Contact Stackgenie today to discover how our Prometheus consulting services can fuel innovation and optimize operations.