How to set up service monitoring in Kubernetes using Prometheus

In modern cloud-based environments, keeping an eye on the health and performance of your applications is important. Through Monitoring, users can spot problems early so they can be fixed before causing issues in production. This blog will guide you through setting up a Service Monitor using Prometheus with the help of the kube-prometheus-stack Helm chart.

What is a Service Monitor?

A Service Monitor is a custom resource used in Kubernetes with the help of Prometheus Operator to specify how services should be monitored. Before diving into Service Monitors, it’s important to understand what the Prometheus Operator is.

The Prometheus Operator uses Kubernetes Custom Resource Definitions (CRDs) to simplify the deployment and configuration of Prometheus, Alertmanager, and other monitoring components.

With the Prometheus Operator, you can easily deploy Prometheus instances, configure scrape targets, and collect custom metrics from endpoints.

What are Scrape targets?

Scrape targets are the data sources from which Prometheus collects (or “scrapes”) metrics at regular intervals

Service Monitors are used to set up these scrape targets. Note that a single Prometheus instance can select and use multiple Service Monitors. Service Monitors specify what Prometheus should scrape. Instead of manually configuring the Prometheus file, you can write a YAML file for the Service Monitor.

To understand more, imagine you have a deployment with two pods that expose /metrics endpoints. Traffic to these pods is managed by a Kubernetes service. To get Prometheus to scrape this service, you deploy a Service Monitor. Here, selectors and labels play a crucial role. Prometheus needs a Service Monitor selector to discover the Service Monitor, and the Service Monitor needs a selector to target the correct Kubernetes service. This ensures that the scrape targets appear in the Prometheus dashboard.

It’s essential to have a Prometheus instance correctly configured to select the appropriate Service Monitors. The Service Monitor must exist in the correct namespace, and label selectors must be accurate. This is why selectors and labels are so important.

Read Related: Top Kubernetes Monitoring Tools to Streamline Cluster Management

Connection Diagram of Prometheus

Before diving into setup, let’s look at the connection diagram of Prometheus within a Kubernetes environment.

This diagram describes the connection between Prometheus and a microservice (myapp-svc) through a Service Monitor in a Kubernetes environment. The architecture is divided into two main namespaces: svc-monitoring and app.

The components consist of:

1. Prometheus Operator:

   It manages and automates the deployment and configuration of the Prometheus server. It handles the management of the Prometheus server.

2. Prometheus Server:

   This component is responsible for scraping and storing metrics from monitored targets and is directly managed by the Prometheus Operator.

3. Prometheus:

   The service responsible for executing the scraping of metrics.

4. Service Monitor:

   A custom resource definition (CRD) that specifies to scrape metrics from a service and configures it to monitor specific endpoints. In this case, it is configured to monitor ‘myapp-svc’.

5. myapp-svc:

   The Kubernetes service that represents a microservice (myapp-pod) exposing metrics, serving as the target for the Service Monitor.

6. myapp-pod:

   This application pod running within the Kubernetes cluster, associated with ‘myapp-svc’ to handle incoming traffic.

This diagram shows how the Prometheus Operator and Service Monitor work together to enable Prometheus to scrape metrics from a specific service within a Kubernetes cluster.

How to set up service monitoring using Prometheus?

Let’s explore the essential steps to configure and optimize service monitoring with Prometheus, ensuring real-time insights and proactive issue resolution for your applications.

1. Prerequisite:

A Kubernetes cluster should be spinned up.

Ensure you have a running Kubernetes cluster.

Install kubectl to communicate with your cluster. Installation Guide

Install helm. Installation Guide

 

 

Step 1- Install kubectl

Follow the installation instructions at Kubernetes kubectl documentation to set up ‘kubectl’, which is necessary for communicating with your Kubernetes cluster.

Step 2 – Install Helm

Helm is required for managing Kubernetes applications. Install it using the instructions available at Helm installation guide.

Step 3 – Connect to Your Cluster Context

Connect to your cluster context and add the prometheus-community hem repo using below commands.

$ helm repo add prometheus-community 
https://github.com/prometheus-community/helm-charts
$ helm repo update

Step 4 – Add the Prometheus Community Helm Repository

Add the Prometheus Helm repository and update your Helm repos with the following commands:

$ helm repo add prometheus-community 
https://github.com/prometheus-community/helm-charts
helm repo update

Step 5 – Install Kube-Prometheus-Stack

Create a namespace called svc-monitoring and install the kube-prometheus-stack:

kubectl create namespace svc-monitoring
helm install prom-svc-monitor 
prometheus-community/kube-prometheus-stack -n svc-monitoring

Step 6 – Verify Installation

Confirm that the Kube-Prometheus-Stack has been installed by checking the installed resources:

Step 7- Configuring a Service Monitor

a. Deploy Sample Application Pod and Service:

Before configuring the Service Monitor, deploy a sample application pod and service. Use the ‘deployment.yaml’ and ‘service.yaml’ files for an example application called “myapp”.

 

 

b. Create a Service Monitor:

Configure a Service Monitor to scrape metrics from the sample application. Ensure the following settings:

• Metadata Labels: The labels should include a release name that matches the Helm chart release name, in this case, “prom-svc-monitor.”
• Selector Match Labels: The selector should match the labels in your service.yaml.
• Metric Path: Ensure that the metric path is correctly set.

c. Confirm Service Monitor Communication:

Ensure the Service Monitor can communicate with the service by verifying the logs and checking if Prometheus is scraping the metrics.

d. Verify Metrics Collection:

Prometheus should now begin collecting metrics. You can confirm this in the Prometheus dashboard. Prometheus dashboard can be exposed by port-forwarding the service “prom-svc-monitor-kube-prom-prometheus”. Access the Prometheus dashboard by port-forwarding the Prometheus service:

Elevate Your Kubernetes Monitoring Strategy with Prometheus

In this blog, we’ve demonstrated how to set up service monitoring in a Kubernetes environment using Prometheus and the kube-prometheus-stack Helm chart. We configured a Service Monitor for an application. Using these tools and techniques, you can efficiently monitor your services, ensuring they remain highly available and perform efficiently.
Ready to revolutionize your organization with DevOps? Contact Stackgenie today to discover how our Prometheus consulting services can fuel innovation and optimize operations.

Frequently Asked Questions

1. Why should I use Kube-Prometheus-Stack in my Kubernetes cluster?

Kube-Prometheus-Stack provides a comprehensive monitoring solution with minimal setup, allowing you to monitor the performance, health, and resource usage of your Kubernetes workloads.

2. What are the prerequisites for setting up Kube-Prometheus-Stack?

You should have a running Kubernetes cluster, Helm installed on your local machine, and appropriate permissions to deploy resources on the cluster.

3. Can I customize the Kube-Prometheus-Stack deployment?

Yes, you can customize the deployment by modifying the values.yaml file or passing custom values during the Helm installation to tailor the deployment to your needs.

4. How do I upgrade or uninstall Kube-Prometheus-Stack?

You can upgrade the stack using the helm upgrade command and uninstall it using the helm uninstall command, specifying the release name of your deployment.