|| Prometheus +Grafana + Kubernetes ||PV, PVC & ConfigMap for Persistent Data

Prometheus + Grafana

If you land here , then definitely you are known to Prometheus and Grafana ,Probably you want to Integrate these tools with K8s, or you already deployed them in K8s but finding the way to make the data persistent of these stateless pods that could be recycle at any time, If yes , then you are at right place .. :)

In this blog I will tell you how to deploy the Prometheus and Grafana in K8s and their data should be remain permanent for that we are using K8s resources PV(Persistent Volume),PVC(Persistent Volume Claim) and ConfigMap.

First Understand the major concepts In Brief….


  • we can say that Monitoring == Keep your eye on something , In DevOps world Monitoring is essential thing to maintain production environment ,Pods , server,data-centers,network,operating system and many other essential things.
  • Thus Monitoring provides feedback from production. Monitoring delivers information about an application’s performance and usage patterns.
  • Take en example: suppose you are working in a IT company and suddenly company’s server goes down!!what about your clients???…Now what you or your company do????of course, you will find ,where is the problem to fix it…That’s the monitoring….In a bigger technical world it will be like trying to find a needle in a haystack …..
  • As In today’s IT industry silos can be seen easily in Development team and Operation Team so we want a Design pattern, A team , A tool to monitor what is happening in real-time?? because if you don’t know what is happening you can’t take decision to what to do and not able to find the solution .
  • In Monitoring world Logs, metrics, and traces are often known as the three pillars of observability.

Prometheus —

Grafana —

  • what if ?, If all the data will be in Visual Form , In form of Graphs , Bars , Charts now will it be easy?? Yes , of course. In visualization Monitoring will be easy ..you can easily see all the fluctuation or any undesired behaviour in real-time very quickly and that’s why Grafana is here..
  • Grafana is a visualization tool that collects data from data-sources[like Prometheus Time series data-base] and create visuals from it that will help in monitoring.
  • Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources. It is expandable through a plug-in system.[source-Wikipedia]

Let’s Jump to Play with K8s—

Now what is ConfigMap:

  • ConfigMaps bind configuration files, command-line arguments, environment variables, port numbers, and other configuration artifacts to your Pods’ containers and system components at runtime. ConfigMaps enable you to separate your configurations from your Pods and components, which helps keep your workloads portable. This makes their configurations easier to change and manage, and prevents hardcoding configuration data to Pod specifications.
  • ConfigMaps are useful for storing and sharing non-sensitive, unencrypted configuration information. To use sensitive information in your clusters, you must use Secrets. [source:cloud.google.com]
  • Here I am creating Kustomization file that is used to manage K8s object . Here using one file I can create the resources for both Prometheus and Grafana.
you can see that I am launching all the resources using a single kustomization file.

I created 3 YAML files :

  • Kustomization.yml
  • grafana.yml
  • Prometheus.yml

Let’s see their code:

Now Understand them one by one:

Prometheus :

  1. Prometheus server :Prometheus by default can instrument its own server at “systemIp:9090/metrics".
  2. Docker: we can configure docker as a Prometheus Target.To configure the Docker daemon as a Prometheus target, you need to specify the metrics-address. The best way to do this is via the daemon.json which is located at /etc/docker/daemon.json [In linux I am using here rhel-8 as base os for docker].
specify metrics-address with IP address of system with any port number at which you want the metrics of docker services.
As you can see with the above address I can see docker is providing its metrics for Instrumenting.

3. RHEL-8 as a Node using Node-exporter : Node Exporter is a Prometheus exporter for hardware and OS metrics with pluggable metric collectors.It allows to measure various machine resources such as memory, disk and CPU utilization.

starting node exporter services ,It will start at 9100 port number of your system
click on Metrics
Here you can see Node-exporter giving metrics for the operating system

In the above shown prometheus.yml file I created all the resources In a one YML file for easiness, just see them in Detail.

Prometheus PV,PVC

PV,PVC for Prometheus

Prometheus ConfigMap

Prometheus Service

Note: This all the things already written in Prometheus.yml file shown above, Here I am just showing separately for more clarity.

Prometheus is running at 9090 container port and 31000 Node port as provided in service.
All the targets of Prometheus


Grafana PV,PVC

here /mnt/data/graf/ the folder which is work as permanent volume for grafana Pod that is in host system (In my case minikube)

Grafana Deployment

Grafana Service

At Port number 3000 exposed grafana and 32000 NodePort

Note: Not forget to write SecurityContext : runAsUser:0 as I written in both grafana and prometheus deployment files otherwise you may be get error due to permission issues.

  • Let’s see Grafana pod running at 32000 port of minikube OS in my case.
Login page of Grafana
  • By default Loginname and password both is admin.you can change it on the next screen .
Here you can change password
select prometheus as a Data source
Herer you need to write URL of Prometheus
  • Pay attention !! Here at the time of writing URL of Prometheus I wrote “http://prom-exose:9090prom-expose [the name of Prometheus service ] and 9090 the port-number of pod [NodePort] because K8S has its own DNS service, kube-dns, within the cluster. For Service with ClusterIP ServiceType, all Pods in the same namespace can lookup and communicate with each other through DNS name. [but in my case I am using NodePort but , remember it also has property of ClusterPort]
  • Means Grafana Pod can communicate to Prometheus Pod by accessing Prometheus Service here prom-expose.
  • I can write this url also: “" as I am using NodePort Service to exposing Prometheus to the outside world.
  • But you can also choose ClusterPort as a service in case of Prometheus because we don’t need to expose the prometheus , because for monitoring we are using Grafana and we don’t have any concern with Prometheus data and database.
  • Here I am just using NodePort for reference.
select to create you first dashboard
  • Now you have to write PromQL: query language to access prometheus Time-series data-base.
  • Remember we can get data of 3 target of Prometheus In my case ,as I showed earlier.
completer monitoring and visualization using Grafana of Prometheus Targets.
  • Now If I delete pods of grafana & prometheus and launch new pods again even than this data of dashboard[remember to save your dashboard],login etc…will be remain persistent.

Hurrah!! Done…..A complete Integration of k8s+Prometheus+Grafana with permanent data.

A Thank From my side: Here I would like to express my gratitude towards Vimal Daga Sir . This was the one of the task given by sir to all of the trainees during DevOps Assembly Line training . I am very thankful for these knowledge and guidance that provided by sir to us in training .