Over the past decade, several organizations have embraced microservices architecture, and this has been due to numerous benefits such as ease of operation, improved performance and enhanced scalability of applications. Companies are now relying on distributed microservices to provide a better experience for their end users. However, such a shift has raised some challenges around effectively monitoring the operation of such a distributed infrastructure.
Monitoring is an important factor for operating a microservices architecture efficiently. It is easy to adapt resources according to the user’s needs when working with a monolith. However, in a scenario where thousands of microservices are involved, it is difficult to observe the state of each microservice and adapt it according to the users’ needs and demands. Therefore, we need to proactively detect problems and failures and optimize the provisioning of resources, like CPU and memory, as per the requirement. In this context, Kubernetes offers monitoring features for cluster administrators to get a holistic view of the cluster, providing information about each deployed resource. Among several observability stack, one widely used stack is Prometheus and Grafana.
Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. The tool collects and stores its metrics as time series data (metrics information is stored with the timestamp at which it was recorded) alongside optional key-value pairs called labels.
Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored. The tool lets you create, explore, and share beautiful dashboards with your team and helps you foster a data driven culture.
In this blog, we will discuss how to setup Prometheus and Grafana in order to monitor a Kubernetes cluster. For illustration purposes, we will deploy a simple Nginx server and monitor it.
To complete this walkthrough:
First, we create a deployment object that manages the Nginx server.
kubectl create deployment nginx-server --image nginx --replicas 3
Once the server is deployed, we need to expose it using Kubernetes services.
kubectl expose deployment nginx-server --port 80 --type NodePort
As we are running the Kubernetes cluster locally, so we have to do port forward to visualise the deployed Nginx in a local browser using the below command:
kubectl port-forward svc/nginx-server 8080:80
Now, as the service is exposed, we can reach it using the localhost and the port that is now exposed –
The continuous monitoring of the deployed microservices infrastructure is important to deliver the best quality services. In the next step, we will set up the monitoring stack of Prometheus and Grafana using the Helm charts tool, which is a package manager for Kubernetes applications. First, we would need to create a namespace Prometheus and add the Prometheus helm chart repository.
kubectl create ns prometheus helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
Next, we will install the Prometheus server in the namespace created previously:
helm install prometheus prometheus-community/prometheus --namespace prometheus --set alertmanager.enabled="false" --set server.persistentVolume.enabled="false"
The Prometheus server is successfully installed.
We will do a port forward to fetch the Prometheus UI in local browser using the below command:
kubectl port-forward svc/prometheus-server -n prometheus 9090:80
Let’s access Prometheus UI in local browser with the URL –
To view all the metrics in Prometheus, we will use /metrics path with Prometheus URL –
To visualize the metrics collected from Prometheus, we will install Grafana in the cluster.
We will create a
grafana.yaml that will contain the configuration parameters of Grafana and allow reading metrics from the Prometheus server deployed on the cluster. The following is a simple configuration file that declares Prometheus as a data source for Grafana.
datasources: datasources.yaml: apiVersion: 1 datasources: - name: Prometheus type: prometheus url: http://prometheus-server.prometheus.svc.cluster.local access: proxy isDefault: true
Next, we will use Helm charts to install Grafana, note that we specified K8S as the admin password.
helm repo add grafana https://grafana.github.io/helm-charts helm install grafana grafana/grafana --set persistence.enabled=false --set adminPassword='K8S' --set service.type=NodePort --values grafana.yaml
The Grafana server is successfully installed.
To access the Grafana dashboard, we will do port forward using the below command:
kubectl port-forward svc/grafana 3000:80
Now we can access Grafana in our browser using the URL –
The credentials to log in to Grafana is below unless changed:
After logging in, we will add an already available dashboard in Grafana with ID 3119 that gives the Kubernetes overview. By accessing the below URL, you can view examples of pre-built Grafana dashboards.
Select Import from the drop-down menu.
Enter the dashboard ID as 3119 and click the Load button.
Next, we add the data source for this dashboard, i.e., Prometheus and after that, we can click on Import:
After the dashboard is imported, the visualisation of Kubernetes cluster overview will appear as below:
In this blog, we covered how to create a monitoring stack based on open source tools such as Prometheus and Grafana to continuously observe your cluster resources and apply the needed actions to provide the best experience for the end users.