How-To: Observe metrics with Grafana

Available dashboards

• System Service
• Sidecars
• Actors

The grafana-system-services-dashboard.json template shows Dapr system component status, dapr-operator, dapr-sidecar-injector, dapr-sentry, and dapr-placement:

The grafana-sidecar-dashboard.json template shows Dapr sidecar status, including sidecar health/resources, throughput/latency of HTTP and gRPC, Actor, mTLS, etc.:

The grafana-actor-dashboard.json template shows Dapr Sidecar status, actor invocation throughput/latency, timer/reminder triggers, and turn-based concurrnecy:

Pre-requisites

• Setup Prometheus

Setup on Kubernetes

Install Grafana

1. Add the Grafana Helm repo:

   helm repo add grafana https://grafana.github.io/helm-charts
   

2. Install the chart:

   helm install grafana grafana/grafana -n dapr-monitoring
   

   Note

   If you are Minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command instead:

   helm install grafana grafana/grafana -n dapr-monitoring --set persistence.enabled=false
   

3. Retrieve the admin password for Grafana login:

   kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
   

   You will get a password similar to cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1%. Remove the % character from the password to get cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1 as the admin password.

4. Validation Grafana is running in your cluster:

   kubectl get pods -n dapr-monitoring
   
   NAME                                                READY   STATUS       RESTARTS   AGE
   dapr-prom-kube-state-metrics-9849d6cc6-t94p8        1/1     Running      0          4m58s
   dapr-prom-prometheus-alertmanager-749cc46f6-9b5t8   2/2     Running      0          4m58s
   dapr-prom-prometheus-node-exporter-5jh8p            1/1     Running      0          4m58s
   dapr-prom-prometheus-node-exporter-88gbg            1/1     Running      0          4m58s
   dapr-prom-prometheus-node-exporter-bjp9f            1/1     Running      0          4m58s
   dapr-prom-prometheus-pushgateway-688665d597-h4xx2   1/1     Running      0          4m58s
   dapr-prom-prometheus-server-694fd8d7c-q5d59         2/2     Running      0          4m58s
   grafana-c49889cff-x56vj                             1/1     Running      0          5m10s
   

Configure Prometheus as data source

First you need to connect Prometheus as a data source to Grafana.

1. Port-forward to svc/grafana:

   kubectl port-forward svc/grafana 8080:80 -n dapr-monitoring
   
   Forwarding from 127.0.0.1:8080 -> 3000
   Forwarding from [::1]:8080 -> 3000
   Handling connection for 8080
   Handling connection for 8080
   

2. Open a browser to http://localhost:8080

3. Login to Grafana

   • Username = admin
   • Password = Password from above

4. Select Configuration and Data Sources

   

5. Add Prometheus as a data source.

   

6. Get your Prometheus HTTP URL

   The Prometheus HTTP URL follows the format http://<prometheus service endpoint>.<namespace>

   Start by getting the Prometheus server endpoint by running the following command:

   kubectl get svc -n dapr-monitoring
   
   NAME                                 TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)             AGE
   dapr-prom-kube-state-metrics         ClusterIP   10.0.174.177      <none>        8080/TCP            7d9h
   dapr-prom-prometheus-alertmanager    ClusterIP   10.0.255.199      <none>        80/TCP              7d9h
   dapr-prom-prometheus-node-exporter   ClusterIP   None              <none>        9100/TCP            7d9h
   dapr-prom-prometheus-pushgateway     ClusterIP   10.0.190.59       <none>        9091/TCP            7d9h
   dapr-prom-prometheus-server          ClusterIP   10.0.172.191      <none>        80/TCP              7d9h
   elasticsearch-master                 ClusterIP   10.0.36.146       <none>        9200/TCP,9300/TCP   7d10h
   elasticsearch-master-headless        ClusterIP   None              <none>        9200/TCP,9300/TCP   7d10h
   grafana                              ClusterIP   10.0.15.229       <none>        80/TCP              5d5h
   kibana-kibana                        ClusterIP   10.0.188.224      <none>        5601/TCP            7d10h
   
   

   In this guide the server name is dapr-prom-prometheus-server and the namespace is dapr-monitoring, so the HTTP URL will be http://dapr-prom-prometheus-server.dapr-monitoring.

7. Fill in the following settings:

   • Name: Dapr
   • HTTP URL: http://dapr-prom-prometheus-server.dapr-monitoring
   • Default: On
   

8. Click Save & Test button to verify that the connection succeeded.

Import dashboards in Grafana

1. In the upper left corner of the Grafana home screen, click the “+” option, then “Import”.

   You can now import Grafana dashboard templates from release assets for your Dapr version:

   

2. Find the dashboard that you imported and enjoy

   

   Tip

   Hover your mouse over the i in the corner to the description of each chart:

   

References

• Dapr Observability
• Prometheus Installation
• Prometheus on Kubernetes
• Prometheus Query Language
• Supported Dapr metrics

Example