Grafana provisioning – How to configure data sources and dashboards

featured image

We don’t have to manually configure data sources and dashboards for Grafana. Thanks to the Grafana provisioning feature, we can use configuration files to set up everything before running the application for the first time.

What we’re going to build

In this article, I’m going to work with the spring-boot-log4j-2-scaffolding project where I already use Prometheus to collect monitoring data on a Spring Boot application. Additionally, you can find the detailed description of applying Grafana (v7.1.3) to this project in the How to set up Grafana with Docker and connect it to Prometheus post. Remember, all applications are run with Docker Compose. Thanks to that, you can easily test the setup on your local machine.

My end goal was to be able to start Grafana with both a Prometheus data source and a dashboard (the predefined JVM dashboard instance) already configured.

Below you can see the grafana directory containing files that I added to my project to supply Grafana configuration:

files with Grafana configuration

Provisioning Prometheus as a Grafana data source

According to my docker compose configuration the prometheus service is available for the other services running within the internal network under prometheus:9090 (http://localhost:9090/ in my browser).

Configure the datasource

In order to use it as a data source for Grafana, specify the minimal required configuration in the provisioning/datasources/datasource.yml file:

Remember that we can use environment variables instead of hardcoded values. Below, you’ll find a short description of used options:

  • name – your unique name of the data source.
  • type – one of the officially supported data sources.
  • url – the URL of your data source (in my case, the Prometheus server).
  • access – access mode. You can choose either proxy or direct. (Server or Browser in the UI). We can learn more from the documentation:

Access mode controls how requests to the data source will be handled. Server should be the preferred way if nothing else stated.
Server access mode (Default):
All requests will be made from the browser to Grafana backend/server which in turn will forward the requests to the data source and by that circumvent possible Cross-Origin Resource Sharing (CORS) requirements. The URL needs to be accessible from the grafana backend/server if you select this access mode.
Browser access mode:
All requests will be made from the browser directly to the data source and may be subject to Cross-Origin Resource Sharing (CORS) requirements. The URL needs to be accessible from the browser if you select this access mode.

https://github.com/grafana/grafana/pull/11531

In effect, this file will configure a default data source for the default organisation in Grafana (identified with the id=1). Because of it, remember to specify the orgId option accordingly for your data sources if needed. For more detail, feel free to browse the official datasource.yml file example.

Mount the configuration to the docker container

Next, we need to mount this configuration to the grafana service. Namely, under the /etc/grafana/provisioning/datasources directory. The /etc/grafana/provisionig is the default value in Grafana Docker for the GF_PATHS_PROVISIONING variable. For this reason, edit the docker-compose.yml file to add the appropriate volume:

Provisioning a predefined Grafana dashboard

We can use one of the predefined, ready to use Grafana dashboards to save time on configuration. Therefore, to display metrics gathered on my Spring Boot project, I’m going to use the „Dashboard for Micrometer instrumented applications (Java, Spring Boot, Micronaut)” i. e. the JVM dashboard.

Add the configuration to the project

First, download the JSON file using the link provided on the dashboard page:

Download the JVM dashboard JSON file

In my case, I got the 9threvision.

Next, save the file in the grafana/provisioning/dashboards/ directory. Consequently, we need to create the dashboard.yml file in the same folder to make Grafana use our JVM dashboard config:

Below you’ll find a short description of used options:

  • name – each provider has to be identified by a unique name. In my example I chose Default as the name of my only provider.
  • folder – name of the dashboard folder. If you provide an empty string here, your dashboards will be displayed under the General folder. I chose Services as the name just like it was presented in the official tutorial.
  • path – path to dashboard files on a disk. Once again, the /etc/grafana/provisionig is the default value in Grafana Docker for the GF_PATHS_PROVISIONING variable.

Mount the configuration to the docker container

In the dashboard.yml file we specified the /etc/grafana/provisioning/dashboards as the path used by our Default provider. Therefore, we have to mount our folder to this location in the container:

However, starting Grafana now will result in the Datasource named ${DS_PROMETHEUS} was not found error once we try to access the dashboard.

How to deal with the Datasource named ${DS_PROMETHEUS} was not found error

The issue is caused by the "datasource": "${DS_PROMETHEUS}" used in the jvm-micrometer_rev9.json file. At the moment of writing this post the issue seems to be still open. Henceforth, I simply replaced all the ${DS_PROMETHEUS} occurrences with the correct data source name – Prometeus.

All in all, the issue occurs only when working with files downloaded from the Official and community dashboard page. In other words, you won’t have to edit the file manually if you copy the config json from a running Grafana instance that already uses the dashboard:

Grafana dahsboard json model that can be used as working configuration for provisioning

Verify that provisioning data sources and dahsboards was successful

Finally, if you are using my docker-compose.yml file, run the following command to start services:

In the grafana service logs I can see that provisioning did not generate any errors:

Now, we can visit http://localhost:3000/datasources to see our Prometeus data source:

Prometheus data source already available at the app startup thanks to Grafana provisioning

Likewise, go to http://localhost:3000/dashboards to verify that the JVM dashboard is indeed located in the Services directory as we specified in the dashboard.yml file:

JVM dashboard provisioned within the Services directory

Next, select the JVM (Micrometer) entry to see the dashboard:

JVM dashboard already available at the app startup thanks to Grafana provisioning

Troubleshooting

http: proxy error connection refused

If you:

  • use the proxy access mode in the datasource.yml config and
  • run services in Docker,

you can’t specify the http://localhost:9090 as the datasource url. In fact, you can use the service_name:port structure. e.g. prometheus:9090. Furthermore, you can find other solutions in this StackOverflow question.

Learn more on provisioning Grafana data sources and dashboards

Photo by Joshua Mcknight from Pexels

Leave a Reply

Your email address will not be published. Required fields are marked *