Gather metrics and statistics from Elastic Stack with Metricbeat and monitor the services using a Kibana dashboard.
We’re going to add the monitoring functionality to the Elastic Stack services used in the spring-boot-log4j-2-scaffolding application. As a result, we’ll be able to view metrics collected from Elasticsearch, Kibana, Logstash and Filebeat in a Kibana dashboard. All services will run in Docker containers. Meanwhile, You can clone the project and test it locally with Docker Compose.
Metricbeat collects the metrics and statistics from services and ships them to the output, in this example – Elasticsearch with minimum required configuration. Thus, it is a recommended tool for monitoring Elastic Stack in a production environment:
https://www.elastic.co/guide/en/elasticsearch/reference/current/monitoring-production.html
Metricbeat is the recommended method for collecting and shipping monitoring data to a monitoring cluster. If you have previously configured internal collection, you should migrate to using Metricbeat collection. Use either Metricbeat collection or internal collection; do not use both.
Firstly, we’re going to enclose our configuration for Metricbeat in a single file – metricbeat.yml. To see all non-deprecated configuration options visit the reference file. The configuration file used in this article has the following structure:
# metricbeat/metricbeat.yml
metricbeat:
modules:
- module: elasticsearch
…
- module: kibana
…
- module: logstash
…
- module: beat
…
setup:
…
output:
…In addition, in the project repository you’ll find:
All services described in this article are run with Docker Compose and use the following default environment variables:
# .env
ELASTIC_STACK_VERSION=7.7.0
ELASTIC_USER=elastic
ELASTIC_PASSWORD=test
ELASTIC_HOST=elasticsearch:9200
…xpack.monitoring.collection.enabled option to true in the docker-compose.yml file:# docker-compose.yml
…
elasticsearch:
environment:
xpack.monitoring.collection.enabled: "true"
…elasticsearch module in the metricbeat configuration:# metricbeat/metricbeat.yml
…
modules:
- module: elasticsearch
metricsets: ["node_stats", "index", "index_recovery", "index_summary", "shard", "ml_job", "ccr", "enrich", "cluster_stats"]
period: 10s
hosts: ["${ELASTIC_HOST}"]
username: "${ELASTIC_USER}"
password: "${ELASTIC_PASSWORD}"
xpack.enabled: "true"
…monitoring.kibana.collection.enabled option to false in the docker-compose.yml file (written in capital letters with underscores as separators):# docker-compose.yml
…
kibana:
environment:
MONITORING_KIBANA_COLLECTION_ENABLED: "false"
…kibana module in the metricbeat configuration:# metricbeat/metricbeat.yml
…
- module: kibana
metricsets: ["stats"]
period: 10s
hosts: ["kibana:5601"]
username: "${ELASTIC_USER}"
password: "${ELASTIC_PASSWORD}"
xpack.enabled: true
…monitoring.enabled option to false in the docker-compose.yml file:# docker-compose.yml
…
logstash:
environment:
MONITORING_ENABLED: "false"
…logstash module in the metricbeat configuration:# metricbeat/metricbeat.yml
…
- module: logstash
metricsets: ["node", "node_stats"]
period: 10s
hosts: ["logstash:9600"]
username: "${ELASTIC_USER}"
password: "${ELASTIC_PASSWORD}"Docker
xpack.enabled: true
…Standalone cluster. Read the Get rid of the Standalone cluster in Kibana monitoring dashboard post if you want to change that.monitoring.enabled option to false and allow external collection of monitoring data by enabling the HTTP endpoint in the filebeat.yml file (documentation):# filebeat.yml
filebeat:
inputs:
…
output:
…
monitoring:
enabled: "false"
http:
enabled: "true"
host: "filebeat"beat module in the metricbeat configuration:# metricbeat/metricbeat.yml
…
- module: beat
metricsets: ["stats", "state"]
period: 10s
hosts: ["filebeat:5066"]
username: "${ELASTIC_USER}"
password: "${ELASTIC_PASSWORD}"
xpack.enabled: true
…Standalone cluster. Read the Get rid of the Standalone cluster in Kibana monitoring dashboard post if you want to change that.Finally, we can configure the output for collected metrics. In this example project we’re using only a single Elasticsearch node. Therefore, we’re going to send the monitoring data there as well:
# metricbeat/metricbeat.yml
…
output:
elasticsearch:
hosts: ["${ELASTIC_HOST}"]
username: "${ELASTIC_USER}"
password: "${ELASTIC_PASSWORD}"To view monitoring data in a Kibana dashboard we have to use the following code:
# metricbeat/metricbeat.yml
…
setup:
dashboards.enabled: true
kibana:
host: "kibana:5601"
…By default, the data will be collected from the cluster specified in the elasticsearch.hosts value in the kibana.yml file – the project uses the default elasticsearch:9200 value which works with our one Elasticsearch node. However, if you run a dedicated cluster for monitoring, don’t forget to set monitoring.ui.elasticsearch.hosts option for the kibana service.
We want to start our metricbeat service in a Docker container. Therefore, we’re going to add the metricbeat service to the Elastic Stack services that were already defined in the docker-compose.yml file. Remember to provide all environment variables that we used in the metricbeat.yml file (the KIBANA_URL is used in our custom entrypoint for this container):
# docker-compose.yml
…
metricbeat:
build:
context: ./metricbeat
args:
ELASTIC_STACK_VERSION: $ELASTIC_STACK_VERSION
environment:
ELASTIC_USER: $ELASTIC_USER
ELASTIC_PASSWORD: $ELASTIC_PASSWORD
ELASTIC_HOST: $ELASTIC_HOST
KIBANA_URL: "kibana:5601"
networks:
- internal
depends_on:
- elasticsearch
- logstash
- filebeat
- kibana
networks:
internal:
…As you can see, the context for this build is expected in the metricbeat directory. We want to apply our custom configuration from the metricbeat.yml file and make sure that the metricbeat service will wait for the kibana service. With this purpose in mind, let’s create the following Dockerfile:
# metricbeat/Dockerfile
ARG ELASTIC_STACK_VERSION
FROM docker.elastic.co/beats/metricbeat:${ELASTIC_STACK_VERSION}
COPY metricbeat.yml /usr/share/metricbeat/metricbeat.yml
COPY wait-for-kibana.sh /usr/share/metricbeat/wait-for-kibana.sh
USER root
RUN chown root:metricbeat /usr/share/metricbeat/metricbeat.yml
RUN chmod go-w /usr/share/metricbeat/metricbeat.yml
USER metricbeat
ENTRYPOINT ["sh", "/usr/share/metricbeat/wait-for-kibana.sh"]
CMD ["-environment", "container"]metricbeat service will be started only after kibana is ready. The KIBANA_URL variable provided in the docker-compose.yml file is required for this script to work. You can read about this part of configuration in the How to make one Docker container wait for another post.At last, we can run all services defined in the docker-compose.yml file with the following command:
$ docker-compose up -dAbove all, wait until the metricbeat service connects to Kibana. You can see the containers by running the $docker ps command in the command line. The output should contain the following information:
IMAGE PORTS NAMES
elasticsearch:7.7.0 0.0.0.0:9200->9200/tcp, 9300/tcp springbootelasticstack_elasticsearch_1
logstash:7.7.0 0.0.0.0:5044->5044/tcp, 0.0.0.0:9600->9600/tcp springbootelasticstack_logstash_1
kibana:7.7.0 0.0.0.0:5601->5601/tcp springbootelasticstack_kibana_1
springbootelasticstack_filebeat springbootelasticstack_filebeat_1
springbootelasticstack_metricbeat springbootelasticstack_metricbeat_1
elastichq/elasticsearch-hq:latest 0.0.0.0:5000->5000/tcp springbootelasticstack_elastichq_1Finally, go to the http://localhost:5601/app/monitoring to see the clusters. Since our example Filebeat instance sends data to a Logstash instance, its metrics will be displayed in the Standalone cluster. Consequently, the rest of the Elastic Stack metrics are available under the docker-cluster:
In addition, read the Get rid of the Standalone cluster in Kibana monitoring dashboard post if you want to change that.
In the end, you can see the docker-cluster metrics on the image below:
Likewise, you can see the Standalone cluster metrics on the image below:
Finally, you can find the code responsible for monitoring Elastic Stack with Metricbeat in the commit c2a6f0d082072a52f56ee3ebc49bcc30ef482b99.
Check out this section in case something goes wrong.
You can test Metricbeat connection to a chosen service, even when data are not shown in the Kibana dashboard. In order to achieve that, you have to enter the container and send a request to the monitored service. Let’s assume that we want to see whether the connection to the Filebeat /stats endpoint works. We can achieve this with the following commands:
$ docker exec -it springbootelasticstack_metricbeat_1 /bin/bash
$ curl -XGET 'filebeat:5066/stats?pretty'In other words, the example correct output may look like on the screenshot below:
Specifically, if you got the Connection refused error, make sure that:
http.host and http.enabled options in the filebeat.yml file are correct;docker-compose.yml file for the filebeat service is correct.Secondly, if we list the metricsets that are not compatible with the config we can get the error similar to the following one caused by wrong metricsets applied in the elasticsearch module configuration:
ERROR instance/beat.go:932 Exiting: The elasticsearch module with xpack.enabled: true must have metricsets: [ccr enrich cluster_stats index index_recovery index_summary ml_job node_stats shard]Fortunately, all required and supported metricsets are listed in the error message. Thanks to that, we know that the elasticsearch module configuration shown below collects correct metricsets:
# metricbeat/metricbeat.yml
metricbeat:
modules:
- module: elasticsearch
metricsets: ["node_stats", "index", "index_recovery", "index_summary", "shard", "ml_job", "ccr", "enrich", "cluster_stats"]
…What’s more, when the permission to the metricbeat.yml file used in the container are invalid, we’ll get the following error:
Exiting: error loading config file: config file ("metricbeat.yml") can only be writable by the owner but the permissions are "-rw-rw-r--" (to fix the permissions use: 'chmod go-w /usr/share/metricbeat/metricbeat.yml')Of course, it can be fixed by applying the correct permissions to the metricbeat.yml file in the Dockerfile for the metricbeat service:
# metricbeat/Dockerfile
…
COPY metricbeat.yml /usr/share/metricbeat/metricbeat.yml
…
USER root
RUN chown root:metricbeat /usr/share/metricbeat/metricbeat.yml
RUN chmod go-w /usr/share/metricbeat/metricbeat.yml
USER metricbeat
…Lastly, if you enabled Elasticsearch security features, make sure that you pass the proper Elasticsearch credentials to Metricbeat and Kibana services. Otherwise, Metricbeat won’t be able to send monitoring data and Kibana won’t be able to read them. Take note, that the environment variables for Elasticsearch username and password in Kibana have slightly different names than the ones used in the other elements of the Elastic Stack:
# docker-compose.yml
…
kibana:
environment:
ELASTICSEARCH_USERNAME: $ELASTIC_USER
ELASTICSEARCH_PASSWORD: $ELASTIC_PASSWORD
…
metricbeat:
environment:
ELASTIC_USER: $ELASTIC_USER
ELASTIC_PASSWORD: $ELASTIC_PASSWORD
…To sum up, in this article we used only one Elasticsearch node for storing business and monitoring data. However, remember to use a separate cluster for metrics in a production environment, as stated in the docs:
In production, you should send monitoring data to a separate monitoring cluster so that historical data is available even when the nodes you are monitoring are not.
https://www.elastic.co/guide/en/elasticsearch/reference/current/monitoring-production.html
In production, we strongly recommend using a separate monitoring cluster. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. For the same reason, we also recommend using a separate Kibana instance for viewing the monitoring data.
https://www.elastic.co/guide/en/elasticsearch/reference/current/monitoring-overview.html
Photo by Lechon Kirb on StockSnap
Spring Security allows us to use role-based control to restrict access to API resources. However,…
A custom annotation in Spring Boot tests is an easy and flexible way to provide…
Delegating user management to Keycloak allows us to better focus on meeting the business needs…
Swagger offers various methods to authorize requests to our Keycloak secured API. I'll show you…
Configuring our Spring Boot API to use Keycloak as an authentication and authorization server can…
Keycloak provides simple integration with Spring applications. As a result, we can easily configure our…
View Comments
What if I have an actual cluster of Elasticsearch nodes? Can I monitor all nodes with a single Metricbeat instance?
The documentation seems to suggest you need a Metricbeat instance for each ES node:
"Install Metricbeat on each Elasticsearch node in the production cluster. Failure to install on each node may result in incomplete or missing results."
https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-metricbeat.html