How to make one Docker container wait for another

We can set up the order of service startup and shutdown using the depends_on option. However, it won’t make a Docker container wait for another one to be ready. There are many situations when we need to be sure that the container on which our service depends is actually running.

What we’re going to build

For this example, we’re going to configure the metricbeat serivce to start only after the kibana container is ready. The result of this post is available as a project in the spring-boot-log4j-2-scaffolding repository. When we’re done the project directory tree will look as follows:

The reason behind this post was a Metricbeat error I got when I tried to run the Elastic Stack using Docker Compose. The error occurred because Metricbeat requires a working connection with Kibana and it takes a lot of time to start Kibana. This article complements the Monitor Elastic Stack post. We’re going to focus on configuring Metricbeat to use the wait-for-kibana.sh script that we’re going to add to its configuration.

Docker Compose config

Below you’ll find the docker-compose.yml file responsible for setting up the two containers:

# docker-compose.yml
version: "3.3"
services:
  …
  kibana:
    image: kibana:$ELASTIC_STACK_VERSION
    environment:
      MONITORING_KIBANA_COLLECTION_ENABLED: "false"
    ports:
      - "5601:5601"
    networks:
      - internal

  metricbeat:
    build:
      context: ./metricbeat
      args:
        ELASTIC_STACK_VERSION: $ELASTIC_STACK_VERSION
    networks:
      - internal
    depends_on:
      - kibana

networks:
  internal:

Assume that we have a metricbeat service built from this custom Dockerfile:

# metricbeat/Dockerfile
ARG ELASTIC_STACK_VERSION
FROM docker.elastic.co/beats/metricbeat:${ELASTIC_STACK_VERSION}
COPY metricbeat.yml /usr/share/metricbeat/metricbeat.yml
USER root
RUN chown root:metricbeat /usr/share/metricbeat/metricbeat.yml
RUN chmod go-w /usr/share/metricbeat/metricbeat.yml
USER metricbeat

This configuration is based on the custom image configuration presented in the Elastic documentation. Later in the article, we’re going to edit this file to make the metricbeat service utilize a bash script that will make it wait for Kibana.

Additionally, the default Elastic Stack version is specified in my .env file:

ELASTIC_STACK_VERSION=7.7.0

Chaos resulting from unmet dependencies

Using solely the depends_on option in the docker-compose.yml file ensures that kibana is run first. However, launching a Kibana instance takes some time. Therefore, metricbeat will try to connect to the service that is not yet ready. As a consequence, the monitoring service will exit with the following error:

Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://kibana:5601/api/status fails: fail to execute the HTTP GET request: Get http://kibana:5601/api/status: dial tcp 172.21.0.7:5601: connect: connection refused.

In this case, we need to force metricbeat to attempt a connection only when kibana is accessible. In other words, we have to make one Docker container wait for another.

Override the default entrypoint to make Docker wait for another container

We’re going to create a bash script delaying metricbeat connection to kibana. Furthermore, the script will be used as a new entrypoint when starting the metricbeat container. We will execute the default entrypoint only when the kibana dependency is met.

How to write a bash script that will wait until a container is ready

Put the following wait-for-kibana.sh file alongside the Dockerfile for metricbeat:

#!/bin/bash
set -e
echo "Initiating..."
until curl --output /dev/null --silent --head --fail "$KIBANA_URL"; do
  >&2 echo "Kibana is unavailable - sleeping"
  sleep 1
done
>&2 echo "Kibana is up"
source /usr/local/bin/docker-entrypoint

You can learn more about scripts like this one in the Control startup and shutdown order in Compose documentation. The most important part is using the curl command to test whether the service on the given url is responding (line 4). We use the following options:

• output <file> – Write to a file instead of stdout, in our case we write to the Unix null device as we don’t need any output.
• silent – Silent mode, doesn’t show progress meter or error messages.
• fail – Fail silently (no output at all) on HTTP errors, since we expect many errors.
• head – Show document info only, often used for:

(…) testing hypertext links for validity, accessibility, and recent modification.

https://tools.ietf.org/html/rfc7231#section-4.3.2

Return control to the original entrypoint

The last line of our script executes the original entrypoint, the /usr/local/bin/docker-entrypoint file (line 9). Therefore, the original script will be executed only after the Kibana instance is ready. You need to explore the image documentation to find the actual path to the default entrypoint.

Pass the kibana url to the metricbeat container

Our script requires the KIBANA_URL environment variable. Remember to add it to the docker-compose.yml:

# docker-compose.yml 
  …  
  metricbeat:
    …
    environment:
      KIBANA_URL: "kibana:5601"
      …

Execute a custom script before the original entrypoint

To actually benefit from our script we have to execute the following actions while building the image:

• copy our script to the container;
• set it as the new entrypoint;
• reconstruct settings/commands that the custom entrypoint may have removed.

Below you’ll find the adjusted Dockerfile (lines 5, 10, 11):

# Dockerfile
ARG ELASTIC_STACK_VERSION
FROM docker.elastic.co/beats/metricbeat:${ELASTIC_STACK_VERSION}
COPY metricbeat.yml /usr/share/metricbeat/metricbeat.yml
COPY wait-for-kibana.sh /usr/share/metricbeat/wait-for-kibana.sh
USER root
RUN chown root:metricbeat /usr/share/metricbeat/metricbeat.yml
RUN chmod go-w /usr/share/metricbeat/metricbeat.yml
USER metricbeat
ENTRYPOINT ["sh", "/usr/share/metricbeat/wait-for-kibana.sh"]
CMD ["-environment", "container"]

Verify that your custom entrypoint was executed

Start the services defined in your docker-compose.yml file with the following command:

$ docker-compose up -d

If you changed the metricbeat settings, remember to rebuild the image with:

$ docker-compose up -d --build metricbeat

Examine logs printed in the metricbeat container:

As we can see, the metricbeat service doesn’t try connecting to the kibana service until the latter one is up.

Learn more on how to make one container wait for another

• How to create a loop in bash that is waiting for a webserver to respond?
• Verify if a URL exists
• Post on writing bash commands

Photo by Vinicius Altava from Pexels