eBPF Continuous Profiling released

Last updated:

Run Private Locations with Docker

Run the Dash0 synthetics worker as a Docker container on a VM or host without Kubernetes, and verify that the private location connects.

You can run a private location without Kubernetes. The synthetics worker is a single container. Run it with Docker or Docker Compose on any Linux VM or host that can reach the endpoints you want to check.

This is the same image the Dash0 Operator deploys. You only set the environment variables that the operator otherwise sets for you.

Before You Start

You need:

  • Docker: Docker Engine, with the Compose plugin (docker compose) if you use the Compose example.
  • A Linux host: The image is published for linux/amd64 and linux/arm64.
  • The values from Before You Start: An auth token, a location ID, and the Outbound Connector endpoint from Settings → Endpoints.
  • An image version: Pin a released tag of ghcr.io/dash0hq/dash0-synthetics-worker, such as 0.2.0. To run the same version as the operator, use the tag in operator.syntheticsWorkerImage.tag of the operator Helm chart values.

The operator accepts location IDs of at most 40 characters, with only lowercase letters, digits, and hyphens, starting and ending with a letter or digit. Use the same format here, so you can move the location to Kubernetes later without a new ID.

Network Requirements

The worker connects out to Dash0. Nothing connects in to it, so you do not open inbound ports.

Allow these outbound connections from the host:

DestinationPortPurpose
Outbound Connector endpoint443 (TCP, TLS)The task stream: Dash0 sends checks to the worker, and the worker sends results back. Required.
OTLP/gRPC endpoint4317 (TCP, TLS)Worker traces, logs, and runtime metrics. Only when you set OTEL_EXPORTER_OTLP_ENDPOINT.
DNS53Resolves the Dash0 endpoints and your check targets.
Your check targetsAs neededThe endpoints your checks request.

The worker keeps one long-lived gRPC stream open and sends keepalive pings every 10 seconds. Firewalls and proxies must allow long-lived HTTP/2 connections.

Outbound Proxy

When the host reaches the internet only through an HTTP proxy, set HTTPS_PROXY on the container, for example HTTPS_PROXY=http://proxy.internal:3128. The worker then opens its connection to Dash0 with an HTTP CONNECT through the proxy. The proxy must allow CONNECT to the Outbound Connector port.

The proxy setting does not apply to the checks. Check requests always connect directly from the container to the target, so the host needs a direct route to every check target.

What Checks Can Reach

Checks run from the container's network. They can reach private address ranges and internal DNS names. The worker blocks link-local addresses, such as 169.254.0.0/16, and cloud metadata endpoints. A check against a blocked address fails with URL validation failed: blocked IP address.

Inside a container, localhost is the container itself, not the host. To check a service that runs on the Docker host, add --add-host host.docker.internal:host-gateway and use host.docker.internal as the target host name.

The container uses Docker's DNS configuration. If your internal names resolve only through a specific DNS server, set it with --dns or the Compose dns key.

Configure the Worker

Create an environment file named synthetics-worker.env:

sh
SYNTHETICS_ADDRESS=<outbound-connector-host>:443
SYNTHETICS_LOCATIONID=dc-fra-1
SYNTHETICS_HEADERS=authorization=Bearer <auth-token>
  • Replace <outbound-connector-host>:443 with the Outbound Connector endpoint, without https://.
  • Replace <auth-token> with your auth token. Keep the authorization=Bearer prefix.
  • Do not put quotes around the values. docker run --env-file passes quotes into the value.

The file contains your token. Restrict access to it and keep it out of version control:

sh
chmod 600 synthetics-worker.env

The Worker Configuration table lists every setting, including the optional OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS for worker telemetry. Add them to the same file if you need them.

Run with Docker

Start the worker:

sh
docker run --detach \
  --name dash0-synthetics-worker \
  --restart unless-stopped \
  --env-file synthetics-worker.env \
  --read-only \
  --cap-drop ALL \
  --security-opt no-new-privileges:true \
  --publish 127.0.0.1:8011:8011 \
  ghcr.io/dash0hq/dash0-synthetics-worker:0.2.0
  • --restart unless-stopped restarts the worker after a crash or a host reboot, if the Docker service starts on boot.
  • --read-only, --cap-drop ALL, and no-new-privileges match the security settings the operator applies. The image already runs as a non-root user (UID 10001).
  • --publish 127.0.0.1:8011:8011 exposes the health server on the host only. Leave it out if you do not use the health check.

To check services on the Docker host, also add --add-host host.docker.internal:host-gateway. To use a proxy, also add --env HTTPS_PROXY=http://<proxy-host>:<port>.

Run with Docker Compose

Create compose.yaml next to synthetics-worker.env:

yaml
services:
  synthetics-worker:
    image: ghcr.io/dash0hq/dash0-synthetics-worker:0.2.0
    restart: unless-stopped
    env_file: synthetics-worker.env
    read_only: true
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    # Uncomment to check services that run on the Docker host:
    # extra_hosts:
    #   - "host.docker.internal:host-gateway"
    # Uncomment to connect to Dash0 through an HTTP proxy:
    # environment:
    #   HTTPS_PROXY: http://<proxy-host>:<port>
    deploy:
      replicas: 2

Start it:

sh
docker compose up --detach

Both replicas join the same location, and each one gets its own worker ID. Two replicas on one host restart independently, but they do not protect you against losing the host. For that, run workers with the same location ID on two hosts. See High Availability.

The example does not publish the health port, because replicas cannot share one host port. To publish it, set replicas: 1 and add ports: ["127.0.0.1:8011:8011"].

Verify the Location

  1. Read the worker log:

    sh
    docker logs dash0-synthetics-worker

    With Compose, run docker compose logs synthetics-worker. A started worker logs its version, then Starting health gRPC server and Synthetics worker starting with your locationId. The worker does not log a separate message when it connects. The warning Synthetics stream closed, reconnecting means the worker is not connected. See Troubleshooting.

  2. In Dash0, open a synthetic check and go to its scheduling configuration. The location appears in the list as soon as its first worker connects. Hover over its health dot. Connected means that Dash0 confirmed the worker connection.

  3. Select the location for a check against an internal endpoint and run it. The Verify the Location section explains the health states.

Check the Health Server

The worker serves the standard gRPC health protocol on port 8011, under two service names:

  • liveness: SERVING while the process runs. It stays SERVING when the worker cannot reach Dash0, because a restart does not fix a network problem.
  • readiness: SERVING only while the task stream to Dash0 is open. A worker that never connected is never ready.

The image does not contain a gRPC health client, so run the check from the host, for example with grpc_health_probe:

sh
grpc_health_probe -addr=127.0.0.1:8011 -service=readiness

A connected worker prints status: SERVING. Use this check in your host monitoring if you want to alert when the worker loses its connection. Dash0 cannot alert on a location going offline yet.

Upgrade the Worker

  1. Pick the new version and replace the image tag in your docker run command or compose.yaml.

  2. Pull the image:

    sh
    docker pull ghcr.io/dash0hq/dash0-synthetics-worker:<new-tag>
  3. Replace the running container.

    With Docker:

    sh
    docker stop dash0-synthetics-worker
    docker rm dash0-synthetics-worker

    Then run the docker run command again with the new tag.

    With Docker Compose:

    sh
    docker compose up --detach

The worker shuts down gracefully on docker stop and closes its stream to Dash0. A check that runs at that moment can fail. To avoid a gap, run workers for the location on two hosts and upgrade one host at a time. Configure retries for checks that must not fail during an upgrade.

Stop or Remove the Worker

To stop the location from running checks, remove the container and remove the location from every check that uses it:

sh
docker rm --force dash0-synthetics-worker

With Compose, run docker compose down. The location stays in Dash0 and keeps its ID. See Remove a Location.

Troubleshooting

The Container Exits at Startup

The log ends with a fatal line that names the missing setting:

  • parse config: address must be set: Add SYNTHETICS_ADDRESS with the Outbound Connector endpoint.
  • parse config: locationId must be set: Add SYNTHETICS_LOCATIONID with your location ID.
  • parse config: headers must carry a non-empty authorization header: Set SYNTHETICS_HEADERS=authorization=Bearer <auth-token>.

The Worker Does Not Connect

The log repeats Synthetics stream closed, reconnecting. The worker retries with a backoff of up to 30 seconds. Read the error field:

  • connection refused, i/o timeout, or a DNS error: The host cannot reach the Outbound Connector endpoint. Check the endpoint value, DNS, the firewall, and the proxy settings.
  • Another error from the server: Dash0 closed the stream, for example because it rejected the token. Check that the token belongs to your organization and that the endpoint is the one shown in Settings → Endpoints.

If the location shows No worker connected, check that the container runs with docker ps and read its log. The state can take about three minutes to change after the last worker disconnects.

A Check Fails but the Location Is Connected

  • URL validation failed: blocked IP address: The target is a link-local or cloud metadata address. See What Checks Can Reach.
  • A check against a service on the Docker host cannot connect: localhost inside the container is the container. Use host.docker.internal and add the host-gateway mapping.
  • A check cannot resolve an internal host name: The container does not use your internal DNS server. Set --dns or the Compose dns key.

For more detail in the log, set LOGLEVEL=debug in the environment file and restart the container.