Run Checks from Private Locations

Dash0's public probes monitor what the internet can reach. A private synthetic location monitors the rest: internal APIs, private load balancers, and services inside a Kubernetes cluster.

You run the synthetics worker where it can already reach those endpoints. It connects out to Dash0, so you do not open your network to incoming traffic.

Before You Start

Prepare the following:

• Authorization token: Use your organization's Dash0 auth token. Keep it in a Kubernetes Secret or a local environment file excluded from version control.
• Location ID: Choose an identifier, such as dc-fra-1. Workers using the same ID and organization join the same location. Use different IDs for locations that should run checks independently. A location ID is permanent: it names the same location for as long as your organization exists, and you cannot reuse it for a different location. Choose it with care.
• Connection endpoint: Open Settings → Endpoints and select Outbound Connector. Copy its Endpoint value, including the port, into SYNTHETICS_ADDRESS. Use the value for your organization and region, in host:port form, without https://.
• Telemetry endpoint: On the same page, select OTLP via gRPC. Use this endpoint for worker traces, logs, and runtime metrics, with an https:// prefix in OTEL_EXPORTER_OTLP_ENDPOINT. It is a separate connection from the Outbound Connector.
• Worker image: Pin a released version of ghcr.io/dash0hq/dash0-synthetics-worker.

Allow outbound connections to the Outbound Connector, the OTLP/gRPC endpoint, DNS, and the endpoints your checks target. Dash0 never connects in to you.

Run with the Dash0 Operator

Follow the operator installation instructions. Add these settings to your Helm values, using the Outbound Connector endpoint from your organization:

1234567
operator:
  syntheticsWorker:
    enabled: true
    serverAddress: "<outbound-connector-host>:443"
    insecure: false
  syntheticsWorkerImage:
    tag: "0.2.0"

Keep the existing Dash0OperatorConfiguration and its export configuration. Merge the following fields into its spec; do not replace the rest of your configuration:

123456789101112
spec:
  selfMonitoring:
    enabled: true
  syntheticsWorker:
    enabled: true
    instances:
      - locationId: dc-fra-1
        authorization:
          secretRef:
            name: dash0-authorization-secret
            key: token
        replicas: 2

Each entry in instances is one location. Add an entry for every location you want to run checks from, and use replicas to make a single location resilient.

The referenced Secret must be in the operator's namespace and contain the token without the Bearer prefix. For this installation method, a location ID must be at most 40 characters, contain only lowercase letters, digits, and hyphens, and start and end with a letter or digit.

Worker Configuration

These settings apply to manual installations. Environment variables override the optional synthetics-worker.yaml file in the working directory or /etc/dash0/.

For a file-based worker configuration, use the following top-level keys. Set the OTLP environment variables separately:

123456
listenAddress: ":8011"
address: "<outbound-connector-host>:443"
locationId: dc-fra-1
headers:
  authorization: "Bearer <auth-token>"
insecure: false

Verify the Location

A location appears in Dash0 as soon as its first worker connects. You do not create it in the UI. Select it in your synthetic check's scheduling configuration and run a check against a reachable internal endpoint.

Hover over the location's health dot to read its connection state:

• Connected: Dash0 has recently confirmed a worker connection.
• No worker connected: Dash0 has not confirmed a connection within the health window. After the last worker disconnects, this can take about three minutes to appear.
• Connection state unknown: Dash0 cannot determine the connection state. This does not establish that the location is down.

Connection health describes the worker's connection to Dash0. A connected worker can still fail to reach a target endpoint.

Create a Check for the Private Location

Define a Dash0SyntheticCheck in a namespace monitored by the operator. See Manage Synthetic Checks as Code for the synchronization requirements.

In schedule.locations, name the location the way you named it in your operator configuration, such as dc-fra-1. You can write the whole check by hand; nothing has to be copied out of the UI first.

Replace the example's location ID with yours and its URL with an internal endpoint reachable by your workers:

123456789101112131415161718192021222324252627282930313233343536373839
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0SyntheticCheck
metadata:
  name: internal-target-probe
  namespace: default
spec:
  display:
    name: internal-target-probe
  enabled: true
  plugin:
    kind: http
    spec:
      request:
        url: http://internal-target.default.svc.cluster.local/
        method: get
        headers: []
        queryParameters: []
        redirects: follow
        tls:
          allowInsecure: false
        tracing:
          addTracingHeaders: false
      assertions:
        criticalAssertions:
          - kind: status_code
            spec:
              operator: is
              value: "200"
        degradedAssertions: []
  schedule:
    strategy: all_locations
    interval: 1m
    locations:
      - dc-fra-1
  retries:
    kind: "off"
    spec: {}
  notifications:
    channels: []

Keep the empty headers, queryParameters, degradedAssertions, channels, and retries.spec fields: they are required. display.name is also required, and method must be lowercase.

A check downloaded from the UI names the location as synthetic_location_<ulid> instead. Both forms work, so you can apply such a file unchanged. If the name you chose is also the name of a Dash0 public region, Dash0 rejects the check and tells you which synthetic_location_<ulid> to use instead.

Monitor the Location

The location health indicator is the only place connection state is reported. See Verify the Location for its three states and update delay.

You cannot alert on a location going offline yet. Connection state is not available as a metric, so a check rule cannot watch it. Watch the indicator until this is supported.

When a check cannot run because its location is unavailable, the attempt is marked error.type=location_unavailable. That tells you the check did not run. It says nothing about whether the target endpoint is healthy.

High Availability

Run several replicas, or run workers with the same location ID and organization token in two clusters or zones. Every worker in the location must be able to reach the same target endpoints. Spread replicas across failure domains; two pods on one node do not protect against losing that node.

Dash0 dispatches each check to an available connected worker for the location. If one cluster is lost, workers in the other cluster can continue accepting checks. An in-flight check can still fail when its worker disappears; configure retries as appropriate.

Replicas provide redundancy, not additional check throughput. Dispatch concurrency is controlled by Dash0; increasing the worker count does not increase that limit.

Remove a Location

You cannot remove a private location. Dash0 has no delete control for locations.

To stop a location from running checks, stop its workers and remove the location from the schedule.locations of every check that uses it. The location stays in the list. Its health indicator reports No worker connected about three minutes after the last worker disconnects.

The location keeps its location ID. If you start a worker with that ID again, it rejoins the same location.