On running the OpenTelemetry Collector on NixOS and rejoicing in its declarative goodness

The OpenTelemetry Collector is a vendor-neutral executable that receives telemetry in a variety of ways, processes it, and exports it to one or more destinations including a variety of open-source and commercial backends. On a NixOS host, the Collector can carry your whole monitoring setup: host and per-process metrics, journald logs, and the health of your systemd units, all filtered and enriched on the way out to whichever OpenTelemetry-compatible backend you use.

In this article, we go from the basics to advanced use cases to monitor your NixOS setup with the OpenTelemetry Collector.

A small disclaimer: Everything below is written against opentelemetry-collector-contrib v0.155.0, which is what nixpkgs builds at the time of writing; some aspects of the Collector change over time, and a default that is true at the time of writing may not be true on your channel.

Adding the OpenTelemetry Collector to your NixOS setup

The opentelemetry-collector nixpkgs module by default sets up a minimal build of the Collector modeled after the core distribution that can get you started, but misses important capabilities like uploading logs from files and journald (because it does not have filelogreceiver and journaldreceiver).

nix

123456789101112
{ pkgs, ... }:

{
  services.opentelemetry-collector = {
    enable = true;
    # Best to start with the contrib release
    package = pkgs.opentelemetry-collector-contrib;

    # IMPORTANT: You need to set the configurations with
    # `settings` or `configFile`, we'll talk about it soon!
  };
}

Provided that you add the missing Collector configuration, the listing above added to your configuration.nix is enough to get a systemd unit running the contrib release of the OpenTelemetry Collector. (Although I prefer putting my monitoring setup in an /etc/nixos/monitoring.nix file imported from /etc/nixos/configuration.nix.)

By the way, a couple paragraphs above I used a turn of phrase ("a minimal build of the Collector modeled after the core distribution") that would make some ears perk up in the OpenTelemetry Community. What I expected going in is that nixpkgs would repackage the binaries of the official OpenTelemetry Collector Releases. Instead, it compiles your Collector binaries based on a manifest using the OpenTelemetry Collector Builder (OCB). And it is a great thing, because it means that you could use that infrastructure to create a Collector with exactly what you need. But I am not going to cover it in this article; it is massive enough as it is.

Now, onto the configuration.

Ingredients of a Collector configuration

A Collector needs to have a (usually rather large) set of configurations to operate. These configurations are mostly made of defining components and declaring how to use them:

Receivers get telemetry in, and despite the word "receiver" implying they are passive components waiting on some sort of socket, many actively get telemetry, like by scraping Prometheus endpoints.
Processors modify the telemetry in flight, like throwing away datapoints not needed, or ensuring the right metadata are set.
Exporters send telemetry downstream to some other system.
Extensions expand the capabilities of the Collector to accomplish tasks not directly involved with processing telemetry data, like authentication to third parties (e.g. bearertokenauthextension) or providing access to the file system to store offsets for receivers to read files (e.g. file_storage).

(There is one more component type, connectors, that you are unlikely to need unless you are really having a lot of fun!)

Finally, pipelines compose receivers, processors and exporters into the flows telemetry takes from a receiver to an exporter.

Two ways to specify Collector configurations in NixOS

In the opentelemetry-collector nixpkgs module, there are two mutually exclusive ways of providing Collector configurations:

settings takes the Collector configuration as Nix data, and the module renders it to YAML
configFile gets a more "OpenTelemetry standard" approach and accepts a path to a file with OpenTelemetry Collector configurations

Largely irrespective of the topic at hand, I am generally known to advise with great conviction that people "do things the OTel way", which in this case would mean taking the configFile route and reusing the same file you would when deploying the Collector on any other system. But, as the saying goes, "it's the exception that proves the rule". When using NixOS, treating the Collector configurations as Nix data via settings is a wonderfully idiomatic experience: you get to compute your configuration with Nix functions and merge it across modules, which is what makes the helpers later in this article possible at all.

(Not to mention that, if you are using coding agents to help with the setup, they generally seem to do a wonderful job of understanding and generating pretty good *.nix, which is a cherry on top of an already deliciously declarative cake.)

So, in this article we go the settings route.

A very minimal first setup

Here below you see a very minimal setup that collects host metrics like CPU, disk and filesystem utilization every 30 seconds, and sends them over the OpenTelemetry Protocol (OTLP) to a fictional endpoint that requires no authentication:

nix

12345678910111213141516171819202122232425262728293031
{ pkgs, ... }:

{
  services.opentelemetry-collector = {
    enable = true;
    # Best to start with the contrib release
    package = pkgs.opentelemetry-collector-contrib;

    settings = {
      # host_metrics and otlp_http are in both distributions, so this much
      # would also run on the default package. The journald section is where contrib
      # starts to matter.
      receivers.host_metrics = {
        collection_interval = "30s";
        scrapers = {
          cpu = { };
          memory = { };
          load = { };
          filesystem = { };
        };
      };

      exporters.otlp_http.endpoint = "https://ingress.example.com";

      service.pipelines.metrics = {
        receivers = [ "host_metrics" ];
        exporters = [ "otlp_http" ];
      };
    };
  };
}

Those empty attribute sets like cpu = { } are not a mistake: in the Collector configurations, null and {} are treated the same, using defaults. So cpu: with no value becomes cpu = { }; in Nix.

Managing secrets

Receivers and exporters are the two kinds of component that tend to need credentials: an exporter has to authenticate to your backend, and a receiver may have to authenticate its callers. It goes without saying that you must not hard-code secrets in a *.nix file: the Nix store is world-readable, and settings is rendered in it, alongside your hard-coded secrets, which will then be readable by any user on the system.

There are multiple options to store secrets, with different levels of security. Systems like sops-nix and agenix encrypt to keys you hold usually outside of your NixOS system: the same secret can be re-encrypted for a new host, shared across a fleet, and rotated by another admin. I did not need any of that for my home setup; you probably should, the moment there is a second host or a second admin.

You could also store your secrets as unencrypted files, e.g. in /var/lib/secrets/, and use file-system permissions to restrict which users can access them; it can work, although it is easy to get wrong. And there is a complication because of the setup of the systemd unit created by the opentelemetry-collector module: it uses dynamic users by default, which means that the user and group IDs change across restarts of the process or of the machine. So, to set permissions to secret files "manually" in your *.nix files, you'd need to ensure static user and group IDs. This is easy enough to do by declaring the opentelemetry-collector user and group statically in a *.nix file. And, by the way, I have been doing precisely this for a while. But then Jochen Schalanda pointed me to a better option: systemd credentials.

The setup of systemd credentials is simple. The secrets are created with a command like the following, reading the plaintext from stdin so it never reaches your shell history:

1
systemd-creds encrypt --name=dash0-token --with-key=host+tpm2 - ./credentials/dash0-token.cred

The command above uses a combination of a host key and the TPM module on the machine to store the secret encrypted at rest. The output path is what LoadCredentialEncrypted= references below. (No TPM2 chip? Use --with-key=host, which seals to the host key alone, which systemd creates on its first startup.) At runtime, systemd makes the secrets available in /run/credentials/${unit}/${id} on a tmpfs mount, so that there is little risk of backing them up unencrypted by mistake.

nix

12345678910111213141516171819202122232425262728293031323334353637383940
{ config, pkgs, ... }:

let
  # Ciphertext, committed to this repo. See after the snippet for why that is fine,
  # and for the one thing it costs you.
  encryptedCredential = id: ./credentials/${id}.cred;

  # Where systemd mounts a LoadCredential= entry. This is deterministic
  # and simple, but you could also expand the directory from the
  # $CREDENTIALS_DIRECTORY environment variable that systemd adds to the
  # unit's environment.
  credentialPath = unit: id: "/run/credentials/${unit}/${id}";
in
{
  services.opentelemetry-collector = {
    enable = true;
    package = pkgs.opentelemetry-collector-contrib;

    settings = {
      extensions."bearertokenauth/dash0" = {
        scheme = "Bearer";
        token = "\${file:${credentialPath "opentelemetry-collector.service" "dash0-token"}}";
      };

      exporters.otlp_http = {
        endpoint = "https://ingress.eu-west-1.aws.dash0.com";
        auth.authenticator = "bearertokenauth/dash0";
      };

      # An extension missing from this list is never started, and nothing
      # warns you: you get a Collector that looks healthy and exports with
      # no Authorization header at all.
      service.extensions = [ "bearertokenauth/dash0" ];
    };
  };

  systemd.services.opentelemetry-collector.serviceConfig = {
    LoadCredentialEncrypted = [ "dash0-token:${encryptedCredential "dash0-token"}" ];
  };
}

By the way, I like the failure mode of systemd credentials a lot: the unit fails to start if the secret cannot be decrypted. This is a much more transparent failure mode than the Collector coming up, and then failing to receive or send telemetry because of a missing auth token.

Moreover, you can commit that .cred file to your version control repository. And while the idea of committing credentials to a git repo rightfully tends to make people twitch uncontrollably, in this specific case it is fine because --with-key=host+tpm2 seals the ciphertext to a specific machine: decrypting it needs both /var/lib/systemd/credential.secret and that TPM, neither of which is in the repository. Whoever clones your repository but has no access to your machine gets a file they cannot decrypt. Of course, there's a catch: if you happen to replace the motherboard, or wipe and reinstall your OS, or install a firmware update that changes the sealed PCR values in the TPM module, every credential is permanently undecryptable. So keep the plaintext in a password manager and expect to re-encrypt after any hardware change.

About secret rotation, I also find that the trade-off is fine: to the best of my knowledge, systemd credentials cannot do secret rotation on live units. The Collector reloads its configuration when it receives a SIGHUP signal. But I find there is no practical advantage to having the Collector reload configurations rather than having it restart via the systemd unit, because reloading the Collector configuration will lose all the telemetry in transit the same way as with a process restart.

One last thing: Mind the Nix escaping, because it cost me an hour of my life I am not getting back. In a double-quoted string, "\${...}" emits a literal ${...}; in an indented string the escape is ''${...}, which reads worse. Also, use indented strings (''...'') for anything containing backslashes, because Nix does not process escapes inside them.

Collecting Journald logs

When you run a service as a systemd unit, like, say, a Minecraft server, some or all of its logs will end up in journald, systemd's logging subsystem. And since among those logs tend to be those about bootstraps and crashes, they are pretty important logs to monitor.

The Collector has the journaldreceiver for getting logs from journald. The opentelemetry-collector module handles most of the setup requirements for us, automatically adding SupplementaryGroups = [ "systemd-journal" ] to the systemd unit of the Collector itself. What is left for us to do is to say which logs to retrieve, by listing the systemd units in the Collector configuration, and setting up a pipeline:

nix

123456789101112131415161718
receivers."journald/minecraft" = {
  units = [ "minecraft.service" ];
  priority = "info";
  start_at = "beginning";
  storage = "file_storage";
};

service.pipelines."logs/minecraft" = {
  receivers = [ "journald/minecraft" ];
  exporters = [ "otlp_http" ];
};

extensions.file_storage = {
  directory = "/var/lib/opentelemetry-collector/storage";
  create_directory = true;
};

service.extensions = [ "file_storage" ];

The file_storage extension is needed to keep track, across restarts of the Collector, of which logs have been read and which not yet. Without it, you will ingest the same logs multiple times.

A critical note about the telemetry produced by journaldreceiver: it uses journalctl --utc --output=json --follow ... behind the scenes, which means that the OpenTelemetry log record will have in its body the entire journald log and its metadata encoded as JSON: a map with MESSAGE, _SYSTEMD_UNIT, PRIORITY and the rest. That leaves a lot to be desired in terms of ergonomics, and unless you use a backend that automatically normalizes that JSON structure to the OpenTelemetry semantic conventions (cough Dash0 cough), you probably should use the transformprocessor to make the telemetry easy to query. ("How" it should be structured largely depends on the way you want to consume it downstream, and mostly on which log analytics tools you use.)

Host and process metrics

The hostmetricsreceiver gives you CPU, memory, disk, filesystem, network and load, to which you opt in by turning on the respective scrapers:

nix

123456789101112
receivers.host_metrics = {
  collection_interval = "30s";
  scrapers = {
    cpu = { };
    memory = { };
    disk = { };
    filesystem = { };
    network = { };
    load = { };
    processes = { };
  };
};

You may think that the processes stanza is giving you metrics about each process, but in reality it is limited to reporting how many processes exist, broken down by state.

For per-process metrics, we need to turn to the process scraper, also part of hostmetricsreceiver, which reports several time series per matched process. And there be dragons: point it at every process, and it can be a lot of mostly unnecessary telemetry. Better point it only at the services you care about, like the Minecraft server and the Collector itself in my case:

nix

1234567891011121314151617181920212223242526272829303132333435
let
  # This is a function, so it belongs in the module's `let` block. Put it
  # next to your options and Nix either rejects it as an unknown option or
  # chokes trying to serialize a function to YAML.
  processScraper = { names, matchType ? "strict" }: {
    collection_interval = "30s";
    scrapers.process = {
      include = {
        inherit names;
        match_type = matchType;
      };

      # The scraper walks every process on the host before it filters, as an
      # unprivileged user. Without these, the Collector's own logs fill up
      # with errors about processes you were never scraping.
      mute_process_name_error = true;
      mute_process_exe_error  = true;
      mute_process_io_error   = true;
      mute_process_user_error = true;

      # Both of these are `enabled: false` in v0.155.0. Opt in explicitly.
      metrics = {
        "process.cpu.utilization".enabled = true;
        "process.threads".enabled = true;
      };
    };
  };
in
{
  # Note the dotted paths. `receivers = { ... }` here would collide with the
  # `receivers.host_metrics` above: Nix fails the evaluation outright with
  # "attribute 'receivers' already defined" rather than picking a winner.
  receivers."host_metrics/minecraft" = processScraper { names = [ "java" ]; };
  receivers."host_metrics/otelcol"   = processScraper { names = [ "otelcol" ]; matchType = "regexp"; };
}

Unfortunately, the scraper can only filter which processes to generate time series about by matching on their process names, and that is not always descriptive enough. I am using my Minecraft server as a running example, and we play the Java version. (Because of course.) The process name for the Minecraft server is java, and configuring the scraper just on the process name means that, if I were to install a second Java application, it would be scraped as a false positive. But fear not! In the next section we cover how to throw away metrics (and other telemetry) you do not care for.

Filter out telemetry

So far, we have set up the collection of a lot of useful telemetry, but among it there can be stuff you do not need. For example, we have discussed in the previous section how the selectors for which process metrics to collect are rather coarse. So, let's discuss now how to throw away telemetry before it ends up in your backend. (The rule of thumb is that, if you cannot prevent some telemetry from being collected, it is best to filter it out as close to the source as possible.)

For filtering telemetry, we are going to use the filterprocessor, which uses the OpenTelemetry Transformation Language (OTTL) to specify matching expressions for which telemetry to throw away in transit through the Collector. For example, if I wanted to throw away all process metrics for processes other than the Collector and my Minecraft server, it would be as simple as this:

nix

123456789101112131415161718
# Declaring the processor is necessary, but you also need to add it to
# one or more pipelines for it to do anything useful.
processors."filter/minecraft_process" = {
  error_mode = "ignore";
  metrics.metric = [
    ''IsMatch(metric.name, "^process[.]") and resource.attributes["process.owner"] != "minecraft"''
  ];
};

# *Use* the processor. Note which receivers: `process.*` metrics come from the
# per-process scrapers of the previous section, not from plain `host_metrics`,
# whose `processes` scraper emits `system.processes.*` instead. Point this at
# `host_metrics` and the condition never matches a thing.
service.pipelines."metrics/minecraft-process" = {
  receivers = [ "host_metrics/minecraft" "host_metrics/otelcol" ];
  processors = [ "filter/minecraft_process" ];
  exporters = [ "otlp_http" ];
};

There are a couple of gotchas worth noting:

The conditions say what to drop, not what to keep, and they are OR-ed: if any condition matches, the piece of telemetry goes; you can combine multiple boolean statements with and. For more about OTTL, there are great resources out there.

Be careful with error_mode: it can be ignore, silent or propagate. ignore is the default, and it logs the error and moves on. It is the better option of the three. silent does what it says: if there is an issue with the OTTL expression, it is ignored and not even logged. propagate is the dangerous one: an OTTL condition that errors returns the error up the pipeline and the whole payload is dropped, which turns a typo in a condition into data loss, potentially affecting unrelated telemetry if you use the batch processor.

Resource attributes

At Dash0, we say that telemetry without context is just data. Without knowing what system a datapoint describes, the value you can get is limited. OpenTelemetry has semantic conventions that provide a large, expressive vocabulary to annotate your telemetry in a standard way and make it as useful as possible.

In my experience, the most consistently overlooked aspect of OpenTelemetry metadata is resource attributes, which describe which system the telemetry talks about. Which specific resource semantic conventions you may want to adopt depends on where you deploy your applications. For example: Kubernetes apps have k8s.*, which are different from the Function as a Service (faas.*) you may use with AWS Lambda and the like.

Since this article is about NixOS, there are a few semantic conventions that are definitely going to be useful:

host.* describes things like the host name and the host ID (such as the machine-id).
os.* covers the OS type (for NixOS: os.type=linux), os.name (nixos, read on NixOS from /etc/os-release).
deployment.* does what you would expect: deployment.environment.name could be test or prod, and it is the important one of the group; deployment.name, deployment.id and deployment.status also exist in the semconvs, but they are used rarely.
cloud.* is fundamental if you are deploying on a cloud.
service.* is widely considered the most important semantic convention in OpenTelemetry, as it denotes the logical layer of your component, and I wrote about it extensively in the past.

But how do you set all of this up in your NixOS setup? Essentially, in one of two ways, depending on which resource attributes can be set automatically for you by a processor, and which you need to configure yourself:

host.name: resourcedetectionprocessor — system detector, on by default
host.id, host.arch, host.cpu.*, host.ip, host.mac, host.interface: resourcedetectionprocessor — system detector, opt-in per attribute
os.type: resourcedetectionprocessor — system detector, on by default
os.name, os.version, os.description, os.build.id: resourcedetectionprocessor — system detector, opt-in per attribute
cloud.*: resourcedetectionprocessor — via cloud detector (ec2, gcp, azure, …)
service.name, service.namespace, service.version: resourceprocessor or transformprocessor — manual setup
service.instance.id: resourceprocessor or transformprocessor — manual setup
deployment.environment.name, deployment.name: resourceprocessor or transformprocessor — manual setup

The following will give you most of host and os; you will need to set up a cloud detector separately.

nix

123456789101112131415161718
processors.resource_detection = {
  detectors = [ "system" ]; # add "ec2", "ecs" for AWS depending on what platform you use, or "azure", "gcp", "digitalocean", "heroku" and more

  # This defaults to `true`, and that default will bite you: it overwrites
  # resource attributes your SDKs already set with this host's values. The
  # processor's own README strongly recommends turning it off, so do.
  override = false;

  system = {
    # Default is [ "dns" "os" ], which prefers the FQDN.
    hostname_sources = [ "os" ]; # host.name becomes the short hostname, not the FQDN
    resource_attributes = {
      "host.id".enabled = true;
      "os.name".enabled = true;
      "os.version".enabled = true;
    };
  };
};

The value of host.id on a non-containerized Linux box is the machine-id, and you may need to persist it deliberately if you ever move to an impermanent /etc.

The attributes you need to set manually are done as follows; here's an excerpt from my setup:

nix

12345678
processors."resource/common".attributes = [
  { key = "service.namespace";           value = "beelink"; action = "upsert"; }
  { key = "deployment.environment.name"; value = "home";    action = "upsert"; }
];

processors."resource/minecraft".attributes = [
  { key = "service.name"; value = "minecraft"; action = "upsert"; }
];

Every one of those is an upsert, which overwrites whatever the telemetry already carried, and that is deliberate. The alternative is insert, which leaves an existing value alone. You may be excused for thinking that insert is the more polite choice, but it is a trap on service.name specifically. OpenTelemetry SDKs must set a service.name, and fall back to an utterly useless unknown_service: plus the executable name when nobody configured a better service name. Better upsert for the identity you are asserting about your own host, and keep insert for the cases where the application genuinely knows better than you do, like its own service.version.

As always, don't forget that none of these processors do a thing until they are in a pipeline, and here the order matters too:

nix

1234567
service.pipelines.metrics = {
  receivers = [ "host_metrics" ];
  # Detection first, then your own values: the order decides who wins when
  # both have an opinion about the same attribute.
  processors = [ "resource_detection" "resource/common" ];
  exporters = [ "otlp_http" ];
};

One last thing: for anything conditional or computed, have a look at the transformprocessor using OTTL, which we already mentioned in the Filter out telemetry section.

On configuration validation

On a Nix system, all the configurations are best validated at build time. For the opentelemetry-collector module using settings, this is opt-in:

nix

1
services.opentelemetry-collector.validateConfigFile = true;

The module then renders your settings to a store path and runs otelcol validate against it during the build, so a malformed configuration fails nixos-rebuild instead of the Collector's next restart.

I was honestly surprised by the build-time validation being opt-in (I think that was an oversight), and I opened a PR to flip it. It is merged and available in unstable, on track to be shipped on stable with nixos-26.05.

Synthetic checks and systemd unit health

Every bit of telemetry we have set up so far is collected from inside the machine and sent outside for evaluation. But how are you going to know if the entire machine, or just the Collector itself, goes down? Hopefully, your backend will let you set up some sort of "dead-man switch", i.e., an alert that fires when some telemetry is not received.

And even so, that will not tell you that the machine is up but nobody can talk to it due to networking issues. I had just that issue a few weeks back, and came up with my own solution based on Dash0 synthetic checks. A synthetic check is when an external system invokes a network endpoint on your machine and checks that the response is what you expect. If you are using a commercial observability solution, chances are it has synthetic checks functionality for HTTPS endpoints. If you do not have one, in the past I used various combinations of vendors specialized in synthetic checks (where the free tier usually gets you covered for one or two machines), or even hand-rolled my own with AWS Lambda.

But which endpoint should be invoked? There isn't necessarily a good one out of the box. We'd like to know if the apps are running and doing, or ready to do, something useful.

I have had an itch to scratch for a while, since I learned that systemd unit health can be queried via D-Bus, so I wrote systemd-unit-healthz: a small Go program that reads unit state over D-Bus and serves it as JSON over HTTPS, terminating TLS itself and requiring a shared secret in a header. It answers 200 when every configured unit is active/running, and 503 otherwise, so a check from outside the house can assert on the status code alone. It ships a NixOS module in its flake:

nix

123456789101112131415161718192021222324252627282930313233
{
  imports = [ inputs.systemd-unit-healthz.nixosModules.default ];

  # The module never touches the firewall, so we need to open the port manually.
  networking.firewall.allowedTCPPorts = [ 443 ];

  # Give this one certificate its own group, rather than reusing the shared
  # `acme` one. Every cert defaults to group `acme`, so joining it would hand
  # an internet-facing service the private key of every certificate on the box.
  security.acme.certs."example.org".group = "healthz-tls";

  services.systemd-unit-healthz = {
    enable = true;
    extraGroups = [ "healthz-tls" ];  # to read the ACME-managed TLS key
    settings = {
      listen = ":443";
      path = "/healthz";
      units = [ "minecraft.service" ];
      tls.certFile = "/var/lib/acme/example.org/fullchain.pem";
      tls.keyFile = "/var/lib/acme/example.org/key.pem";
      auth = {
        kind = "header";
        header = "X-Health-Token";
        # Same helpers as the secrets section above.
        tokenFile = credentialPath "systemd-unit-healthz.service" "health-token";
      };
    };
  };

  systemd.services.systemd-unit-healthz.serviceConfig = {
    LoadCredentialEncrypted = [ "health-token:${encryptedCredential "health-token"}" ];
  };
}

In my setup I am using acme to manage Let's Encrypt TLS certificates for a custom domain I have and use to expose my machine to the internet.

Conclusions

There is a lot of work to be done to monitor a machine well. NixOS does make a lot of things easier, and I can hardly express in words the sheer joy I have in being able to do complex monitoring setups in a fully declarative way.

There are a few things I originally wanted to add to this article, and cut out because of the already massive length. One was how to build your own Collector using the OpenTelemetry Collector Builder, which nixpkgs uses already behind the scenes.

Another was how to report NixOS updates as deployment events in Dash0 using the dash0 CLI, which is available as a Nix flake (because I wanted it for myself).

Finally, I considered walking through the derivation I used to validate the configuration before the module could pass --set for me, but I would rather the PR land and nobody need it.

If you'd like to see that content, you know what to do in the comments!