Mastering the OpenTelemetry Resource Processor

The OpenTelemetry Collector is pivotal for managing and refining telemetry data in modern observability pipelines.

Among its array of powerful components, the resource processor is your go-to for customizing the attributes that describe where your telemetry originates.

It is useful for ensuring consistent identities across your services, adding valuable infrastructure context, and standardizing resource-level metadata within your data pipelines.

While the standard documentation offers a solid foundation, this guide aims to move beyond the basics by offering a thorough and practical exploration.

Let’s get started!

What are OpenTelemetry attributes?

In OpenTelemetry, attributes are pieces of metadata that provide crucial context to your telemetry data. An attribute is simply a key-value pair that describes a specific characteristic of the event, operation, or entity being observed.

Within the OpenTelemetry data model, attributes can be found in various locations, including Resources, Instrumentation Scopes, Metric Points, Spans, Span Events, Span Links, and Log Records.

In the OpenTelemetry debug exporter, attributes appear like this:

Understanding resource attributes

While the term “attribute” is broad, resource attributes hold a special significance.

A Resource in OpenTelemetry represents the entity that generates telemetry. This could be a specific service instance, a physical host, a container, a Kubernetes pod, or even a cloud environment. Resource attributes are precisely those key-value pairs that describe this very source.

Resource attributes in Dash0

What makes resource attributes unique?

Their distinctiveness stems from their specific role and application:

1. Scope: Resource attributes are inherently tied to the Resource object, which then gets associated with all telemetry produced by that particular source. They provide overarching context about where the data came from.
2. Stability: Unlike other attributes whose values might change with each operation, resource attributes are expected to remain consistent for a given resource instance throughout its operational lifespan.
3. Identity: Resource attributes uniquely identify and categorize the origins of your telemetry. They form the basis for constructing accurate service maps, filtering data by deployment environment, and pinpointing which infrastructure component experienced an issue.

Some examples of resource attributes include:

1234567
service.name: "authentication-service"
service.version: "2.1.0"
host.name: "ec2-instance-prod-01"
os.type: "linux"
cloud.provider: "aws"
k8s.cluster.name: "main-prod-cluster"
k8s.pod.name: "auth-pod-789abc"

Why is this distinction important?

Resource attributes allows observability backends to aggregate and correlate data from the same source consistently, irrespective of the individual operations or events. This is fundamental for building reliable service maps and understanding infrastructure health.

Service maps in Dash0

By defining attributes once at the resource level, you avoid redundant data transmission on every single telemetry signal, which helps reduce bandwidth and processing requirements.

This separation also provides clarity: general attributes allow you to piece together what happened, while resource attributes clarify where it happened.

The Collector’s resource processor is dedicated to managing these resource attributes. For manipulating general attributes tied to individual spans, metrics, or logs, you would typically utilize the attributes processor.

When to use the resource processor

The resource processor is the one to reach for whenever you need to modify or enrich your telemetry’s origin context. Its common applications include:

• Standardizing service identity: Ensuring that service attributes like service.nameCopy or service.namespaceCopy are consistent across diverse deployments or teams.
• Infusing infrastructure metadata: Adding uniform host.nameCopy, cloud.providerCopy, k8s.cluster.nameCopy, or container.idCopy details to all telemetry originating from a specific source.
• Overriding auto-detected attributes: Correcting or supplementing resource attributes that are automatically discovered by other components like the resource detection processor.
• Redacting sensitive resource information: Hashing or deleting private details from resource attributes, such as internal host names or IP addresses that should remain within your network boundaries.
• Normalizing environment context: Guaranteeing that deployment.environmentCopy or similar attributes are uniformly applied across all telemetry from a specific operational environment.

Quick Start: adding the resource processor

To begin utilizing the resourceCopy processor, you must first define it within the processorsCopy section of your Collector configuration.

You have the flexibility to establish multiple instances of the resourceCopy processor, each designated by a unique name. This enables you to apply different sets of resource attribute adjustments to various pipelines or for distinct purposes.

After defining your instances, you simply reference the chosen named instance in the pipelines section where you want it active:

otelcol.yaml
1234567891011121314151617181920212223242526272829
processors:
  resource/environment_injector: # Named instance 1
    attributes:
      - key: deployment.environment.name
        value: production
        action: upsert
      - key: cloud.region
        value: us-east-1
        action: insert

  resource/host_obfuscator: # Named instance 2
    attributes:
      - key: host.name
        action: hash

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource/environment_injector, resource/host_obfuscator] # Apply both to traces
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [resource/environment_injector] # Apply only the environment injector to metrics
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [resource/environment_injector, resource/host_obfuscator] # Apply both to logs
      exporters: [otlp]

The specific actions supported by the resource processor are functionally identical to those found in the attributes processor, except that they are applied only to Resource attributesCopy.

These operations are carried out sequentially, following the order in which they are defined in your configuration to ensure predictability.

Adding and modifying attributes

The following actions allow you to populate resource attributes with new values:

insertCopy

This action adds a new resource attribute only if it does not already exist:

otelcol.yaml
123
- key: new_resource_attribute
  value: "a_default_value"
  action: insert # Adds `new_resource_attribute` if it's currently missing

updateCopy

This action modifies an existing resource attribute only if it already exists:

otelcol.yaml
123
- key: existing_resource_attribute
  value: "new_value"
  action: update # Changes `existing_resource_attribute` to `new_value`

upsertCopy

This versatile action either adds a new resource attribute (if it’s absent) or updates an existing one (if it’s present). It’s often the most practical choice:

otelcol.yaml
123
- key: deployment.tier
  value: "backend-api"
  action: upsert # Ensures `deployment.tier` is set to `backend-api`

Populating attribute values

When utilizing insertCopy, updateCopy, or upsertCopy actions, you have three distinct methods to populate the value of the target resource attributes:

valueCopy

This method enables you to provide a literal value directly. The processor automatically infers the data type from your configuration:

otelcol.yaml
123456789101112131415
processors:
  resource:
    attributes:
      - key: service.name
        value: my-auth-service
        action: upsert
      - key: deployment.environment
        value: production
        action: upsert
      - key: cloud.provider
        value: aws
        action: upsert
      - key: host.arch
        value: amd64
        action: upsert

from_attributeCopy

This option copies the new resource attribute’s value from another existing resource attribute within the same telemetry signal:

otelcol.yaml
12345
# Copy the value of `host.arch` to a new attribute `system.architecture`
attributes:
  - key: system.architecture
    from_attribute: host.arch
    action: upsert

A common technique involves using from_attributeCopy to duplicate a value to a new resource attribute, followed by deleting the original:

otelcol.yaml
1234567
# Rename `aws_id` to `cloud.resource_id`
attributes:
  - key: cloud.resource_id
    from_attribute: aws_id
    action: upsert
  - key: aws_id
    action: delete

Should the source resource attribute (e.g., aws_idCopy in this scenario) not exist, the action will simply be skipped.

from_contextCopy

This powerful feature allows you to extract values from the OpenTelemetry Collector’s internal operational context. This includes data such as gRPC metadata, HTTP headers from incoming requests, the client’s address, or authentication details. Here are the available patterns:

• metadata.<key>Copy: To extract HTTP headers or gRPC metadata, you must prepend the key with metadata.Copy. Crucially, you’ll need to enable include_metadata: trueCopy on your OTLP receiver to make these attributes accessible:

1234567891011121314
  receivers:
    otlp:
      protocols:
        grpc:
          include_metadata: true # Essential for 'from_context: metadata.*'
        http:
          include_metadata: true # Essential for 'from_context: metadata.*'

  processors:
    resource:
      attributes:
        - key: client_connection.user_agent
          from_context: metadata.user-agent # Extract User-Agent header from incoming request
          action: upsert

• auth.<key>Copy: This provides access to authentication details supplied by server authenticators. The exact keys you can use depend on the specific authenticator deployed; always consult its documentation for available attributes.

otelcol.yaml
123
  - key: client_auth.id
    from_context: auth.client.id # Example if an authenticator exposes a client ID
    action: upsert

• client.addressCopy: This captures the IP address of the client that sent the telemetry data to the Collector. If the context key holds multiple values, they will be concatenated and separated by a semicolon.

otelcol.yaml
123
  - key: collector_ingress.client_ip
    from_context: client.address
    action: upsert

Removing unwanted resource attributes

The deleteCopy action offers a straightforward way to remove resourceCopy attributes. You can target specific attributes by their keyCopy for an exact match, or use a patternCopy for regex-based deletion:

otelcol.yaml
1234567
# Delete a specific resource attribute
- key: legacy_resource_tag
  action: delete

# Delete all resource attributes whose names start with "temp."
- pattern: "^temp\\..*"
  action: delete

Redacting sensitive information

The hashCopy action replaces a resource attribute’s original value with its SHA256 hash.

Similar to the deleteCopy action, you can specify a precise keyCopy or employ a regular expression patternCopy to target attributes:

otelcol.yaml
1234567
# Hash a specific resource attribute value
- key: resource.private_identifier
  action: hash

# Hash any resource attribute containing "secret" in its name
- pattern: ".*secret.*"
  action: hash

For a deeper dive into data redaction techniques, refer to our comprehensive guide on scrubbing sensitive data from logs, traces, and metrics.

Parsing values with regular expressions

The extract action enables you to dissect portions of a resource attribute’s value using a regular expression and then populate these extracted segments into new resource attributes.

For example, imagine your host.name resource attribute contains combined information like "webserver-production-00012", where webserverCopy indicates the host type, productionCopy the deployment environment, and 00012Copy the instance ID. You want to extract these into discrete attributes:

otelcol.yaml
123456
processors:
  resource/extract_host_info:
    attributes:
      - key: host.name # The source resource attribute to extract from
        pattern: "^(?P<host_type>[^-]+)-(?P<deployment_environment_name>[^-]+)-(?P<host_id>\\d+)$"
        action: extract

The extract action mandates both a key (the source attribute) and a regex pattern that must incorporate named capture groups. In this illustration:

• (?P<host_type>[^-]+)Copy captures “webserver” and assigns it to a new host_typeCopy resource attribute.
• (?P<deployment_host_name>[^-]+)Copy captures “production” and assigns it to a new deployment_environment_name resource attribute.
• (?P<host_id>\\d+)Copy captures “00012” and assigns it to a new host_idCopy resource attribute.

The extracted values are automatically added as new resource attributes to the telemetry signal or overwritten if they already exist.

Note that you must use a subsequent action (or another processor) to rename attributes extracted in this manner to the desired OpenTelemetry semantic convention.

Changing attribute types

The convertCopy action allows you to alter the data type of a resource attribute’s value to one of the supported types: intCopy, doubleCopy, or stringCopy. This functionality is particularly useful for normalizing data to align with semantic conventions or specific backend requirements.

otelcol.yaml
123
- key: host.cpu.cache.l2.size
  action: convert
  converted_type: int # Ensures l2 memory cache is represented as an int

If the attribute’s value cannot be successfully converted to the specified type (e.g., attempting to convert the string “xyz” to a doubleCopy), the original value will remain unchanged.

Resource processor tips and best practices

• Action order matters: The sequence in which you define actions is crucial, as they are executed from top to bottom. For instance, if you intend to copy a resource attribute to a new key and then delete the original, ensure the copy action precedes the delete action.
• Validate before deployment: Always use the debug exporter to verify your configuration and confirm that resource attribute modifications behave precisely as anticipated before introducing them to your live system.
• Monitor performance impact: While the resource processor actions are generally efficient, configurations involving complex regex patterns or an extensive number of rules can introduce notable processing overhead.
• Exercise caution with resource identity changes: Modifying resource attributes like service.nameCopy, host.nameCopy, or k8s.pod.uidCopy can impact how your observability backend identifies and groups your services and underlying infrastructure. Always proceed with a clear understanding of these implications.

Final thoughts

The utility of resource attributes is fully realized when your data is sent to OpenTelemetry-native platforms. Once your data, including critical resource context, is received and processed in the Collector, the final step is sending it to such a platform.

At Dash0, we embrace what we call Resource Centricity: all your telemetry, irrespective of which agent or instrumentation collects it, automatically correlates around the rich metadata you provide as resource attributes. This unified, resource-centric view transforms your raw data into actionable insights, making your observability truly powerful.

Take control of your observability data and try Dash0 today by signing up for a free trial.