Inconsistencies and incompleteness in metadata is a significant challenge for organizations. Due to poor metadata hygiene or technical limitations, telemetry coming from different sources (like an OpenTelemetry SDK and a logging agent) often has considerably different metadata.

Background: How PromQL Solves This

In Prometheus, INFO metrics are constant metrics (usually with the value 1) that carry "shared labels" about some component. For example, the node_uname_info metric from Awesome Prometheus Alerts:

promql

1
(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10) * on(instance) group_left (nodename) node_uname_info{nodename=~".+"}

The <metric_1> on(<shared labels>) group_left (<labels to copy left>) <metric_2> syntax specifies that labels from <metric_2> are copied to <metric_1>, provided that the time series match on the values for <shared labels>. This is effectively the PromQL equivalent of SQL's LEFT JOIN.

Resource Coalescing

Resource equality rules create coalesced resources that merge attributes across resource dictionaries coming along with telemetry.

Consider the following example with two spans and one log record, each with different resources but sharing a common k8s.pod.uid:

json

1234567
{
  "resource": {
    "k8s.pod.uid": "1234",
    "service.name": "foo"
  },
  "spans": [{ "name": "Span 1" }]
}

json

12345678
{
  "resource": {
    "k8s.pod.uid": "1234",
    "service.name": "bar",
    "service.namespace": "baz"
  },
  "spans": [{ "name": "Span 2" }]
}

json

1234567
{
  "resource": {
    "k8s.pod.uid": "1234",
    "team.name": "yolo"
  },
  "logRecords": [{ "body": "I like puppies!" }]
}

All three resources are coalesced into one due to the resource equality rule on k8s.pod.uid.

The Challenge with Simple Coalescing

With simple coalescing, the log record "I like puppies!" would be returned when filtering for:

Filter	Result	Reasoning
`service.name = foo`	✅ Match	The log could belong to either service
`service.name = bar`	✅ Match	The log could belong to either service
`service.name is one of foo \ bar`	✅ Match	The log could belong to either service
`service.name = foo AND service.name = bar`	❌ Contradiction	A log should belong to at most one service
`service.name = foo AND service.namespace = baz`	❌ Contradiction	The service `foo` is not in the namespace `baz`

The contradictions result from the fact that the service.* semantic convention namespace should be applied consistently, not mixed from multiple resources.

Enhanced Coalescing Rules

Enhanced resource coalescing combines the concept of PromQL's group_left with a mechanism to avoid merging data in OpenTelemetry resource attribute namespaces that have semantic conflicts.

Rule 0: Do Not Merge Conflicting Namespaces

If resource R₁ contains a key <namespace>.<key> with value A, and another resource R₂ with the same dash0.resource.id contains <namespace>.<key> with value B, no keys prefixed by <namespace>. will be imported from R₂ into R₁.

Note

Conflicts disqualify the entire top-level namespace. For example, if resource R₁ has process.runtime.name=java and R₂ has process.runtime.name=nodejs, no keys starting with process. (like process.pid) will be copied to R₁.

Rule 1: Uncertainty Principle

If resource R₁ contains no keys from the <namespace> namespace, and resources R₂ and R₃ (with the same dash0.resource.id) contain conflicting values for that namespace, R₁ is treated as having EITHER the entirety of the values from R₂ OR from R₃, but not mixed.

Using the earlier example, these filters will match the log record:

service.name = foo
service.name = bar
service.name is one of foo | bar
service.name is one of foo | bar, service.namespace = baz

These filters will NOT match the log record:

service.name = foo AND service.namespace = baz
service.name = foo AND service.name = bar

The intuition: "If it's uncertain which service the telemetry belongs to, it can be any of the known services in this coalesced resource."

Rule 2: Subordination to telemetry.*

A conflict in the telemetry.* namespace (set by OpenTelemetry SDKs) implies a conflict in the process.* namespace.

Rationale: Each SDK implements the telemetry.sdk attributes out of the box, but not all implement the process.* attributes. Without this rule, the lack of process.* support in one SDK could cause process.* attributes from another SDK (in the same pod) to leak in. This rule assumes each process contains at most one OpenTelemetry SDK.

Rule 3: Subordination to container.* and k8s.container.*

A conflict in the container.* or k8s.container.* namespace implies a conflict in:

telemetry.*
k8s.container.*
process.*

Rationale: Different containers mean different processes and different OpenTelemetry SDK instances.

Rule 4: Subordination to os.*

A conflict in the os.* namespace implies a conflict in:

telemetry.*
process.*
container.*

Rationale: Different operating systems mean different processes, containers, and OpenTelemetry SDK instances.

Note

In some cases, the OS reported by one resource may be the host's, while another reports the container userland. This rule may lead to some false negatives, but since os.* is seldom reported outside of host-related monitoring, this is an acceptable trade-off.

Rule 5: Subordination to system.*

A conflict in the system.* namespace implies a conflict in:

os.*
telemetry.*
process.*
container.*

Rationale: Different systems mean different operating systems, processes, containers, and OpenTelemetry SDK instances.

Summary

The enhanced resource coalescing rules ensure that:

Conflicting attribute namespaces are not merged incorrectly
Telemetry can still be found using any of the possible attribute values when uncertainty exists
Contradictory queries (like filtering for two different service names simultaneously) correctly return no results
Namespace hierarchies are respected (container conflicts imply process conflicts, etc.)

Common Resource Attribute Conflict Issues