Last updated:
Observability Platform Evaluation Scorecard: 15 Tests for Your PoC (2026)
You can have the fastest queries, the nicest dashboards, and an AI assistant that seems to know your infrastructure better than you do. If the platform fails a mandatory isolation test, though, none of those features can buy it a pass. Security requirements have an irritating habit of remaining requirements.
First, work out what kind of platform you actually need. The interactive profiler below turns your deployment constraints, telemetry requirements, operating capacity, portability needs, and pricing tolerance into a requirements profile, platform categories, and example vendors to research. It does not score or rank vendors. The 15 tests that follow are where candidates have to prove themselves with evidence.
Already have two or three candidates? Use the profiler to check whether you've overlooked a hard constraint, then give every candidate the same workload and acceptance criteria. Otherwise you're comparing sales presentations, not platforms.
Define the platform you need
Start with the requirements that can disqualify a platform outright: deployment and governance. The profiler treats them as constraints, not as a few points to be made up elsewhere by attractive features. It suggests platform categories and example vendors without assigning hidden capability scores. Every example still has to pass the same PoC tests, including Dash0.
Find your observability platform fit
Answer ten questions for a platform profile, investigation categories, and focused PoC tests.
Answer all ten questions to generate your requirements profile and investigation starting points.
You'll get four dimensions: deployment, architecture, signal coverage, and operating ownership. They are independent for a reason. “Self-hosted,” “full-stack,” and “OpenTelemetry-native” aren't mutually exclusive choices on a restaurant menu. A platform can be all three, and you still have to decide who will maintain it.
Use the categories to start your research, then verify each candidate's current deployment options, regions, pricing, integrations, and contractual terms before adding it to the shortlist. If your requirements contradict one another, fix that first. A vendor demo won't resolve a disagreement inside your own team.
Set the rules before you start the trial
Give every vendor the same problem. Otherwise, you'll learn what each vendor is particularly good at demonstrating, which is not necessarily what your team needs to operate. Pick a small, representative slice of your environment: a customer-facing service, a downstream dependency, and an asynchronous worker if you have one. Include the languages and deployment patterns most likely to make instrumentation awkward.
Write down the workload: records and bytes per day, spans per second, active metric series, collection intervals, retention periods, concurrent users, and required regions. Record normal traffic and a plausible incident burst. Freeze the versions, sampling rules, and resource limits used in the comparison.
Then choose three investigation scenarios. For example: a release that slows one customer cohort, a dependency that intermittently times out, and a telemetry pipeline that stops delivering data. Reproduce failures in an approved test environment. Keep a record of the injected cause so you can assess whether an explanation is correct.
Give each criterion an owner and a baseline requirement. Set targets for ingestion delay, query latency, instrumentation overhead, investigation time, and monthly cost using your workload. A two-second query limit might be sensible for one operation and ridiculous for another. Make the target fit the investigation, not the other way around.
Define the requirements that cannot be compensated for by a higher score elsewhere. Data residency, access isolation, critical signal coverage, and a hard budget ceiling are common examples. Fail one, and the candidate is ineligible. Haven't tested it yet? The decision remains open. A nice demo doesn't turn an unknown into a pass.
How to test each criterion
1. Stack coverage and instrumentation effort
Integration catalogs are wonderful for counting logos. They are less helpful when you need to know whether the agent works with your actual framework version, your container security context, and the signals your team depends on. You might have excellent tracing support and discover that metrics never arrive. Or find out the supported framework is the version you upgraded from six months ago. That's a less charming surprise after you've signed the contract.
Start with the systems that matter to your incidents, and test the actual framework versions and deployment patterns. Instrument the representative services and collect the signals each use case requires, including profiles, browser telemetry, synthetic checks, or AI workload telemetry where relevant. Measure engineering time and compare application CPU, memory, and latency with instrumentation enabled and disabled under equivalent load.
What success looks like: Required coverage works in your environment, deployment is repeatable, and setup effort and overhead stay within your agreed limits. Record the proprietary agents, custom exporters, and unsupported components still in the picture. They will be waiting for you when you test how easy it is to leave.
2. OpenTelemetry fidelity and metric correctness
Getting an HTTP success response when you send OTLP is encouraging. It also proves rather less than you might hope. The backend can accept a payload and still change its meaning on the way in: an integer attribute becomes a string, resource attributes such as k8s.pod.name or service.version get flattened or dropped, and a deployment filter stops working. The unpleasant part is that nothing necessarily fails loudly. You discover the missing context when you're trying to investigate a real incident.
Send telemetry with known identifiers, attribute types, resource metadata, and expected metric values. Inspect what arrives and what can actually be queried, through both the UI and the query language.
Metrics need their own test, partly because they can be wrong in a very convincing way. Restart a process and check the temporality your instrumentation uses. Delta values describe an interval; cumulative values accumulate from a start point. Confuse the two, and you can end up with a beautiful, smooth chart describing something that never happened. The OpenTelemetry metrics data model explains why these distinctions affect interpretation, and how OpenTelemetry metrics work walks through aggregation temporality with examples. Check histogram handling and units too.
What success looks like: Required fields remain usable with their original types, transformations are documented and understood, and totals or rates match the controlled workload within an agreed tolerance. Dash0's guide to what makes a good OpenTelemetry backend provides additional fidelity checks.
3. Ingestion freshness and failure recovery
A platform can show fresh telemetry remarkably quickly and still lose part of it along the way. Those are two different tests. Emit identifiable records, measure the time between emission and query visibility using synchronized clocks or a common harness, and count what never arrived. Report the whole latency distribution. A two-second median is not particularly reassuring if the slowest 5% show up after an alert has already evaluated an empty window.
Then do what a sales demo rarely does: break the pipeline on purpose. Interrupt the export path, restore it, and account for generated, intentionally filtered, delivered, duplicated, and lost records. Watch backlog growth and recovery while new data continues to arrive. The Collector's resilience mechanisms include queues and retries, but queue exhaustion and retry limits still cause loss. Dash0's guide to batching, queuing, and retries in the OpenTelemetry Collector covers where that loss happens and how to size for it.
What success looks like: Freshness and recovery meet your baseline, and missing data is detectable. Set the required outage duration and loss budget explicitly; "reliable ingestion" is too vague to verify.
4. Cardinality and schema flexibility
A customer ID looks innocent enough when somebody adds it to a metric label. Then the series count multiplies. Depending on the backend, you might meet a rate limit, a cardinality limit, or an invoice that would have been nice to discuss before merging the pull request. Cardinality is one of those problems that can take its time announcing itself. Each unique Prometheus label combination creates a time series, so adding request IDs to metrics is very different from attaching them to spans. What is cardinality explains the difference between cardinality and churn, which behave differently under load.
Test log and span attribute flexibility separately from metric labels. Search for a synthetic customer identifier, add a new attribute, and check whether existing records remain queryable without re-ingestion or a special index. For metrics, increase the number of label combinations in a bounded test and measure series growth, limits, query latency, and billable usage. Check whether the platform can show you which metrics drive cardinality before they become a problem.
What success looks like: The dimensions you need remain searchable at the expected scale, and limits and costs are visible. Don't reward a promise of "unlimited cardinality" without checking its behavior for each signal.
5. Sampling and telemetry reduction
Sampling looks terrific on a cost report. You're collecting less, the bill is smaller, and the dashboards might barely notice. Then someone asks for the trace of one slow request, which happens to be in the 90% you discarded. The missing trace is not going to come back because you now have a good reason to want it. And if your sampler retained every error, the error rate calculated from those sampled traces won't represent the original traffic either.
Generate known proportions of successful, slow, and failed requests, apply your intended sampling and filtering policies, and check which investigations remain possible. Head and tail sampling make decisions using different information, and a tail sampler cannot recover traces already discarded upstream. Retaining all errors also changes the composition of the stored dataset, so its raw error fraction is not the application's error rate.
What success looks like: Your team can explain and control the policy, find the required failure cases, and distinguish representative metrics from selectively retained traces. Document where full-traffic request and error counts come from. Also test policy changes during a burst.
6. Cross-signal correlation
Putting a log next to a trace is easy. Proving that the log belongs to that trace is the interesting part. If the match is based on a service name and overlapping timestamps, an incident on a busy service can offer plenty of plausible candidates. Very generous of the dashboard. Less helpful to the engineer trying to find the failing request. Trace context provides a stronger relationship, provided the instrumentation actually propagates it.
Begin with a failing request. Trace its downstream work, find the associated logs, and identify the service version and infrastructure involved. Include a queue boundary if asynchronous processing matters. Context propagation supplies the information needed to connect work across services, so first verify that the source telemetry contains that context. Otherwise you may be measuring an instrumentation gap rather than a backend limitation. Traces vs. logs for debugging distributed systems covers how that correlation should work when instrumentation is right.
What success looks like: Responders can follow the relevant relationships without ambiguous matches. Distinguish exact trace-context matches from records that merely share a service and time window. Record any copied identifiers, manual joins, or context lost along the way.
7. Query performance under realistic load
The query in the demo is fast. Of course it is; everyone already knows which query will be run. The useful test is what happens when an engineer asks something nobody prepared for, during a traffic spike, while four colleagues are investigating the same incident. A query engine that looks effortless on a quiet dataset can struggle under that combination.
Build your own suite: find a specific request, compare releases, group errors by an attribute, and inspect a longer historical window. Include queries the vendor hasn't prepared. Run them repeatedly while ingestion continues and several users investigate. Record result correctness, timeouts, scanned data where exposed, and latency distributions. Separate first-run results from repeated queries that may benefit from caching, and keep time range, retention tier, dataset, and concurrency comparable.
What success looks like: The baseline query suite returns correct results within your latency and concurrency targets. Fast answers obtained by dropping required detail belong in the tradeoff record, not in an unqualified performance win.
8. Investigation effectiveness and usability
Ask the person who set up the trial to investigate the incident, and they'll probably look impressively competent. They know where the dashboards are because they put them there. They also know what broke, which rather spoils the exercise. And if everyone tries one platform before the other, the second gets an advantage simply because the team has already worked through the incident once.
Have engineers who did not configure the trial investigate the controlled scenarios. Give each platform equivalent onboarding time, and rotate the order or use equivalent scenarios. Measure time to an evidence-supported explanation, incorrect hypotheses, assistance required, and unresolved questions. Validate the explanation against the injected cause, and record whether the engineer can explain the affected users and the relevant change as well as the failing component.
What success looks like: Intended users reach a correct, supported explanation within your target without a vendor specialist directing the investigation. Report individual trial results; a handful of exercises cannot establish a statistically reliable reduction in production MTTR.
9. Alerting and SLO correctness
Alert configuration is the easy part to admire. The on-call engineer is considerably more interested in whether the alert fires at the right time, reaches the right person, and resolves correctly. Try a brief spike, a low-traffic service where two failures matter a great deal, and a service that stops sending telemetry altogether. That last case deserves particular attention. Silence from the service is not proof that the service is healthy, however much your pager might appreciate the interpretation.
Recreate a critical alert and an SLO with known inputs. Test sustained failure, a brief spike, recovery, low traffic, and missing telemetry. Check routing, grouping, silencing, and the runbook link. Google's SRE guidance on SLO alerting evaluates precision, recall, detection time, and reset time; use those dimensions rather than counting available alert types. If you alert on burn rate, confirm the calculation matches your error-budget definition.
What success looks like: Significant failures trigger the expected notification, recovery behaves correctly, and the SLO calculation matches your definition. Missing data must not silently become evidence that a service is healthy.
10. Retention and historical investigation
A year of retention sounds reassuring. Now ask how long it takes to query month eleven. Depending on the platform, the answer might involve a restore job, a per-scan charge, or discovering that the archived data no longer has the attributes or metric resolution you need. That's a rather different definition of 'available' from the one your engineer has in mind during an investigation.
Repeat a real investigation against data at the age you expect to need it. Check whether raw records, attributes, and metric resolution survive retention changes, and measure restore time and cost. A short trial may not naturally cross a retention boundary, so ask for an aged dataset or a supported lifecycle test. Verify the applicable policy and pricing separately. If a behavior cannot be demonstrated, keep it unverified.
What success looks like: Historical data answers the required questions within your access-time and cost limits.
11. Access control and data governance
The UI might hide a dataset from a restricted user. Good. What happens when the same user calls the API? Or exports the data? Or asks the AI assistant about it? Permissions that work only in the interface you remembered to test are not much of an access-control model. These are exactly the paths worth checking before a contract depends on them.
Create identities for an application team, a restricted responder, and an administrator. Try permitted and forbidden actions through the UI, API, exports, and any AI interface in scope. Revoke a user or token and test again. Use synthetic sensitive values to verify redaction at the intended boundary. Have the appropriate reviewers examine audit evidence, storage and processing locations, deletion behavior, subprocessors, and AI data-handling terms against your organization's requirements.
What success looks like: Required controls are verified and reviewers accept the supporting evidence. A certification badge does not establish tenant isolation or prove that your configuration enforces the access model. Any mandatory control remains a separate gate.
12. Configuration, operating effort, and support
It's lovely watching someone build a dashboard in a demo. The less photogenic question is what happens when that dashboard needs a review, a rollback, or the same change deployed to three environments. If every edit means clicking through a UI, sooner or later those environments will disagree about what they are supposed to monitor.
Deploy a dashboard and alert from version control, modify them, and roll back. Check which steps have supported APIs, which require manual work, and how credential rotation behaves. For managed services, clarify availability commitments, exclusions, and escalation ownership. For self-hosted options, include upgrades, backup recovery, and capacity work. Submit a real support issue during the trial and record the response, while recognizing that one interaction doesn't prove future support quality.
What success looks like: Required changes are repeatable, operating responsibilities are assigned, and measured work fits your staffing budget. Contractual support commitments and a successful trial interaction are separate evidence.
13. Cost predictability and control
An observability platform is liable to be busiest precisely when everything else is having a bad day. Errors produce more telemetry, engineers run heavier queries, and the rest of the team joins the investigation. If your cost model only covers a quiet Tuesday, the incident-month invoice may be an unwelcome sequel.
The billing unit can also change how people use the platform. Per-gigabyte charges can discourage engineers from adding the context that makes OpenTelemetry useful. Per-seat charges can make teams ration access to the very people expected to investigate incidents. Neither tradeoff shows up in the product's feature checklist, but both belong in the evaluation.
Model a normal month, a growth month, and an incident month with a telemetry burst and heavier querying. Use measured quantities and each vendor's billing rules, including retention, seats, support, queries, network traffic, and applicable AI charges. For illustration, 100,000 time series producing one billable point every 30 seconds generate 8.64 billion points in a 30-day month. A backend billed by active series uses a different calculation, and histogram representation and vendor metering may change billable counts. Validate the model against trial usage.
What success looks like: Finance and engineering can reproduce the estimate and reconcile it within an agreed tolerance. Test what budget controls actually do: alerting, throttling, dropping data, and stopping ingestion have very different operational consequences.
14. Portability and exit effort
An OTLP endpoint is not an exit strategy. It makes sending tomorrow's telemetry somewhere else easier. Your dashboards, alert rules, saved queries, and runbook links, unfortunately, do not pack themselves. If they rely on proprietary formats and query languages, moving means translating or rebuilding them, sometimes by hand. Why vendor lock-in is still a problem in 2026 covers where the lock-in has moved.
Route a bounded workload to a second backend, move a useful dashboard and alert, and query an exported historical sample independently. Record required edits, engineering hours, missing features, export throughput, and overlap costs. An open dashboard specification such as Perses helps with definitions, but queries, plugins, and data sources still need compatibility. PromQL expressions likewise depend on the metric names and labels available at the destination.
What success looks like: The tested exit path fits your migration budget, and the required export rights and access periods are documented. Changing an ingestion endpoint is only one part of leaving.
15. AI assistance and automation
An AI assistant can give you a wonderfully coherent explanation of an incident that never happened. It may even present the theory with exactly the confidence you'd like from a senior engineer. Unfortunately, confidence isn't evidence. If the assistant can't separate what the telemetry shows from what it merely suspects, you've made it easier to investigate the wrong thing.
Test it on known-cause incidents, an ambiguous case, and one with essential evidence missing. Check for retrievable supporting telemetry, separation of observations from hypotheses, and honest uncertainty. Repeat cases to expose inconsistent answers. Compare assisted and unassisted investigations under equivalent conditions, and record correctness, verification effort, elapsed time, and usage cost. If the platform can take action, test approval boundaries, scoped credentials, audit trails, and recovery from a failed action in the test environment.
What success looks like: Assistance provides verifiable value and respects the required access and action controls. A confident explanation with no supporting evidence does not pass. If AI has no near-term role in your operations, remove this test from scope before evaluating any candidate.
Run the proof of concept in a controlled sequence
A two-week trial is enough to uncover a lot of inconvenient details, provided you don't spend the first week deciding what to test. Freeze the workload, gates, weights, targets, and test owners before day one. Start security and contract review at the same time; they rarely finish just because the engineers have reached Friday.
Spend the first three days instrumenting services and establishing correctness, freshness, and overhead. That baseline makes everything after it comparable. Use days four to six for stress: bursts, sampling, cardinality, and concurrent queries. Days seven and eight belong to investigations, alert tests, and any AI tasks in scope. Use the last two days to test exit and operating workflows, reconcile costs against trial usage, and write the decision record, including every requirement that's still unverified. Friday afternoon is not an evidence source.
Where possible, send equivalent telemetry to every candidate through the same controlled pipeline. Record backend-side sampling or transformations that make the resulting datasets differ. Also record special trial resources, paid-tier features, and vendor assistance that wouldn't be included in the proposed purchase.
How to interpret the result
The profiler produces a requirements profile and research starting points, not a league table. It applies hard deployment conflicts first, then suggests platform categories whose operating models align with the remaining answers. The example vendors are not scored, and their presence is not evidence that they meet your requirements.
Treat each category as a research direction. The explanation beneath it shows which answers made it relevant. Verify current vendor capabilities with documentation, contractual terms, and the focused PoC tests. Don't skip the remaining tests just because a product sounds like a good match.
Do not let a broad match excuse a failed hard requirement. A platform with excellent investigation workflows is still ineligible if it cannot meet a mandatory residency or isolation rule. Likewise, an unverified requirement remains open rather than becoming a pass or a failure.
When two vendors remain close, run the incident scenarios that expose the tradeoff between them. Measure investigation time, telemetry fidelity, operating effort, and projected cost using the same workload. The purpose is not to manufacture a universal winner. It is to make the important differences reproducible.
Applying the PoC framework to Dash0
Dash0 should be put through the same slightly inconvenient tests as everyone else. Its OpenTelemetry-native architecture gives you specific things to verify, particularly where the backend's handling of resource context, queries, and configuration matters.
For fidelity (2) and correlation (6), Dash0 stores OTLP data as it arrives and ties logs, spans, and metrics to the resources that produced them. Resource semantic conventions help it recognize when signals describe the same pod, host, or service, so an engineer can investigate a slow span alongside telemetry from the same workload. That's the claim worth testing. Bring your own resource attributes, especially the custom ones, and see whether the relationships survive.
For portability (14) and configuration (12), Dash0 queries every signal with PromQL, uses Perses for dashboards, and supports managing dashboards and check rules as code. Export one of each and load it elsewhere as part of the exit test.
For cost (13), Dash0 pricing counts spans, log records, web events, and metric data points, with no per-seat charge. Adding attributes to spans and logs doesn't change the bill, though high-cardinality metric labels do add data points. Check the monthly budget limit and cost forecast against your incident-month model.
SignalControl belongs in the telemetry-reduction test (5), and Agent0 in the AI-assistance test (15). Scope both through dataset permissions during the access-control test (11).
None of this exempts Dash0 from the same tests as every other candidate. Check ingestion, query correctness and performance, access controls, cost, and investigation outcomes against your workload, then record what passed and what still needs evidence.
Final thoughts
The point of a PoC isn't to produce a tidy score that justifies the platform somebody already preferred. It's to find the awkward bits before they become production incidents, surprise bills, or a migration project nobody budgeted for. Define the requirements first, run the same workload everywhere, and make every result traceable to evidence. If a platform fails a mandatory control, it fails. If you haven't tested something, you haven't tested it. There is no need to get creative with the score.
If you'd like to see how an OpenTelemetry-native platform performs against these requirements, try Dash0 free for 14 days with a representative service and apply the same 15 tests. No credit card required.
