Continuous profiling is now GA

Last updated:

Explore Profiles

Select an OpenTelemetry profile type and move from a time-series datapoint to code-level analysis in the Profiling Explorer.

Open Profiling under Telemetry to see All profiles in the selected dataset.

Select the data to investigate

  1. Select the dataset in the top-right corner.
  2. Set the time range with the time picker.
  3. Choose a measurement in Profile type. The screenshots use On-CPU.
  4. Use Filter profiles... to select a workload.

Start with one workload and a time range that contains the behavior you want to explain. Combining unrelated workloads can hide a hotspot. Check the profile type and units before interpreting values.

Read the metrics chart

Set Chart to Metrics to show the time-series chart. For On-CPU, CPU cores shows CPU consumption over time. With Sum by set to None, the legend shows All samples. Choose an attribute in Sum by to split the chart into workload-specific series.

CPU cores chart split by Kubernetes pod name, above the profile visualization

Look for spikes or sustained changes, and choose a baseline for comparison. Read the axis label and units for the selected profile type. See CPU cores for how the CPU unit relates to elapsed time.

Select a datapoint to inspect profiles for its interval. When the explorer shows a Merged profiles of query summary, check the start, end, and duration before reading the code view. A datapoint represents an interval of profiling data, not an individual function call.

To investigate what changed, use Compare Profiles.

Choose a code view

Choose a view in the code visualization toolbar:

  • Flame Graph: Start here to inspect aggregate call paths. Click a frame to expand it and focus the flame graph on it. Use Group by to separate samples by attributes.
  • Table: Find functions by name and inspect their Self and Total contributions.
  • Flame Chart: Inspect samples along a time axis and separate them into timelines with Samples group by.
  • Sandwich: Inspect the paths leading into a frame and the functions it calls.

See Key Concepts for how these views represent profiling data. Use the differential Flame Graph when comparing two selections.

Make room for code analysis

Select None beside Chart to hide the metrics chart. Select Metrics to restore it.

Use the visualization's Settings to adjust frame colors, function-name alignment, and the small-frame visibility threshold. See Analyze Flame Graphs.

Save an investigation view

Click Save as in the header to save the current view. A saved view is most useful for stack and frame filters you apply repeatedly. For example, save a view that applies the Hide V8 internals preset so you can open a Node.js service's profiles with only its JavaScript code.

The saved view also keeps query changes, such as an attribute selected in Sum by.

If the result is empty or unexpected

  • No profiles: Check the dataset, time range, profile type, and profile filters. Check the ingestion token's signal types and permissions. See Collect and Send Profiles for collection and forwarding setup.
  • Too much unrelated code: Narrow the workload with profile filters, then apply stack or frame filters in the visualization.
  • Frames show memory addresses instead of function names: Symbolization failed, usually because debuginfo is missing for a C, C++, Rust, or Go binary. See Symbolize Profiles.
  • CPU time does not explain request latency: On-CPU profiles account for execution, not time spent waiting. Collect off-CPU profiles to see where threads block, for example on locks or I/O. The operator-managed Parca Agent supports opt-in off-CPU profiling. For waiting on other services, continue in Tracing.

Further Reading