Continuous profiling is now GA

Last updated:

Inspect Samples over Time

Use the profiling flame chart and sample grouping to investigate when profiling samples occur across workloads.

Select Flame Chart to examine samples along a time axis. Use it to distinguish short bursts from sustained activity that an aggregate flame graph can hide.

Flame Chart with Samples group by set to two attributes and sparse and dense sample groups on a timeline

Group samples

  1. Set the profile filters and time range for the workload you want to inspect.
  2. Select Flame Chart.
  3. Open Samples group by in the flame chart toolbar.
  4. Select the attributes that identify where samples run.

When the profiles carry them, the flame chart groups samples by node and logical CPU number by default. The screenshot uses this kind of grouping, shown as 2 selected, with k8s.node.name in its row labels.

Choose a grouping in which no two samples can overlap in time on one row. One CPU on one node runs a single call stack at any moment, so node and CPU is a valid grouping. Node, process, and thread is also valid. Grouping by pod alone is not, because a pod can run on several CPUs at once.

Samples group by controls the timeline groups. Sum by above the metrics chart controls the chart's series. You can use different groupings for each part of the explorer.

Look for activity patterns

Decide first which code should occupy the CPUs. For a workload that should keep every core busy, such as query execution, its stacks should fill the entire timeline of each CPU.

  • Interruptions: Other processes appear on a CPU between your workload's samples. The kernel scheduled other work on that CPU. Consider moving that work to another node so your workload runs uninterrupted.
  • Gaps: For On-CPU, a gap means the profiler recorded no work on that CPU during that period. Check that the view does not hide it: group by node and CPU and remove other profile and code filters, because filters also remove samples of other processes on that CPU.
  • Bursts: Samples cluster in a short period instead of spreading throughout the interval.
  • Uneven activity: Compare sample patterns across similar groups to choose a workload for closer inspection.

Use the metrics chart and its units to quantify the selected measurement. See Profile and sample.

Read the flame chart's timestamps rather than carrying over horizontal positions from the aggregate flame graph.

Continue the investigation

Use Add filter to narrow the visible stacks or frames. Switch to Flame Graph to inspect aggregate call paths, or Table to rank functions.

If an interval differs from normal activity, compare its profiles against a baseline datapoint.

Further Reading