Last updated:
Upgrading and Uninstallation
This guide covers upgrading the Dash0 operator, CRD version migrations, disabling monitoring for namespaces, and uninstallation procedures.
Table of Contents
Upgrading the Operator
To upgrade the Dash0 operator to a newer version, run the following commands:
helm repo update dash0-operator
helm upgrade --wait --namespace dash0-system dash0-operator dash0-operator/dash0-operatorThe Helm hook that runs after helm install and helm upgrade waits up to four minutes per attempt until the rollout
of the operator deployment has finished and the operator's webhooks are reachable.
If the rollout takes longer, for example because pulling the images is slow, helm install or helm upgrade fails,
also without --wait.
Use a longer --timeout in this case, which allows up to three attempts of the hook.
In namespaces with
instrumentWorkloads.mode: pods, the
operator does not restart pods after an upgrade.
Existing pods keep the instrumentation of the previous operator version until they are recreated.
Operator Version 0.158.0: Operator-Managed Webhook Certificates
Starting with version 0.158.0, the operator manages the TLS certificates for its webhooks instead of the Helm chart, see Webhook Certificates. This does not apply if you use cert-manager. When upgrading from a version before 0.158.0:
- Helm CLI and Flux: No preparation is needed.
Once the rollout has finished, you can delete the Secret
dash0-operator-certificates. - Flux with drift detection: Add the following ignore rules to the
HelmRelease, and keep them. Otherwise, Flux resets thecaBundlevalues to those of the last upgrade whenever the operator changes them, and requests to the webhooks fail briefly each time.yamlspec: driftDetection: mode: enabled ignore: - paths: ["/webhooks/0/clientConfig/caBundle"] target: kind: MutatingWebhookConfiguration - paths: ["/webhooks/0/clientConfig/caBundle"] target: kind: ValidatingWebhookConfiguration - paths: ["/spec/conversion/webhook/clientConfig/caBundle"] target: kind: CustomResourceDefinition name: dash0monitorings.operator.dash0.com - ArgoCD: Prepare the ArgoCD application before the upgrade, see Notes on ArgoCD: Upgrading to Version 0.158.0.
- Pipelines that render the chart without access to the cluster, for example
helm templatefollowed bykubectl apply, or Kustomize'shelmCharts: Requests to the operator's webhooks fail from applying the manifests until the new operator pod has written the CA bundle, with the effects described for ArgoCD without preparation.
Rolling back to a version before 0.158.0:
- Helm CLI and Flux:
helm rollbackrestores the certificates of that release. Requests to the webhooks fail until the first operator pod of the old version is ready. The certificate of the old release expires 365 days after it has been generated. - ArgoCD: Remove the
ignoreDifferencesentries for the certificates before syncing back, see Notes on ArgoCD: Upgrading to Version 0.158.0.
CRD Version Upgrades
Occasionally, the custom resource definitions (CRDs) used by the Dash0 operator (Dash0OperatorConfiguration, Dash0Monitoring) will be updated to new versions. Whenever possible, this will happen in a way that requires no manual intervention by users. This section contains details about CRD version updates and version migrations.
Operator Version 0.71.0: v1alpha1 to v1beta1 Migration
With operator version 0.71.0, the Dash0 operator's Dash0Monitoring custom resource definition (CRD) is upgraded from
version v1alpha1 to v1beta1.
The operator handles both versions correctly, that is version v1alpha1 to v1beta1 are both fully supported.
Here is what you need to know about this version update for Dash0Monitoring:
- If you have existing
Dash0Monitoringresources in versionv1alpha1in your cluster, they will be automatically converted on the fly, for example when the Dash0 operator reads thev1alpha1resource version. At some point Kubernetes might also convert the resource permanently and store it in versionv1beta1. - After the upgrade to version 0.71.0, you can still deploy new
Dash0Monitoringresource in versionv1alpha1(for example viakubectl apply). They will be automatically converted and stored asv1beta1resources by Kubernetes. - If you want to migrate a
Dash0Monitoringtemplate (e.g. a yaml file) from versionv1alpha1tov1beta1, follow these steps:- If the template specifies the workload instrumentation mode via
spec.instrumentWorkloads, replace that withspec.instrumentWorkloads.mode. That is:becomesyamlspec: instrumentWorkloads: created-and-updatedIf the template does not specify the workload instrumentation mode explicitly (that is, it relies on using the default instrumentation mode), no change is necessary here.yamlspec: instrumentWorkloads: mode: created-and-updated - If the template contains the attribute
spec.prometheusScrapingEnabled, replace that withspec.prometheusScraping.enabled. That is:becomesyamlspec: prometheusScrapingEnabled: trueThe attributeyamlspec: prometheusScraping: enabled: truespec.prometheusScraping.enabledis also already valid forv1alpha1, so this particular change can be applied independently of the CRD version change. If the template does not specify whether prometheusScraping is enabled or not (that is, it relies on using the default value), no change is necessary here.
- If the template specifies the workload instrumentation mode via
- We recommend to update your templates from
v1alpha1tov1beta1at some point. However, there are currently no plans to remove support for versionv1alpha1. - If you want to use the new trace context propagators option that has been added in version 0.71.0 (see
Auto-Instrumentation), you need to use version
v1beta1of theDash0Monitoringresource. This includes updating your Yaml templates to that version, as described above. - After upgrading to operator version 0.71.0 or later, you can no longer easily downgrade to a version before 0.71.0.
In particular, this downgrade would require to manually delete all
Dash0Monitoringresources in the cluster. The reason is that the Dash0Monitoring resources are now stored as versionv1beta1by Kubernetes and there is no automatic downward conversion fromv1beta1back tov1alpha1. - The only supported version for
Dash0OperatorConfigurationis stillv1alpha1, that is, trying to useoperator.dash0.com/v1beta1/Dash0OperatorConfigurationwill not work.
Uninstallation
To remove the Dash0 operator from your cluster, run the following command:
helm uninstall dash0-operator --namespace dash0-systemDepending on the command you used to install the operator, you may need to use a different Helm release name or namespace.
This will also automatically disable Dash0 monitoring for all namespaces by deleting the Dash0 monitoring resources in
all namespaces.
All workload modifications applied by the Dash0 operator will be reverted.
This will restart the pods of all workloads that were previously instrumented by the Dash0 operator.
In namespaces with
instrumentWorkloads.mode: pods, pods
that have been instrumented when they were created keep their instrumentation until they are recreated.
Only workloads that still have instrumentation in their pod template from before the namespace was switched to
mode: pods are reverted, which restarts their pods.
Optionally, after helm uninstall has finished, remove the namespace that has been created for the operator:
kubectl delete namespace dash0-systemIf you choose to not remove the namespace, you might want to consider removing the secret with the Dash0 authorization token (if such a secret has been created):
kubectl delete secret --namespace dash0-system dash0-authorization-secretIf you later decide to install the operator again, you will need to perform the initial configuration steps again:
- Set up a Dash0 backend connection, and
- Enable Dash0 monitoring in each namespace you want to monitor, see Enable Dash0 Monitoring For a Namespace.
Unsupported Uninstallation Procedures
Warning: Do not use the following uninstallation procedures:
- Do not delete the Dash0 operator controller deployment manually, always use
helm uninstallto remove the operator. - Do not delete the Dash0 operator's namespace before running
helm uninstall(this would also implicitly delete the operator deployment).
Deleting the Dash0 operator deployment without running helm uninstall will lead to an inconsistent state.
In particular, the operator's admission webhooks are still registered, but the service that responds to the webhook
requests has been removed, so all webhook requests will time out.
This will make requests to delete Dash0 monitoring resources fail.
In addition, the service that is responsible for removing the finalizer from the Dash0 monitoring resources is no longer
there.
In turn, this will make it harder to delete namespaces with a Dash0 monitoring resource, the namespace will get stuck in
the "Terminating" state, due to the finalizer in the monitoring resource no longer being handled correctly.
Recovery from Improper Uninstallation
To rectify an improper uninstallation, follow these steps:
- Delete all Dash0 validating/mutating webhook configs manually (exact command depends on the Helm release name):
console
kubectl delete validatingwebhookconfiguration --ignore-not-found dash0-operator-monitoring-validator dash0-operator-operator-configuration-validator dash0-operator-signal-control-validator kubectl delete mutatingwebhookconfiguration dash0-operator-injector dash0-operator-monitoring-mutating dash0-operator-operator-configuration-mutating - Remove the finalizer from all Dash0 monitoring resources:
console
kubectl patch dash0monitorings <name> -n <namespace> --type=json -p='[{"op":"remove","path":"/metadata/finalizers"}]' - Delete the Dash0 monitoring resources:
console
kubectl delete dash0monitorings <name> -n <namespace>
Related Documentation
- Installation - Initial installation instructions
- Configuration - Configuration options
- Troubleshooting - Troubleshooting common issues