Let upstream projects handle CRDs¶
- Status: accepted
- Deciders: Compliant Kubernetes Arch Meeting
- Date: 2021-04-29
Technical Story: #446 #369 #391 #402 #436.
Context and Problem Statement¶
CustomResourceDefinitions (CRDs) are tricky. They are essentially a mechanism to change the API of Kubernetes. Helm 2 had zero support for CRDs. Helm 3 has support for installing CRDs, but not upgrading them.
How should we handle CRDs?
- CRDs add complexity and need to be treated specially.
- Generally need to “trim fat” and rely on upstream.
- Install and upgrade CRDs as part of the bootstrap step, which is a Helm 2 legacy.
- Rely on whatever mechanism is proposed by upstream Helm Charts.
Chosen option: "Rely on upstream", because it trims fat and reduces astonishment.
At installation, rely on upstream's approach to install CRDs (see below). At upgrade, propagate upstream migration steps in CK8s migration steps in each release notes. An issue template was created to ensure we won't forget.
Since we "vendor in" all Charts, CRDs can be discovered using:
grep -R 'kind: CustomResourceDefinition'
- Less astonishing, compared to installing Chart "by hand".
- Less maintenance, i.e., there is only one source of truth for CRDs.
- None really.
A detailed audit was performed of all CRDs in Compliant Kubernetes on 2021-04-27.
As a summary, all projects encourage installing CRDs as part of standard
helm install. Most projects encourage following manual migration steps to handle CRDs. Some projects handle CRD upgrades.
A detailed analysis is listed below:
- Installation: The cert-manager Helm Chart includes the
installCRDsvalue -- by default it is set to
false. If set to
true, then CRDs are automatically installed when installing cert-manager, albeit not using the CRDs mechanism provided by Helm.
- Upgrade: CRDs are supposed to be upgraded manually.
Dex can be configured without CRDs. ADR-0012 argues for that approach.
- Installation: Gatekeeper installs CRDs using the mechanism provided by Helm.
- Upgrade: Gatekeeper wants you to either uninstall-install or run a helm_migrate.sh.
- Installation: kube-prometheus-stack installs CRDs using standard Helm mechanism.
- Upgrade: kube-prometheus-stack expects you to run manual upgrade steps.
- Installation: Velero install CRDs using standard Helm mechanism.
- Upgrade: Velero includes magic to upgrade CRDs.