[Superseded by ADR-0046] Let upstream projects handle CRDs¶
- Status: superseded by ADR-0046
- Deciders: Compliant Kubernetes Arch Meeting
- Date: 2021-04-29
Technical Story: #446 #369 #391 #402 #436.
Context and Problem Statement¶
CustomResourceDefinitions (CRDs) are tricky. They are essentially a mechanism to change the API of Kubernetes. Helm 2 had zero support for CRDs. Helm 3 has support for installing CRDs, but not upgrading them.
How should we handle CRDs?
Decision Drivers¶
- CRDs add complexity and need to be treated specially.
- Generally need to “trim fat” and rely on upstream.
Considered Options¶
- Install and upgrade CRDs as part of the bootstrap step, which is a Helm 2 legacy.
- Rely on whatever mechanism is proposed by upstream Helm Charts.
Decision Outcome¶
Chosen option: "Rely on upstream", because it trims fat and reduces astonishment.
At installation, rely on upstream's approach to install CRDs (see below). At upgrade, propagate upstream migration steps in CK8s migration steps in each release notes. An issue template was created to ensure we won't forget.
Since we "vendor in" all Charts, CRDs can be discovered using:
grep -R 'kind: CustomResourceDefinition'
Positive Consequences¶
- Less astonishing, compared to installing Chart "by hand".
- Less maintenance, i.e., there is only one source of truth for CRDs.
Negative Consequences¶
- None really.
Detailed Audit¶
A detailed audit was performed of all CRDs in Compliant Kubernetes on 2021-04-27.
As a summary, all projects encourage installing CRDs as part of standard helm install
. Most projects encourage following manual migration steps to handle CRDs. Some projects handle CRD upgrades.
A detailed analysis is listed below:
cert-manager¶
- Installation: The cert-manager Helm Chart includes the
installCRDs
value -- by default it is set tofalse
. If set totrue
, then CRDs are automatically installed when installing cert-manager, albeit not using the CRDs mechanism provided by Helm. - Upgrade: CRDs are supposed to be upgraded manually.
Dex¶
Dex can be configured without CRDs. ADR-0012 argues for that approach.
gatekeeper¶
- Installation: Gatekeeper installs CRDs using the mechanism provided by Helm.
- Upgrade: Gatekeeper wants you to either uninstall-install or run a helm_migrate.sh.
Prometheus (kube-prometheus-stack)¶
- Installation: kube-prometheus-stack installs CRDs using standard Helm mechanism.
- Upgrade: kube-prometheus-stack expects you to run manual upgrade steps.
Velero¶
- Installation: Velero install CRDs using standard Helm mechanism.
- Upgrade: Velero includes magic to upgrade CRDs.