Using Custom Root of Trust in Welkin

Welkin has support for custom root of trust as to be able to comply with certain information security regulations.

The custom root of trust feature in Welkin provides the means to configure the cryptographic root of trust in your (encryption) key infrastructure. Typical uses include setting up and managing a certificate authority (CA) service that issues all certificates within an enterprise, to ensure that no outside dependencies exist and that all certificates are controlled by an internal department. The custom root of trust feature makes Welkin, and services deployed onto it, trust only certificates that have been issued by a set of CAs your organization deems trustworthy, whether that is only your internal one, or a default list of public CAs in addition to your internal one. This page explains how Platform Administrators can enable the custom root trust feature in Welkin.

Preparing the Cluster Nodes

To start of, if your Management Cluster is using a custom certificate authority used for TLS of services such as Dex and Harbor, you will need the Nodes of the Kubernetes Clusters to trust your custom certificate(s).

Kubespray

For environments set up with Kubespray, the following ansible-playbook can be used to add a custom root certificate custom-rootcert.crt to each Node in a Cluster:

# install-rootcert.yml
---
- name: Install custom root CA certificate
  hosts: k8s_nodes
  become: yes

  tasks:
    - name: Copy root CA certificate to trust store
      ansible.builtin.copy:
        src: custom-rootcert.crt # this cert needs to exist in the same folder as the playbook
        dest: /usr/local/share/ca-certificates/custom-rootcert.crt
        owner: root
        group: root
        mode: '0644'
      notify: Update ca-certificates

  handlers:
    - name: Update ca-certificates
      ansible.builtin.command:
        cmd: update-ca-certificates
      listen: update ca-certificates

Run the playbook above with the following command for both the Management Cluster and Workload Cluster:

CLUSTER=<sc|wc>
ansible-playbook -i "${CK8S_CONFIG_PATH}/${CLUSTER}-config/inventory.ini" install-rootcert.yml # add --ask-become-pass if your nodes require password for sudo commands

Note

You may have to restart containerd manually for it to pick up the certificate. This is needed for containerd to trust e.g. Harbor with a custom root of trust. For Kubespray environments you can run the following Ansible command to restart containerd:

CLUSTER=<sc|wc>
ansible -i "${CK8S_CONFIG_PATH}/${CLUSTER}-config/inventory.ini" all -m shell -a 'sudo systemctl restart containerd'

Cluster API

For environments set up with Cluster API, the following config can be used to add a custom root certificate custom-rootcert.crt to each Node in a Cluster.

clusterDefaults:
  nodeGroupDefaults:
    kubeadmConfigSpec:
      preKubeadmCommands:
        - update-ca-certificates
      files:
        - path: "/usr/local/share/ca-certificates/custom-rootcert.crt"
          owner: "root:root"
          permissions: "0644"
          content: |-
            <CERTIFICATE DATA HERE>
  controlPlane:
    kubeadmConfigSpec:
      preKubeadmCommands:
        - update-ca-certificates
      files:
        - path: "/usr/local/share/ca-certificates/custom-rootcert.crt"
          owner: "root:root"
          permissions: "0644"
          content: |-
            <CERTIFICATE DATA HERE>

Preparing the Welkin Configuration

In Welkin, the custom root of trust feature is supported with the help of trust-manager for provisioning bundles containing a custom root of trust certificate, and Kyverno policies used to mutate Pods to inject such bundles. The feature can be enabled by configuring the following in common-config.yaml:

Note

If you initialized your config using CK8S_FLAVOR=air-gapped, most of the config below will already be set by default with some values being set to set-me to ensure they are properly configured before applying Welkin.

trustManager:
  enabled: true
  defaultBundle:
    name: default-bundle
    inline: | # add custom certificate that should be trusted in your Welkin environment
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
    useDefaultCAs: true # whether or not default CAs should be included in the bundle

  # You can create multiple extra bundles and specify which namespace to distribute them to
  extraBundles: []
  # - name: set-me
  #   inLine: |
  #     set-me
  #   useDefaultCAs: true
  #   configMapAnnotations: {}
  #   configMapLabels:
  #     elastisys.io/trust-bundle: "custom"
  #   pemKey: "ca-certificates.crt"
  #   jksKey: ""
  #   pkcs12Key: ""
  #   matchNamespaceLabels: {}

kyverno:
  enabled: true
  policies:
    injectCertificates:
      enabled: true
      mode: OptOut # possible options are `OptIn` and `OptOut`

Injection for Pods is controlled by what mode is configured, either OptOut or OptIn. In OptIn mode, Pods are injected by adding the annotation elastisys.io/inject-custom-ca and setting the value to the name of the bundle you want a Pod to inject. If using OptOut mode, all Pods are injected with the configured default bundle. Pods can be opt-out by setting the annotation with an empty string as value: elastisys.io/inject-custom-ca="" For a Pod to pick up and use the bundle, the injection adds the SSL_CERT_FILE environment variable to point to the mount path of the bundle, and also adds SSL_CERT_DIR=/dev/null to better guarantee that the application running actually picks it up.

Applications part of the Welkin platform that are known to require custom root of trust in environments where external services uses custom certificates will always be injected with the default bundle, independent of what mode is used. For more details on the behavior of how injection is done based on mode, please refer to the schema. For more details on the configuration available for trust-manager bundles, see the schema here.

Using a custom issuer

You can use a custom issuer in Welkin to provision your custom certificate. See example below for how to set it up in common-config.yaml with a custom ACME issuer:

# common-config.yaml
global:
  issuer: custom-root # this is set by default when using air-gapped flavor
issuers:
  letsencrypt:
    enabled: false # this is already false by default for dev and air-gapped flavor
  customACME: # for more information about acme issuer in cert-manager, refer to https://cert-manager.io/docs/configuration/acme/
    enabled: true
    server: https://ca.example.domain/acme/acme/directory
    email: admin.example@example.com
    solvers: # below are example for solvers for both dns01 and http01
      - selector:
          dnsZones:
            - example.domain
        dns01:
          acmeDNS:
            host: https://acmedns.example.domain
            accountSecretRef:
              name: acme-dns
              key: acmedns.json
      - selector: {}
        http01:
          ingress:
            ingressClassName: nginx
      - selector: {}
        http01:
          ingress:
            ingressClassName: traefik

You can also configure extra issuers by configuring it under extraIssuers:

extraIssuers: # ref: https://cert-manager.io/docs/configuration/
  - apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: selfsigned-issuer
      namespace: sandbox
    spec:
      selfSigned: {}

Trusting Custom Root of Trust

Note

These steps assumes you are running a Debian-based OS. Especially Welkin assumes Ubuntu for its dependencies.

When operating on environments that uses a custom root of trust, you may want your local machine to be able to trust it. This can be done by adding the public root certificates into /usr/local/share/ca-certificates and then running update-ca-certificates on your local machine.