Skip to content

Use of Credentials

Welkin interacts with a lot of credentials. This document captures all of them in an orderly fashion, layer-by-layer.

Terminology

  • Purpose: Why are these credentials necessary, what can be done with them.
  • Owner: The person (e.g., John Smith) or computing system (e.g., control plane Node, Pod) who controls the credentials, and is responsible for their safe storage and usage.
  • Type: Individual credentials identify a person, while service accounts identify a computing system.
  • Use for: What should these credentials be used for.
  • Do not use for: When should these credentials NOT be used, although they technically could.

Single Sign-On (SSO) Credentials

  • Example: Company Google Accounts
  • Purpose: authenticate a person with various system, in particular
    • Kubernetes API via Dex
    • Grafana via Dex
    • OpenSearch Dashboards via Dex
    • Harbor via Dex
  • Owner: individual person (user or administrator)
  • Type: individual credentials
  • Use for: identifying yourself
  • Do not use for:
    • These credentials are super valuable and should not be shared with anyone, not even family, friends, workmates, etc., even if requested. Report such sharing requests.
  • Misc:
    • Protect using 2FA

Infrastructure Provider Credentials

  • Purpose: create infrastructure, e.g., VMs, load balancers, networks, buckets.
  • Owner: administrator
  • Type: individual credentials
  • Use for:
    • Terraform layer in Kubespray
    • Creating and destroying buckets via helper scripts
  • Do not use for:

SSH Keys

  • Purpose: access Nodes for setup, break glass or disaster recovery
  • Owner: administrator
  • Type: individual credentials
  • Use for:
    • Accessing Nodes via SSH
  • Do not use for:
    • Giving a system access to a Git repository. Create a separate SSH key only for that purpose instead.
  • Important considerations:

PGP Keys

  • Purpose: encrypt/decrypt sensitive information, e.g., service account credentials, Application Developer names, incident reports, financial information, etc.
  • Owner: administrator
  • Type: individual credentials
  • Use for:
    • Encrypting/decrypting sensitive information
  • Do not use for:
    • Encrypting/decrypting individual credentials. These are meant to be individual and never shared.
    • Encrypting/decrypting SSH key. These are meant to be individual and never shared. Prefer protecting your SSH key with a passphrase or storing it on a YubiKey.
    • Encrypting non-sensitive information. This leads to a culture of "security by obscurity" in which people over-rely on encryption. Prefer being mindful about what data you store and why. If unsure, prefer not storing credentials, as Infrastructure Provider Credentials and SSH keys should be enough to restore any access.
  • Important considerations:

Cloud Controller (Integration) Credentials

  • Purpose: allow Kubernetes control Nodes, specifically the cloud-controller-manager, to create LoadBalancers and PersistentVolumes
  • Owner: each Kubernetes cluster should have their own
  • Type: service account
  • Use for:
    • Configuring Kubespray to set up a Kubernetes cluster with cloud integration
  • Do not use for:
    • AWS. Use AWS IAM Node Roles instead.
    • Exoscale. We currently don't integrate with Exoscale for LoadBalancer or PersistentVolumes.
    • Terraform layer in Kubespray

Backup and Long-Term Logging Credentials

  • Purpose:
    • Allow backup of various components, e.g., PVCs via Velero, Thanos metrics, OpenSearch Indexes, PostgreSQL databases.
    • Allow long-term logging, e.g., Management Cluster logs
  • Owner: each Welkin cluster should have their own
  • Type: service account
  • Use for:
    • Backup
    • Logging
  • Do not use for:
    • Other object storage, e.g., Harbor container images
    • Disaster recovery, investigations. Use Infrastructure Provider credentials instead.
  • Misc:

OpsGenie Credentials

  • Purpose:
    • Allow the Cluster to issue alerts to OpsGenie.
  • Owner: each Welkin cluster should have their own
  • Type: service account
  • Use for: alerting
  • Do not use for:

Dex OpenID Client Secret

  • Purpose:
    • Complete the "OAuth dance" between Grafana, OpenSearch Dashboard, Harbor and kubectl, on one side, and Dex, on the other side.
    • Used both by administrators and users.
  • Owner: each Welkin cluster should have their own
  • Type: not secret
  • Misc:
    • We have determined that the OpenID client secret should not be treated as a secret. See risk analysis here and here.

Kubeconfig with OpenID Authentication

  • Purpose: access the Kubernetes API in normal situations
  • Owner: shared between administrators and users
  • Type: not secret
  • Use for:
    • Routine checks
    • Routine maintenance
    • Investigations
    • "Simple" recovery
  • Misc:
    • If these credentials become unusable, you are in a "break glass" situation. Use Infrastructure Provider credentials or SSH keys to initiate disaster recovery.

Kubeconfig with Client Certificate Key

  • Purpose: access the Kubernetes API for disaster recovery, break glass or initial setup
  • Owner: shared between administrators
  • Type: special
  • Use for:
    • Initial setup
    • Break glass
    • Disaster recovery
  • Do not use for:
    • Routine maintenance or investigation. Use Kubeconfig with OpenID Authentication
  • Misc:
    • Such a Kubeconfig is available on all control plane Nodes at /etc/kubernetes/admin.conf. SSH into a control plane Node then type sudo su and you can readily use kubectl commands.
    • Unless absolutely necessary, avoid storing this file outside the control plane Nodes.
    • If, for some good reason, you downloaded this file, shred it after usage.