This page is out of date

We are currently working on internal documentation to streamline Compliant Kubernetes onboarding for selected cloud providers. Until those documents are ready, and until we have capacity to make parts of that documentation public, this page is out-of-date.

Nevertheless, parts of it are useful. Use at your own risk and don't expect things to work smoothly.

Compliant Kubernetes Deployment on Exoscale¶

This document contains instructions on how to setup a service cluster and a workload cluster in Exoscale. The following are the main tasks addressed in this document:

Infrastructure setup for two clusters: one service and one workload cluster
Deploying Compliant Kubernetes on top of the two clusters.
Creating DNS Records
Deploying Rook Storage Orchestration Service
Deploying Compliant Kubernetes apps

The instructions below are just samples, you need to update them according to your requirements. Besides, the exoscale cli is used to manage DNS. If you are using any other DNS service provider for managing your DNS you can skip it.

Before starting, make sure you have all necessary tools.

Note

This guide is written for compliantkubernetes-apps v0.17.0

Setup¶

Choose names for your service cluster and workload cluster, as well as a name for your environment:

SERVICE_CLUSTER="testsc"
WORKLOAD_CLUSTERS=( "testwc0" )
CK8S_ENVIRONMENT_NAME=my-environment-name

Infrastructure Setup using Terraform¶

Before trying any of the steps, clone the Elastisys Compliant Kubernetes Kubespray repo as follows:

git clone --recursive https://github.com/elastisys/compliantkubernetes-kubespray
cd compliantkubernetes-kubespray/kubespray

Expose Exoscale credentials to Terraform¶

For authentication create the file ~/.cloudstack.ini and put your Exoscale credentials in it. The file should look like something like this:

[cloudstack]
key = <API key>
secret = <API secret>

Customize your infrastructure¶

Create a configuration for the service and the workload clusters:

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
  cp -r inventory/sample inventory/$CLUSTER
  cp contrib/terraform/exoscale/default.tfvars inventory/$CLUSTER/
done

Review and, if needed, adjust the files in inventory/$CLUSTER/default.tfvars, where $CLUSTER is the cluster name:

Use different value for the prefix field in /default.tfvars for the two clusters. Failing to do so will result in a name conflict.
Set a non-zero value for ceph_partition_size field, e.g., "ceph_partition_size": 50, as it will be used by Rook storage service to provide local disk storage.
To security harden your cluster, set ssh_whitelist and api_server_whitelist to the IP addresses from which you expect to operate the cluster.
Make sure you configure your SSH keys in ssh_public_keys.

Important

The Linux Ubuntu 20.04 LTS 64-bit image on Exoscale is regularly upgraded, which might cause unexpected changes during terraform apply. Consider uploading your own dated Ubuntu image to reduce the risk of downtime.

Initialize and Apply Terraform¶

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    pushd contrib/terraform/exoscale
    terraform init
    terraform apply \
        -var-file=../../../inventory/$CLUSTER/default.tfvars \
        -state=../../../inventory/$CLUSTER/tfstate-$CLUSTER.tfstate
    cp inventory.ini ../../../inventory/$CLUSTER/
    popd
done

Important

The Terraform state is stored in inventory/$CLUSTER/tfstate-$CLUSTER.tfstate, where $CLUSTER is the cluster name. It is precious. Consider backing it up or using Terraform Cloud.

You should now have inventory file named inventory/$CLUSTER/inventory.ini for each cluster that you can use with kubespray.

Test access to all nodes¶

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    pushd inventory/$CLUSTER
    ANSIBLE_HOST_KEY_CHECKING=False ansible all -i inventory.ini -m ping
    popd
done

Deploying vanilla Kubernetes clusters using Kubespray¶

With the infrastructure provisioned, we can now deploy Kubernetes using kubespray. First, if you haven't done so already, install the pre-requisites and change to the compliantkubernetes-kubespray root directory.

pip3 install -r requirements.txt

cd ..

Init the Kubespray config in your config path¶

export DOMAIN=<your_domain> # DNS domain to expose the services inside the service cluster i.e. "example.com"
export CK8S_CONFIG_PATH=~/.ck8s/exoscale
export CK8S_PGP_FP=<your GPG key fingerprint>  # retrieve with gpg --list-secret-keys

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    ./bin/ck8s-kubespray init $CLUSTER default $CK8S_PGP_FP
done

Copy the generated inventory files in the right location¶

Please copy the two inventory files, kubespray/inventory/$CLUSTER/inventory.ini, generated by Terraform to ${CK8S_CONFIG_PATH}/$CLUSTER-config/, where $CLUSTER the name of each cluster (i.e., testsc, testwc0).

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    cp kubespray/inventory/$CLUSTER/inventory.ini ${CK8S_CONFIG_PATH}/$CLUSTER-config/
done

Run kubespray to deploy the Kubernetes clusters¶

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    ./bin/ck8s-kubespray apply $CLUSTER --flush-cache
done

This may take up to 10 minutes for each cluster, 20 minutes in total.

Correct the Kubernetes API IP addresses¶

Locate the encrypted kubeconfigs in ${CK8S_CONFIG_PATH}/.state/kube_config_*.yaml and edit them using sops. Copy the public IP address of the load balancer from inventory files ${CK8S_CONFIG_PATH}/*-config/inventory.ini and replace the private IP address for the server field in ${CK8S_CONFIG_PATH}/.state/kube_config_*.yaml.

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml
done

Test access to the clusters as follows¶

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops exec-file ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml \
        'kubectl --kubeconfig {} get nodes'
done

Create the DNS Records¶

You will need to setup a number of DNS entries for traffic to be routed correctly. Determine the public IP of the load-balancer fronting the Ingress controller of the clusters from the Terraform state file generated during infrastructure setup.

To get the load-balancer IP, run the following command:

SC_INGRESS_LB_IP_ADDRESS=$(terraform output -state kubespray/inventory/$SERVICE_CLUSTER/tfstate-$SERVICE_CLUSTER.tfstate -raw ingress_controller_lb_ip_address)
echo $SC_INGRESS_LB_IP_ADDRESS

Configure the exoscale CLI:

exo config

Then point these domains to the service cluster using 'exoscale cli' as follows:

exo dns add A $DOMAIN -a $SC_INGRESS_LB_IP_ADDRESS -n *.ops.$CK8S_ENVIRONMENT_NAME
exo dns add A $DOMAIN -a $SC_INGRESS_LB_IP_ADDRESS -n *.$CK8S_ENVIRONMENT_NAME

Deploy Rook¶

To deploy Rook, please go to the compliantkubernetes-kubespray repo root directory and run the following.

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops --decrypt ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml > $CLUSTER.yaml
    export KUBECONFIG=$CLUSTER.yaml
    ./rook/deploy-rook.sh
    shred -zu $CLUSTER.yaml
done

Please restart the operator pod, rook-ceph-operator*, if some pods stalls in initialization state as shown below:

rook-ceph     rook-ceph-crashcollector-minion-0-b75b9fc64-tv2vg    0/1     Init:0/2   0          24m
rook-ceph     rook-ceph-crashcollector-minion-1-5cfb88b66f-mggrh   0/1     Init:0/2   0          36m
rook-ceph     rook-ceph-crashcollector-minion-2-5c74ffffb6-jwk55   0/1     Init:0/2   0          14m

Important

Pods in pending state usually indicate resource shortage. In such cases you need to use bigger instances.

Test Rook¶

To test Rook, proceed as follows:

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops exec-file ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml 'kubectl --kubeconfig {} apply -f https://raw.githubusercontent.com/rook/rook/release-1.5/cluster/examples/kubernetes/ceph/csi/rbd/pvc.yaml';
done

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops exec-file ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml 'kubectl --kubeconfig {} get pvc';
done

You should see PVCs in Bound state. If you want to clean the previously created PVCs:

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops exec-file ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml 'kubectl --kubeconfig {} delete pvc rbd-pvc';
done

Deploying Compliant Kubernetes Apps¶

Now that the Kubernetes clusters are up and running, we are ready to install the Compliant Kubernetes apps.

Clone `compliantkubernetes-apps` and Install Pre-requisites¶

If you haven't done so already, clone the compliantkubernetes-apps repo and install pre-requisites.

git clone https://github.com/elastisys/compliantkubernetes-apps.git
cd compliantkubernetes-apps
ansible-playbook -e 'ansible_python_interpreter=/usr/bin/python3' --ask-become-pass --connection local --inventory 127.0.0.1, get-requirements.yaml

Initialize the apps configuration¶

export CK8S_ENVIRONMENT_NAME=my-environment-name
#export CK8S_FLAVOR=[dev|prod] # defaults to dev
export CK8S_CONFIG_PATH=~/.ck8s/my-cluster-path
export CK8S_CLOUD_PROVIDER=# [exoscale|safespring|citycloud|aws|baremetal]
export CK8S_PGP_FP=<your GPG key fingerprint>  # retrieve with gpg --list-secret-keys

./bin/ck8s init

This will initialise the configuration in the ${CK8S_CONFIG_PATH} directory. Generating configuration files sc-config.yaml and wc-config.yaml, as well as secrets with randomly generated passwords in secrets.yaml. This will also generate read-only default configuration under the directory defaults/ which can be used as a guide for available and suggested options.

ls -l $CK8S_CONFIG_PATH

Configure the apps¶

Edit the configuration files ${CK8S_CONFIG_PATH}/sc-config.yaml, ${CK8S_CONFIG_PATH}/wc-config.yaml and ${CK8S_CONFIG_PATH}/secrets.yaml and set the appropriate values for some of the configuration fields. Note that, the latter is encrypted.

vim ${CK8S_CONFIG_PATH}/sc-config.yaml
vim ${CK8S_CONFIG_PATH}/wc-config.yaml
sops ${CK8S_CONFIG_PATH}/secrets.yaml

Tip

The default configuration for the service cluster and workload cluster are available in the directory ${CK8S_CONFIG_PATH}/defaults/ and can be used as a reference for available options.

Warning

Do not modify the read-only default configurations files found in the directory ${CK8S_CONFIG_PATH}/defaults/. Instead configure the cluster by modifying the regular files ${CK8S_CONFIG_PATH}/sc-config.yaml and ${CK8S_CONFIG_PATH}/wc-config.yaml as they will override the default options.

The following are the minimum change you should perform:

# ${CK8S_CONFIG_PATH}/sc-config.yaml and ${CK8S_CONFIG_PATH}/wc-config.yaml
global:
  baseDomain: set-me # set to $CK8S_ENVIRONMENT_NAME.$DOMAIN
  opsDomain: set-me  # set to ops.$CK8S_ENVIRONMENT_NAME.$DOMAIN
  issuer: letsencrypt-prod

objectStorage:
  # type: s3 # set as default for prod flavor, defaults to "none" for dev
  s3:
    region: set-me          # Region for S3 buckets, e.g. ch-gva-2
    regionEndpoint: set-me  # Region endpoint for S3 buckets, e.g. https://sos-ch-gva-2.exo.io
    # forcePathStyle: false # set as default

clusterAdmin:
  users: # set to the cluster admin users
    - set-me
    - admin@example.com

# ${CK8S_CONFIG_PATH}/sc-config.yaml (in addition to the changes above)
user:
  grafana:
    oidc:
      allowedDomains: # set to your domain(s), or unset using [] to deny all
        - set-me
        - example.com
harbor:
  # persistence:
  #  type: objectStorage    # set as default for prod flavor, defaults to "filesystem" for dev
  #  disableRedirect: false # set as default
  oidc:
    groupClaimName: set-me # set to group claim name used by OIDC provider
    adminGroupName: set-me # name of the group that automatically will get admin

elasticsearch:
  extraRoleMappings: # set to configure elasticsearch access, or unset using []
    - mapping_name: kibana_user
      definition:
        users:
          - set-me
    - mapping_name: kubernetes_log_reader
      definition:
        users:
          - set-me
    - mapping_name: all_access
      definition:
        users:
          - set-me

alerts:
  opsGenieHeartbeat:
    # enabled: true # set as default for prod flavour, defaults to "false" for dev
    name: set-me    # set to name the heartbeat if enabled

issuers:
  letsencrypt:
    prod:
      email: set-me # set this to an email to receive LetsEncrypt notifications
    staging:
      email: set-me # set this to an email to receive LetsEncrypt notifications

# ${CK8S_CONFIG_PATH}/wc-config.yaml (in addition to the changes above)
user:
  namespaces: # set this to create user namespaces, or unset using []
    - set-me
    - production
    - staging
  adminUsers: # set this to create admins in the user namespaces, or unset using []
    - set-me
    - admin@example.com
  adminGroups: # set this to create admin groups in the user namespaces, or unset using []
    - set-me
  # alertmanager: # add this block to enable user accessible alertmanager
  #   enabled: true
  #   namespace: alertmanager # note that this namespace must be listed above under "user.namespaces"

opa:
  imageRegistry:
    URL: # set this to the allowed image registry, or unset using [] to deny all
      - set-me
      - harbor.example.com

# ${CK8S_CONFIG_PATH}/secrets.yaml
objectStorage:
  s3:
    accessKey: set-me # set to your s3 accesskey
    secretKey: set-me # set to your s3 secretKey

Create S3 buckets¶

You can use the following script to create required S3 buckets. The script uses s3cmd in the background and gets configuration and credentials for your S3 provider from ${HOME}/.s3cfg file.

# Use your default s3cmd config file: ${HOME}/.s3cfg
scripts/S3/entry.sh create

Important

You should not use your own credentials for S3. Rather create a new set of credentials with write-only access, when supported by the object storage provider (check a feature matrix).

Test S3¶

To ensure that you have configured S3 correctly, run the following snippet:

(
    access_key=$(sops exec-file ${CK8S_CONFIG_PATH}/secrets.yaml 'yq r {} "objectStorage.s3.accessKey"')
    secret_key=$(sops exec-file ${CK8S_CONFIG_PATH}/secrets.yaml 'yq r {} "objectStorage.s3.secretKey"')
    sc_config=$(yq m ${CK8S_CONFIG_PATH}/defaults/sc-config.yaml ${CK8S_CONFIG_PATH}/sc-config.yaml -a overwrite -x)
    region=$(echo ${sc_config} | yq r - 'objectStorage.s3.region')
    host=$(echo ${sc_config} | yq r -  'objectStorage.s3.regionEndpoint')

    for bucket in $(echo ${sc_config} | yq r -  'objectStorage.buckets.*'); do
        s3cmd --access_key=${access_key} --secret_key=${secret_key} \
            --region=${region} --host=${host} \
            ls s3://${bucket} > /dev/null
        [ ${?} = 0 ] && echo "Bucket ${bucket} exists!"
    done
)

Install Compliant Kubernetes apps¶

Start with the service cluster:

ln -sf $CK8S_CONFIG_PATH/.state/kube_config_${SERVICE_CLUSTER}.yaml $CK8S_CONFIG_PATH/.state/kube_config_sc.yaml
./bin/ck8s apply sc  # Respond "n" if you get a WARN

Then the workload clusters:

for CLUSTER in "${WORKLOAD_CLUSTERS[@]}"; do
    ln -sf $CK8S_CONFIG_PATH/.state/kube_config_${CLUSTER}.yaml $CK8S_CONFIG_PATH/.state/kube_config_wc.yaml
    ./bin/ck8s apply wc  # Respond "n" if you get a WARN
done

Settling¶

Important

Leave sufficient time for the system to settle, e.g., request TLS certificates from LetsEncrypt, perhaps as much as 20 minutes.

You can check if the system settled as follows:

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops exec-file ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml \
        'kubectl --kubeconfig {} get --all-namespaces pods'
done

Check the output of the command above. All Pods needs to be Running or Completed.

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    sops exec-file ${CK8S_CONFIG_PATH}/.state/kube_config_$CLUSTER.yaml \
        'kubectl --kubeconfig {} get --all-namespaces issuers,clusterissuers,certificates'
done

Check the output of the command above. All resources need to have the Ready column True.

Testing¶

After completing the installation step you can test if the apps are properly installed and ready using the commands below.

Start with the service cluster:

ln -sf $CK8S_CONFIG_PATH/.state/kube_config_${SERVICE_CLUSTER}.yaml $CK8S_CONFIG_PATH/.state/kube_config_sc.yaml
./bin/ck8s test sc  # Respond "n" if you get a WARN

Then the workload clusters:

for CLUSTER in "${WORKLOAD_CLUSTERS[@]}"; do
    ln -sf $CK8S_CONFIG_PATH/.state/kube_config_${CLUSTER}.yaml $CK8S_CONFIG_PATH/.state/kube_config_wc.yaml
    ./bin/ck8s test wc  # Respond "n" if you get a WARN
done

Done. Navigate to the endpoints, for example grafana.$BASE_DOMAIN, kibana.$BASE_DOMAIN, harbor.$BASE_DOMAIN, etc. to discover Compliant Kubernetes's features.

Teardown¶

Removing Compliant Kubernetes Apps from your cluster¶

To remove the applications added by compliant kubernetes you can use the two scripts clean-sc.sh and clean-wc.sh, they are located here in the scripts folder.

They perform the following actions:

Delete the added helm charts
Delete the added namespaces
Delete any remaining PersistentVolumes
Delete the added CustomResourceDefinitions

Note: if user namespaces are managed by Compliant Kubernetes apps then they will also be deleted if you clean up the workload cluster.

Remove infrastructure¶

To teardown the infrastructure, please switch to the root directory of the exoscale branch of the Kubespray repo (see the Terraform section).

for CLUSTER in ${SERVICE_CLUSTER} "${WORKLOAD_CLUSTERS[@]}"; do
    pushd contrib/terraform/exoscale
    terraform init
    terraform destroy \
        -var-file=../../../inventory/$CLUSTER/default.tfvars \
        -state=../../../inventory/$CLUSTER/tfstate-$CLUSTER.tfstate
    popd
done

# Remove DNS records
exo dns remove $DOMAIN *.ops.$CK8S_ENVIRONMENT_NAME
exo dns remove $DOMAIN *.$CK8S_ENVIRONMENT_NAME

Compliant Kubernetes Deployment on Exoscale¶

Setup¶

Infrastructure Setup using Terraform¶

Expose Exoscale credentials to Terraform¶

Customize your infrastructure¶

Initialize and Apply Terraform¶

Test access to all nodes¶

Deploying vanilla Kubernetes clusters using Kubespray¶

Init the Kubespray config in your config path¶

Copy the generated inventory files in the right location¶

Run kubespray to deploy the Kubernetes clusters¶

Correct the Kubernetes API IP addresses¶

Test access to the clusters as follows¶

Create the DNS Records¶

Deploy Rook¶

Test Rook¶

Deploying Compliant Kubernetes Apps¶

Clone compliantkubernetes-apps and Install Pre-requisites¶

Initialize the apps configuration¶

Configure the apps¶

Create S3 buckets¶

Test S3¶

Install Compliant Kubernetes apps¶

Settling¶

Testing¶

Teardown¶

Removing Compliant Kubernetes Apps from your cluster¶

Remove infrastructure¶

Further Reading¶

Clone `compliantkubernetes-apps` and Install Pre-requisites¶