This page is out of date

We are currently working on internal documentation to streamline Compliant Kubernetes onboarding for selected cloud providers. Until those documents are ready, and until we have capacity to make parts of that documentation public, this page is out-of-date.

Nevertheless, parts of it are useful. Use at your own risk and don't expect things to work smoothly.

Compliant Kubernetes on EKS-D based clusters

This document contains instructions on how to install Compliant Kubernetes on AWS using EKS-D.

Note

This guide is written for compliantkubernetes-apps v0.13.0

Requirements

  • An AWS account with billing enabled.
  • A hosted zone in Route53.
  • yq v3.4.1 installed on you machine.
  • gpg2 installed on your machine with at least one key available.
  • kubectl installed on your machine.

Infrastructure and Kubernetes

Get EKS-D

git clone https://github.com/aws/eks-distro.git
cd eks-distro/development/kops
git checkout v1-19-eks-1

Configure your AWS environment

Follow the instructions in Getting Started with kOps on AWS up until you reach Creating your first cluster. Unless you have very specific requirements you shouldn't need to take any action when it comes to the DNS configuration.

If you followed the instructions you should have:

  • An IAM user for kOps with the correct permissions.
  • Set AWS credentials and any other AWS environment variables you require in your shell.
  • An S3 bucket for storing the kOps cluster state.

Create initial kOps cluster configurations

export AWS_REGION=<region where you want the infrastructure to be created>
export KOPS_STATE_STORE=s3://<name of the bucket you created in previous step>

SERVICE_CLUSTER="<xyz, e.g. test-sc>.<your hosted zone in Route53, e.g. example.com>"
WORKLOAD_CLUSTER="<xyz, e.g. test-wc>.<your hosted zone in Route53, e.g. example.com>"

for CLUSTER in ${SERVICE_CLUSTER} ${WORKLOAD_CLUSTER}; do
    export KOPS_CLUSTER_NAME=${CLUSTER}
    ./create_values_yaml.sh
    ./create_configuration.sh
done

Modify kOps cluster configurations

for CLUSTER in ${SERVICE_CLUSTER} ${WORKLOAD_CLUSTER}; do
echo '
---
- command: update
  path: spec.etcdClusters[0].manager
  value:
    env:
    - name: ETCD_LISTEN_METRICS_URLS
      value: http://0.0.0.0:8081
    - name: ETCD_METRICS
      value: basic

- command: update
  path: spec.networking
  value:
    calico:
      encapsulationMode: ipip

- command: update
  path: spec.metricsServer.enabled
  value:
    false

- command: update
  path: spec.kubeAPIServer
  value:
    image: public.ecr.aws/eks-distro/kubernetes/kube-apiserver:v1.19.6-eks-1-19-1
    auditLogMaxAge: 7
    auditLogMaxBackups: 1
    auditLogMaxSize: 100
    auditLogPath: /var/log/kubernetes/audit/kube-apiserver-audit.log
    auditPolicyFile: /srv/kubernetes/audit/policy-config.yaml
    enableAdmissionPlugins:
    - "PodSecurityPolicy"
    - "NamespaceLifecycle"
    - "LimitRanger"
    - "ServiceAccount"
    - "DefaultStorageClass"
    - "DefaultTolerationSeconds"
    - "MutatingAdmissionWebhook"
    - "ValidatingAdmissionWebhook"
    - "ResourceQuota"
    - "NodeRestriction"

- command: update
  path: spec.fileAssets
  value:
  - name: audit-policy-config
    path: /srv/kubernetes/audit/policy-config.yaml
    roles:
    - Master
    content: |
      apiVersion: audit.k8s.io/v1
      kind: Policy
      rules:
      - level: RequestResponse
        resources:
        - group: ""
          resources: ["pods"]
      - level: Metadata
        resources:
        - group: ""
          resources: ["pods/log", "pods/status"]
      - level: None
        resources:
        - group: ""
          resources: ["configmaps"]
          resourceNames: ["controller-leader"]
      - level: None
        users: ["system:kube-proxy"]
        verbs: ["watch"]
        resources:
        - group: "" # core API group
          resources: ["endpoints", "services"]
      - level: None
        userGroups: ["system:authenticated"]
        nonResourceURLs:
        - "/api*" # Wildcard matching.
        - "/version"
      - level: Request
        resources:
        - group: "" # core API group
          resources: ["configmaps"]
        namespaces: ["kube-system"]
      - level: Metadata
        resources:
        - group: "" # core API group
          resources: ["secrets", "configmaps"]
      - level: Request
        resources:
        - group: "" # core API group
        - group: "extensions" # Version of group should NOT be included.
      - level: Metadata
        omitStages:
          - "RequestReceived"
' | yq w -i -s - ${CLUSTER}/${CLUSTER}.yaml
done

# Configure OIDC flags for kube-apiserver.
for CLUSTER in ${WORKLOAD_CLUSTERS}; do
    yq w -i ${CLUSTER}/${CLUSTER}.yaml 'spec.kubeAPIServer.oidcIssuerURL' https://dex.${SERVICE_CLUSTER}
    yq w -i ${CLUSTER}/${CLUSTER}.yaml 'spec.kubeAPIServer.oidcUsernameClaim' email
    yq w -i ${CLUSTER}/${CLUSTER}.yaml 'spec.kubeAPIServer.oidcClientID' kubelogin
done

# Use bigger machines for service cluster worker nodes.
yq w -i -d2 ${SERVICE_CLUSTER}/${SERVICE_CLUSTER}.yaml 'spec.machineType' t3.large

# Update kOps cluster configurations in state bucket.
for CLUSTER in ${SERVICE_CLUSTER} ${WORKLOAD_CLUSTER}; do
    ./bin/kops-1-19 replace -f "./${CLUSTER}/${CLUSTER}.yaml"
done

Create clusters

for CLUSTER in ${SERVICE_CLUSTER} ${WORKLOAD_CLUSTER}; do
    export KOPS_CLUSTER_NAME=${CLUSTER}
    ./create_cluster.sh
done

The creation of the clusters might take anywhere from 5 minutes to 20 minutes. You should run the ./cluster_wait.sh script against all of your clusters as it creates a configmap needed by the aws-iam-authenticator pod, e.g.

for CLUSTER in ${SERVICE_CLUSTER} ${WORKLOAD_CLUSTER}; do
    export KOPS_CLUSTER_NAME=${CLUSTER}
    kubectl config use-context ${CLUSTER}
    timeout 600 ./cluster_wait.sh
done

Compliant Kubernetes Apps

Get Compliant Kubernetes Apps

git clone git@github.com:elastisys/compliantkubernetes-apps
cd compliantkubernetes-apps
git checkout v0.13.0

Install requirements

ansible-playbook -e 'ansible_python_interpreter=/usr/bin/python3' --ask-become-pass --connection local --inventory 127.0.0.1, get-requirements.yaml

Initialize configuration

export CK8S_ENVIRONMENT_NAME=aws-eks-d
#export CK8S_FLAVOR=[dev|prod] # defaults to dev
export CK8S_CONFIG_PATH=~/.ck8s/aws-eks-d
export CK8S_CLOUD_PROVIDER=aws
export CK8S_PGP_FP=<your GPG key ID>  # retrieve with gpg --list-secret-keys
./bin/ck8s init

Three files, sc-config.yaml and wc-config.yaml, and secrets.yaml, were generated in the ${CK8S_CONFIG_PATH} directory.

ls -l ${CK8S_CONFIG_PATH}

Edit configuration files

Edit the configuration files sc-config.yaml, wc-config.yaml and secrets.yaml and set the appropriate values for some of the configuration fields. Note that, the latter is encrypted.

vim ${CK8S_CONFIG_PATH}/sc-config.yaml
vim ${CK8S_CONFIG_PATH}/wc-config.yaml
sops ${CK8S_CONFIG_PATH}/secrets.yaml

You should perform the following changes:

# sc-config.yaml
global:
  baseDomain: "set-me"     # Set to ${SERVICE_CLUSTER}
  opsDomain: "set-me"      # Set to ops.${SERVICE_CLUSTER}
  issuer: letsencrypt-prod
  verifyTls: true
  clusterDNS: 100.64.0.10

storageClasses:
  default: kops-ssd-1-17
  nfs:
    enabled: false
  cinder:
    enabled: false
  local:
    enabled: false
  ebs:
    enabled: false

objectStorage:
  type: "s3"
  s3:
    region: "set-me"          # e.g. eu-north-1
    regionEndpoint: "set-me"  # e.g. https://s3.eu-north-1.amazonaws.com

issuers:
  letsencrypt:
    prod:
      email: "set-me"  # Set to a valid email address
    staging:
      email: "set-me"  # Set to a valid email address

# wc-config.yaml
global:
  baseDomain: "set-me"     # Set to ${WORKLOAD_CLUSTER}
  opsDomain: "set-me"      # Set to ops.${SERVICE_CLUSTER}
  issuer: letsencrypt-prod
  verifyTls: true
  clusterDNS: 100.64.0.10

storageClasses:
  default: kops-ssd-1-17
  nfs:
    enabled: false
  cinder:
    enabled: false
  local:
    enabled: false
  ebs:
    enabled: false

objectStorage:
  type: "s3"
  s3:
    region: "set-me"          # e.g. eu-north-1
    regionEndpoint: "set-me"  # e.g. https://s3.eu-north-1.amazonaws.com


opa:
  enabled: false # Does not work with k8s 1.19+

# secrets.yaml
objectStorage:
  s3:
    accessKey: "set-me" # Set to your AWS S3 accesskey
    secretKey: "set-me" # Set to your AWS S3 secretKey

PSP and RBAC

Since we've enabled the PodSecurityPolicy admission plugin in the kube-apiserver we'll need to create some basic PSPs and RBAC rules that both you and Compliant Kubernetes Apps will need to run workloads.

for CLUSTER in ${SERVICE_CLUSTER} ${WORKLOAD_CLUSTER}; do
kubectl config use-context ${CLUSTER}

# Install 'restricted' and 'privileged' podSecurityPolicies.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/privileged-psp.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/restricted-psp.yaml

# Install RBAC so authenticated users are be able to use the 'restricted' psp.
echo '
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
  name: psp:restricted
rules:
- apiGroups:
  - policy
  resourceNames:
  - restricted
  resources:
  - podsecuritypolicies
  verbs:
  - use

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: psp:any:restricted
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp:restricted
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:authenticated
' | kubectl apply -f -
done

Create placeholder DNS records

To avoid negative caching and other surprises. Create the following records using your favorite tool or you can use the Import zone file feature in Route53:

echo "
*.${SERVICE_CLUSTER}     60s A 203.0.113.123
*.${WORKLOAD_CLUSTER}    60s A 203.0.113.123
*.ops.${SERVICE_CLUSTER} 60s A 203.0.113.123
"

Create S3 buckets

Depending on you configuration you may want to create S3 buckets. Create the following buckets using your favorite tool or via the AWS console:

# List bucket names.
{ yq r ${CK8S_CONFIG_PATH}/wc-config.yaml 'objectStorage.buckets.*';\
    yq r ${CK8S_CONFIG_PATH}/sc-config.yaml 'objectStorage.buckets.*'; } | sort | uniq

# Create buckets using the AWS CLI.
# Assumes that the same bucket is used for velero in both service and workload cluster.
for BUCKET in $(yq r ${CK8S_CONFIG_PATH}/sc-config.yaml 'objectStorage.buckets.*'); do
    aws s3api create-bucket\
        --bucket ${BUCKET} \
        --create-bucket-configuration LocationConstraint=${AWS_REGION}
done

Prepare kubeconfigs

Compliant Kubernetes Apps demands that the kube contexts for the workload and service cluster are found in separate files encrypted with sops.

kubectl config view --minify --flatten --context=${SERVICE_CLUSTER} > ${CK8S_CONFIG_PATH}/.state/kube_config_sc.yaml
sops -e -i --config ${CK8S_CONFIG_PATH}/.sops.yaml ${CK8S_CONFIG_PATH}/.state/kube_config_sc.yaml

kubectl config view --minify --flatten --context=${WORKLOAD_CLUSTER} > ${CK8S_CONFIG_PATH}/.state/kube_config_wc.yaml
sops -e -i --config ${CK8S_CONFIG_PATH}/.sops.yaml ${CK8S_CONFIG_PATH}/.state/kube_config_wc.yaml

Install apps

You can install apps in parallel, although it is recommended to install the service cluster before the workload cluster.

# Service cluster
./bin/ck8s apply sc # Respond "n" if you get a WARN

# Workload cluster
./bin/ck8s apply wc # Respond "n" if you get a WARN

Run the following to get metrics from etcd-manager 

# Service cluster
./bin/ck8s ops helmfile sc -l app=kube-prometheus-stack apply --skip-deps --set kubeEtcd.service.selector.k8s-app=etcd-manager-main --set kubeEtcd.service.targetPort=8081

# Workload cluster
./bin/ck8s ops helmfile wc -l app=kube-prometheus-stack apply --skip-deps --set kubeEtcd.service.selector.k8s-app=etcd-manager-main --set kubeEtcd.service.targetPort=8081

Update DNS records

Now that we've installed all applications, the loadbalancer fronting the ingress controller should be ready. Run the following commands and update the A records in Route53.

sc_lb=$(./bin/ck8s ops kubectl sc -n ingress-nginx get svc ingress-nginx-controller -ojsonpath={.status.loadBalancer.ingress[0].hostname})
wc_lb=$(./bin/ck8s ops kubectl wc -n ingress-nginx get svc ingress-nginx-controller -ojsonpath={.status.loadBalancer.ingress[0].hostname})
sc_lb_ip=$(dig +short ${sc_lb} | head -1)
wc_lb_ip=$(dig +short ${wc_lb} | head -1)

echo "
*.${SERVICE_CLUSTER}     60s A ${sc_lb_ip}
*.${WORKLOAD_CLUSTER}    60s A ${wc_lb_ip}
*.ops.${SERVICE_CLUSTER} 60s A ${sc_lb_ip}
"

Teardown

Compliant Kubernetes Apps

This step is optional. If this is not run you'll have to check and manually remove any leftover cloud resources like S3 buckets, ELBs, and EBS volumes.

git checkout 6f2e386
timeout 180 ./scripts/clean-wc.sh
timeout 180 ./scripts/clean-sc.sh

# Delete buckets
for BUCKET in $(yq r ${CK8S_CONFIG_PATH}/sc-config.yaml 'objectStorage.buckets.*'); do
    aws s3 rb --force s3://${BUCKET}
done

# Delete config repo
rm -rf ${CK8S_CONFIG_PATH}

Remember to also remove the A records from Route53.

Infrastructure and Kubernetes

Enter eks-distro/development/kops and run:

# Destroy clusters and local cluster configurations.
for CLUSTER in ${SERVICE_CLUSTER} ${WORKLOAD_CLUSTER}; do
    export KOPS_CLUSTER_NAME=${CLUSTER}
    ./delete_cluster.sh
    rm -rf ${CLUSTER}
done

You'll have to manually remove the leftover kOps A records from Route53.

# Get names of the A records to be removed.
for CLUSTER in ${SERVICE_CLUSTER} ${WORKLOAD_CLUSTER}; do
    echo kops-controller.internal.${CLUSTER}
done

Finally, you'll also need to remove the ${KOPS_STATE_STORE} from S3 and the IAM user that you used for this guide.