My road to Certified Kubernetes Security Specialist (CKS)

WARNING: Before reading this doc :-) :-)

1: As everyone needs to sign NDA with CNCF, I can’t tell you the exact question asked during the exam neither I have GB of memory, but I can give you the pointer what to expect in the exam

2: As we all know, Kubernetes world updates everyday, so some of the stuff might not be relevant after a few days/weeks/month.

3: Please don’t ask for any exam dumps or questions; that defeats the whole purpose of the exam.

I have been preparing for the Kubernetes exam for the last two years (~1 year for CKS). I purchased the exam voucher, then canceled it, re-purchased it again, booked the exam, and then withdrew it, till I reached the point where I only had a few months left either to write the exam or lose $$$.

Exam Preparation

Topics

You must need to know these 6 concepts to clear the exam

  • Admission Controller
  • AppArmor
  • NetworkPolicy
  • Falco
  • PSP
  • Auditing

Some low hanging fruits are

  • Secrets
  • Trivy
  • RBAC
  • CIS Benchmarks
  • Runtimeclasses(gvisor)

I am sharing few of the tricks to make you life easier. These involved some of mistake that I have done during preparation, as most of these concepts are new to me

  • Filename and the profile name may be different.
# cat /etc/apparmor.d/backend#include <tunables/global>profile my-backend
  • As you can see in the above case filename is backend but the profile name is my-backend. So make sure in the annotation you need to specify profile name(my-backend) not the filename.
annotations:container.apparmor.security.beta.kubernetes.io/nginx: localhost/my-backend
  • Similarly the key you are specifying for annotation took container name not the pod name
key: container.apparmor.security.beta.kubernetes.io/<container_name>

So in the below pod definition file,container name is nginx but the pod name is my-site. Make sure as an annotation specify the container name not the pod name.

apiVersion: v1kind: Podmetadata:labels:run: nginxname: my-sitenamespace: omniannotations:container.apparmor.security.beta.kubernetes.io/nginx: localhost/restricted-frontendspec:containers:- image: nginx:alpineimagePullPolicy: IfNotPresentname: nginx
  • Annotation will look like this
annotations:container.apparmor.security.beta.kubernetes.io/nginx: localhost/my-backend

For more info check the Apparmor doc https://kubernetes.io/docs/tutorials/clusters/apparmor/

2. Secrets

  • Remember the command to decode the password
echo "cGFzc3dvcmQK" |base64 -d

OR if you are comfortable with jsonpath you can directly extract the secret from the secret itself

kubectl -n demo get secrets my-super-secret -o jsonpath='{.data.SUPER_PASSWORD}' | base64 --decode

3. Extracting the image name

  • If you have multiple pod in namespace and you want to get the image name
kubectl describe pod -n demo |grep "Image:" |awk '{print $2}’

Similarly if you are comfortable jsonpath

kubectl -n demo get pods -o json | jq -r '.items[].spec.containers[].image'

4. With trivy you can pass multiple severity level like HIGH and CRITICAL

trivy image --severity HIGH,CRITICAL
  • You can write a simple for loop to go over the image with severity HIGH and CRITICAL
for image in `kubectl describe pod -n demo |grep "Image:" |awk '{print $2}'`; do trivy image --severity HIGH,CRITICAL $image; done

5. Location of seccomp profiles

ps aux|grep -i kubeletroot 3099 5.6 3.6 1959932 73840 ? Ssl 13:51 5:34 /usr/bin/kubelet — bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf — kubeconfig=/etc/kubernetes/kubelet.conf — config=/var/lib/kubelet/config.yaml — network-plugin=cni — pod-infra-container-image=k8s.gcr.io/pause:3.2# ls -l /var/lib/kubelet/seccomp/profiles/total 4-rw-r — r — 1 root root 40 Sep 19 15:29 audit.json
  • To verify if the seccomp profile is loaded
# kubectl describe pod audit-nginx |grep -i seccomp
Annotations: seccomp.security.alpha.kubernetes.io/pod: localhost/profiles/audit.json

For more info check https://kubernetes.io/docs/tutorials/clusters/seccomp/

6. Before making any changes to static pod, don’t forget to take the backup

cp /etc/kubernetes/manifests/kube-apiserver.yaml /root

In case if something got messed, you have an option to revert it or check the logs under

# cd /var/log/pods/root@controlplane:/var/log/pods# ls -ltrtotal 60drwxr-xr-x 3 root root 4096 Sep 19 13:51 kube-system_kube-controller-manager-controlplane_a875134e700993a22f67999011829566drwxr-xr-x 3 root root 4096 Sep 19 13:51 kube-system_etcd-controlplane_2f8bb5f0185e15d99a68ed2d44d2f886drwxr-xr-x 3 root root 4096 Sep 19 13:51 kube-system_kube-scheduler-controlplane_81d2d21449d64d5e6d5e9069a7ca99eddrwxr-xr-x 3 root root 4096 Sep 19 13:52 kube-system_kube-proxy-bqjr9_a144abe0-9cf4-44f0-97cb-d55799e0fecbdrwxr-xr-x 5 root root 4096 Sep 19 13:52 kube-system_weave-net-wztjk_a57bcb51-b36d-4f14-8a26-f7821b8e768bdrwxr-xr-x 3 root root 4096 Sep 19 13:52 kube-system_coredns-74ff55c5b-24q65_978b7833-c1fb-4a15-82ce-33d4a20c5425drwxr-xr-x 3 root root 4096 Sep 19 13:52 kube-system_coredns-74ff55c5b-76rf4_2c66c2b7-bf13-4fbf-8489-ba98356d9063drwxr-xr-x 3 root root 4096 Sep 19 15:01 delta_simple-webapp-2_ef6cebd9-55fe-4ed7-bebd-224fb8840934drwxr-xr-x 3 root root 4096 Sep 19 15:03 default_app-1401_51453c92-cc07-45d1-a8e7-091dd4022a70drwxr-xr-x 3 root root 4096 Sep 19 15:03 magnum_app-0403_b8742e0f-2daf-4b8f-8bef-b2f9b3f82336drwxr-xr-x 3 root root 4096 Sep 19 15:03 default_image-bouncer-webhook-f95dfcbdd-9rgh9_e251d15d-05c1-452f-9af6-d1c37b2155c5drwxr-xr-x 3 root root 4096 Sep 19 15:09 omni_frontend-site_0d920b3b-35e5-40a4-a02e-9b2f4ade53e1drwxr-xr-x 3 root root 4096 Sep 19 15:21 orion_app-xyz_0b795e17-51d8-44ed-b2b3-f18c7759b8c4drwxr-xr-x 3 root root 4096 Sep 19 15:30 default_audit-nginx_5aeaf92c-5ec3-4607-abe4-09692896fcf0drwxr-xr-x 3 root root 4096 Sep 19 15:32 kube-system_kube-apiserver-controlplane_815c2fc080069ad9aee6a41d37ae1744

After making any changes, always verify if the pod is up in kube-system namespace

# kubectl get pod -n kube-system

You can run it in loop either using -w flag or using watch command

watch kubectl get pod -n kube-system

7. Falco

  • Always check if falco service is up and running(In exam you don’t need to install, it come pre-installed)
systemctl status falco
  • To get the log for falco service
journalctl -fu falco
  • To list all syscall
falco --list=syscall

This is important if you don’t want to go to falco official doc during the exam to look for events https://falco.org/docs/rules/supported-fields/

Remember file_output is disabled by default, don’t forget to enable it

file_output:enabled: false → truekeep_alive: falsefilename: ./events.txt

8. For gvisor the name of the class is in small

spec:
runtimeClassName: gvisor

As it mentioned in camelcase with V capital at most if the places sometime its confusing(gVisor)

9. When you want to perform a rolebinding using service account remember the format is namespace:service account

--serviceaccount=namespace:serviceaccountname

So your rolebinding argument accept both namespace and then service account

kubectl create myrolebinding my-role-binding --role=my-write --serviceaccount=demo:developer

10. Don’t Forget to restart the daemon For e.g. If using a Kubelet config file, edit the file to set authorization: mode to Webhook

systemctl daemon-reload
systemctl restart kubelet.service

11. Auditing

  • You can’t enable/create auditing by using kubectl
kubectl create -f /etc/kubernetes/my-audit.yamlerror: unable to recognize "/etc/kubernetes/prod-audit.yaml": no matches for kind "Policy" in version "audit.k8s.io/v1”
  • Sample my-audit.yaml file look like this
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# Log pod changes at RequestResponse level
-
level: RequestResponse
resources:
-
group: ""
# Resource "pods" doesn't match requests to any subresource of pods,
# which is consistent with the RBAC policy.
resources: ["pods"]
  • To enable it
--audit-policy-file=/etc/kubernetes/audit-policy.yaml \
--audit-log-path=/var/log/audit.log
  • and don’t forget to mount the volume
volumeMounts:
-
mountPath: /etc/kubernetes/audit-policy.yaml
name: audit
readOnly: true
-
mountPath: /var/log/audit.log
name: audit-log
readOnly: false
  • and finally configure the hostPath:
...
-
name: audit
hostPath:
path: /etc/kubernetes/audit-policy.yaml
type: File
- name: audit-log
hostPath:
path: /var/log/audit.log
type: FileOrCreate

Reference: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

In case something got messed up, check Kube-api server logs

# cd /var/log/pods/kube-system_kube-apiserver-controlplane_815c2fc080069ad9aee6a41d37ae1744
# tail -f kube-apiserver/0.log

12. Don’t forget to enable PodSecurity Policy

- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy

13. You can’t create admission controller using kubectl

kubectl create -f /etc/kubernetes/pki/admission_configuration.yamlerror: unable to recognize "/etc/kubernetes/pki/admission_configuration.yaml": no matches for kind "AdmissionConfiguration" in version "apiserver.config.k8s.io/v1”

Sample admission controller config file look like this

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
-
name: ImagePolicyWebhook
configuration:
imagePolicy:
kubeConfigFile: <path-to-kubeconfig-file>
allowTTL: 50
denyTTL: 50
retryBackoff: 500
defaultAllow: true
  • To enable it
- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ImagePolicyWebhook- --admission-control-config-file=/etc/kubernetes/pki/admission_configuration.yaml

For more info

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

Some of the tips I mentioned here is too basic for few people but as I am new to this, these are some of the mistakes I have done while preparing for this exam. Hopefully you will find these tips useful but if you have any other tips you have encountered please leave it in the comment section.

In the end I will say this exam is challenging, so stay calm and give your best shot.

The best way to connect with me is via any of the below mediums

AWS Community Builder, Ex-Redhat, Author, Blogger, YouTuber, RHCA, RHCDS, RHCE, Docker Certified,4XAWS, CCNA, MCP, Certified Jenkins, Terraform Certified, 1XGCP