Day 6 — Open Policy Agent(OPA) vs Kyverno

Before I dive into the OPA vs. Kyverno comparison, we first need to understand Policy as Code?

What is Policy as Code?

As per definition: Policy as Code is the process of managing and provisioning policy enforcement tooling through machine-readable definition files, rather than best practice documentation or interactive configuration tools (a GUI with buttons to click).

In other words, the main idea behind it is to enforce standards and rules across specific clusters or organizations.

Now we have a policy in place. The next big task is how to enforce standards; that is where we can use tools like

• Open Policy Agent(OPA)
• Kyverno

Open Policy Agent(OPA)

Open Policy Agent(OPA) is a generic policy engine used to enforce policy on a system like Kubernetes, CI/CD, microservices, etc. It does that by using declarative language Rego.

Integrating OPA with Kubernetes

To integrate OPA with Kubernetes, we need to use OPA as an admission controller.

Source: https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

Now the question is what is an admission controller?

In a nutshell, Kubernetes admission controllers are plugins that govern and enforce how the cluster is used. They can be thought of as a gatekeeper that intercept (authenticated) API requests and may change the request object or deny the request altogether. The admission control process has two phases: the mutating phase is executed first, followed by the validatingphase. Consequently, admission controllers can act as mutating or validating controllers or as a combination of both. Reference https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

Installing OPA Gatekeeper as CRD

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml
namespace/gatekeeper-system created
resourcequota/gatekeeper-critical-pods created
customresourcedefinition.apiextensions.k8s.io/configs.config.gatekeeper.sh created
customresourcedefinition.apiextensions.k8s.io/constraintpodstatuses.status.gatekeeper.sh created
customresourcedefinition.apiextensions.k8s.io/constrainttemplatepodstatuses.status.gatekeeper.sh created
customresourcedefinition.apiextensions.k8s.io/constrainttemplates.templates.gatekeeper.sh created
customresourcedefinition.apiextensions.k8s.io/providers.externaldata.gatekeeper.sh created
serviceaccount/gatekeeper-admin created
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/gatekeeper-admin created
role.rbac.authorization.k8s.io/gatekeeper-manager-role created
clusterrole.rbac.authorization.k8s.io/gatekeeper-manager-role created
rolebinding.rbac.authorization.k8s.io/gatekeeper-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/gatekeeper-manager-rolebinding created
secret/gatekeeper-webhook-server-cert created
service/gatekeeper-webhook-service created
deployment.apps/gatekeeper-audit created
deployment.apps/gatekeeper-controller-manager created
Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy/gatekeeper-controller-manager created
validatingwebhookconfiguration.admissionregistration.k8s.io/gatekeeper-validating-webhook-configuration created

• This will create a namespace gatekeeper-system

kubectl get ns
NAME                 STATUS   AGE
default              Active   2d10h
gatekeeper-system    Active   22s
kube-node-lease      Active   2d10h
kube-public          Active   2d10h
kube-system          Active   2d10h
local-path-storage   Active   2d10h

• Verify the gatekeeper CRD is installed successfully

kubectl get crd
NAME                                                 CREATED AT
configs.config.gatekeeper.sh                         2021-11-01T04:11:42Z
constraintpodstatuses.status.gatekeeper.sh           2021-11-01T04:11:42Z
constrainttemplatepodstatuses.status.gatekeeper.sh   2021-11-01T04:11:42Z
constrainttemplates.templates.gatekeeper.sh          2021-11-01T04:11:42Z
providers.externaldata.gatekeeper.sh                 2021-11-01T04:11:42Z

• Verify all the resources in the gatekeeper-system namespace is up and running

kubectl get all -n gatekeeper-system
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/gatekeeper-audit-76bc76bfb6-wsclz                1/1     Running   0          26s
pod/gatekeeper-controller-manager-5c6c56cccf-7jsjf   1/1     Running   0          26s
pod/gatekeeper-controller-manager-5c6c56cccf-8558v   1/1     Running   0          26s
pod/gatekeeper-controller-manager-5c6c56cccf-8wlxb   1/1     Running   0          26s
NAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/gatekeeper-webhook-service   ClusterIP   10.96.43.235   <none>        443/TCP   26s
NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/gatekeeper-audit                1/1     1            1           26s
deployment.apps/gatekeeper-controller-manager   3/3     3            3           26s
NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/gatekeeper-audit-76bc76bfb6                1         1         1       26s
replicaset.apps/gatekeeper-controller-manager-5c6c56cccf   3         3         3       26s

• Now we have all the resources, Pod, and Services running in the gatekeeper-system namespace. The next step is to define policies. In this example, I will create a policy using Rego that denies all pod creation. The first step is to define ConstraintTemplate and Constraint CRD by using Rego.

In the code above once, the count reaches greater than 0(1> 0), policy violation will occur, and the message(msg: msg) will be displayed to the user.

• Create the template

kubectl create -f template.yaml
constrainttemplate.templates.gatekeeper.sh/k8salwaysdeny created

• Verify its create successfully

kubectl get constrainttemplates
NAME            AGE
k8salwaysdeny   22s
> kubectl get crd |grep -i deny
k8salwaysdeny.constraints.gatekeeper.sh              2021-11-01T04:22:28Z

• Next step is to create the constraint which uses this template

Note: Make sure the kind matches the kind defined in the constraint template.

Next, create the constraint and verify it

> kubectl create -f constraints.yaml
k8salwaysdeny.constraints.gatekeeper.sh/deny-pod-creation created
> kubectl get k8salwaysdeny
NAME                AGE
deny-pod-creation   36s

• Verify if the policy is working as expected

> kubectl run nginx --image=nginx
Error from server ([deny-pod-creation] Pod creation denied): admission webhook "validation.gatekeeper.sh" denied the request: [deny-pod-creation] Pod creation denied

• Check the violation under the status section:

> kubectl describe k8sAlwaysDeny deny-pod-creation
Name:         deny-pod-creation
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  constraints.gatekeeper.sh/v1beta1
Kind:         K8sAlwaysDeny
Metadata:
Creation Timestamp:  2021-11-01T04:26:52Z
Generation:          1
Managed Fields:
API Version:  constraints.gatekeeper.sh/v1beta1
Fields Type:  FieldsV1
fieldsV1:
f:status:
Manager:      gatekeeper
Operation:    Update
Time:         2021-11-01T04:26:52Z
API Version:  constraints.gatekeeper.sh/v1beta1
Fields Type:  FieldsV1
fieldsV1:
f:spec:
.:
f:match:
.:
f:kinds:
f:parameters:
.:
f:message:
Manager:         kubectl-create
Operation:       Update
Time:            2021-11-01T04:26:52Z
Resource Version:  1908
UID:               c19da666-371b-4d84-be71-8db885e0f643
Spec:
Match:
Kinds:
API Groups:
Kinds:
Pod
Parameters:
Message:  Pod creation denied
Status:
Audit Timestamp:  2021-11-01T04:27:51Z
By Pod:
Constraint UID:       c19da666-371b-4d84-be71-8db885e0f643
Enforced:             true
Id:                   gatekeeper-audit-76bc76bfb6-wsclz
Observed Generation:  1
Operations:
audit
status
Constraint UID:       c19da666-371b-4d84-be71-8db885e0f643
Enforced:             true
Id:                   gatekeeper-controller-manager-5c6c56cccf-7jsjf
Observed Generation:  1
Operations:
webhook
Constraint UID:       c19da666-371b-4d84-be71-8db885e0f643
Enforced:             true
Id:                   gatekeeper-controller-manager-5c6c56cccf-8558v
Observed Generation:  1
Operations:
webhook
Constraint UID:       c19da666-371b-4d84-be71-8db885e0f643
Enforced:             true
Id:                   gatekeeper-controller-manager-5c6c56cccf-8wlxb
Observed Generation:  1
Operations:
webhook
Total Violations:  17
Violations:
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                gatekeeper-audit-76bc76bfb6-wsclz
Namespace:           gatekeeper-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                gatekeeper-controller-manager-5c6c56cccf-7jsjf
Namespace:           gatekeeper-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                gatekeeper-controller-manager-5c6c56cccf-8558v
Namespace:           gatekeeper-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                gatekeeper-controller-manager-5c6c56cccf-8wlxb
Namespace:           gatekeeper-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                coredns-558bd4d5db-9pwpt
Namespace:           kube-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                coredns-558bd4d5db-bk8w2
Namespace:           kube-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                etcd-my-kind-cluster-control-plane
Namespace:           kube-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                kindnet-hbhv6
Namespace:           kube-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                kindnet-tnh79
Namespace:           kube-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                kindnet-xg4l4
Namespace:           kube-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                kube-apiserver-my-kind-cluster-control-plane
Namespace:           kube-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                kube-controller-manager-my-kind-cluster-control-plane
Namespace:           kube-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                kube-proxy-46bnb
Namespace:           kube-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                kube-proxy-6x9zg
Namespace:           kube-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                kube-proxy-7mtww
Namespace:           kube-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                kube-scheduler-my-kind-cluster-control-plane
Namespace:           kube-system
Enforcement Action:  deny
Kind:                Pod
Message:             Pod creation denied
Name:                local-path-provisioner-547f784dff-g6qwd
Namespace:           local-path-storage
Events:                  <none>

Advantages

• You can express complex policy
• Support Kubernetes and other systems like CI/CD, microservices, etc
• Community support

Disadvantage

• You need to learn a new language(rego) to write policies
• The policy is complex and hard to read

Kyverno

One of the major challenges with OPA is you need to learn a new language rego to enforce the policy. In the case of Kyverno, you don’t need to learn a new language, but it’s specific to Kubernetes.

Installing Kyverno

kubectl create -f https://raw.githubusercontent.com/kyverno/kyverno/release-1.5/definitions/release/install.yaml
namespace/kyverno created
customresourcedefinition.apiextensions.k8s.io/clusterpolicies.kyverno.io created
customresourcedefinition.apiextensions.k8s.io/clusterpolicyreports.wgpolicyk8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterreportchangerequests.kyverno.io created
customresourcedefinition.apiextensions.k8s.io/generaterequests.kyverno.io created
customresourcedefinition.apiextensions.k8s.io/policies.kyverno.io created
customresourcedefinition.apiextensions.k8s.io/policyreports.wgpolicyk8s.io created
customresourcedefinition.apiextensions.k8s.io/reportchangerequests.kyverno.io created
serviceaccount/kyverno-service-account created
clusterrole.rbac.authorization.k8s.io/kyverno:admin-policies created
clusterrole.rbac.authorization.k8s.io/kyverno:admin-policyreport created
clusterrole.rbac.authorization.k8s.io/kyverno:admin-reportchangerequest created
clusterrole.rbac.authorization.k8s.io/kyverno:customresources created
clusterrole.rbac.authorization.k8s.io/kyverno:generatecontroller created
clusterrole.rbac.authorization.k8s.io/kyverno:leaderelection created
clusterrole.rbac.authorization.k8s.io/kyverno:policycontroller created
clusterrole.rbac.authorization.k8s.io/kyverno:userinfo created
clusterrole.rbac.authorization.k8s.io/kyverno:webhook created
clusterrolebinding.rbac.authorization.k8s.io/kyverno:customresources created
clusterrolebinding.rbac.authorization.k8s.io/kyverno:generatecontroller created
clusterrolebinding.rbac.authorization.k8s.io/kyverno:leaderelection created
clusterrolebinding.rbac.authorization.k8s.io/kyverno:policycontroller created
clusterrolebinding.rbac.authorization.k8s.io/kyverno:userinfo created
clusterrolebinding.rbac.authorization.k8s.io/kyverno:webhook created
configmap/kyverno created
configmap/kyverno-metrics created
service/kyverno-svc created
service/kyverno-svc-metrics created
deployment.apps/kyverno created
Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy/kyverno created

• Create the pod which enforce the policy that all the pod must have labels app.kubernetes.io/name

• Create the policy

kubectl create -f policy.yaml
clusterpolicy.kyverno.io/require-labels created

• Verify if it’s working as expected

> kubectl run nginx --image nginx
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
resource Pod/default/nginx was blocked due to the following policies
require-labels:check-for-labels: 'validation error: label ''app.kubernetes.io/name'' is required.
Rule check-for-labels failed at path /metadata/labels/app.kubernetes.io/name/'

Advantage

• Kubernetes style
• Easy to read

Disadvantages

• As it’s not using any programming language, writing complex policies may be difficult
• Relatively new

Conclusion

Whatever policy engine you choose entirely depends upon your requirement and your use case, but it’s essential to implement policies to secure your Kubernetes cluster.