Day 6 — Open Policy Agent(OPA) vs Kyverno

Before I dive into the OPA vs. Kyverno comparison, we first need to understand Policy as Code?

What is Policy as Code?

As per definition: Policy as Code is the process of managing and provisioning policy enforcement tooling through machine-readable definition files, rather than best practice documentation or interactive configuration tools (a GUI with buttons to click).

In other words, the main idea behind it is to enforce standards and rules across specific clusters or organizations.

Now we have a policy in place. The next big task is how to enforce standards; that is where we can use tools like

  • Open Policy Agent(OPA)
  • Kyverno

Open Policy Agent(OPA)

Open Policy Agent(OPA) is a generic policy engine used to enforce policy on a system like Kubernetes, CI/CD, microservices, etc. It does that by using declarative language Rego.

Integrating OPA with Kubernetes

To integrate OPA with Kubernetes, we need to use OPA as an admission controller.

Source: https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

Now the question is what is an admission controller?

In a nutshell, Kubernetes admission controllers are plugins that govern and enforce how the cluster is used. They can be thought of as a gatekeeper that intercept (authenticated) API requests and may change the request object or deny the request altogether. The admission control process has two phases: the mutating phase is executed first, followed by the validatingphase. Consequently, admission controllers can act as mutating or validating controllers or as a combination of both. Reference https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

Installing OPA Gatekeeper as CRD

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yamlnamespace/gatekeeper-system createdresourcequota/gatekeeper-critical-pods createdcustomresourcedefinition.apiextensions.k8s.io/configs.config.gatekeeper.sh createdcustomresourcedefinition.apiextensions.k8s.io/constraintpodstatuses.status.gatekeeper.sh createdcustomresourcedefinition.apiextensions.k8s.io/constrainttemplatepodstatuses.status.gatekeeper.sh createdcustomresourcedefinition.apiextensions.k8s.io/constrainttemplates.templates.gatekeeper.sh createdcustomresourcedefinition.apiextensions.k8s.io/providers.externaldata.gatekeeper.sh createdserviceaccount/gatekeeper-admin createdWarning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+podsecuritypolicy.policy/gatekeeper-admin createdrole.rbac.authorization.k8s.io/gatekeeper-manager-role createdclusterrole.rbac.authorization.k8s.io/gatekeeper-manager-role createdrolebinding.rbac.authorization.k8s.io/gatekeeper-manager-rolebinding createdclusterrolebinding.rbac.authorization.k8s.io/gatekeeper-manager-rolebinding createdsecret/gatekeeper-webhook-server-cert createdservice/gatekeeper-webhook-service createddeployment.apps/gatekeeper-audit createddeployment.apps/gatekeeper-controller-manager createdWarning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudgetpoddisruptionbudget.policy/gatekeeper-controller-manager createdvalidatingwebhookconfiguration.admissionregistration.k8s.io/gatekeeper-validating-webhook-configuration created
  • This will create a namespace gatekeeper-system
kubectl get nsNAME                 STATUS   AGEdefault              Active   2d10hgatekeeper-system    Active   22skube-node-lease      Active   2d10hkube-public          Active   2d10hkube-system          Active   2d10hlocal-path-storage   Active   2d10h
  • Verify the gatekeeper CRD is installed successfully
kubectl get crdNAME                                                 CREATED ATconfigs.config.gatekeeper.sh                         2021-11-01T04:11:42Zconstraintpodstatuses.status.gatekeeper.sh           2021-11-01T04:11:42Zconstrainttemplatepodstatuses.status.gatekeeper.sh   2021-11-01T04:11:42Zconstrainttemplates.templates.gatekeeper.sh          2021-11-01T04:11:42Zproviders.externaldata.gatekeeper.sh                 2021-11-01T04:11:42Z
  • Verify all the resources in the gatekeeper-system namespace is up and running
kubectl get all -n gatekeeper-systemNAME                                                 READY   STATUS    RESTARTS   AGEpod/gatekeeper-audit-76bc76bfb6-wsclz                1/1     Running   0          26spod/gatekeeper-controller-manager-5c6c56cccf-7jsjf   1/1     Running   0          26spod/gatekeeper-controller-manager-5c6c56cccf-8558v   1/1     Running   0          26spod/gatekeeper-controller-manager-5c6c56cccf-8wlxb   1/1     Running   0          26sNAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGEservice/gatekeeper-webhook-service   ClusterIP   10.96.43.235   <none>        443/TCP   26sNAME                                            READY   UP-TO-DATE   AVAILABLE   AGEdeployment.apps/gatekeeper-audit                1/1     1            1           26sdeployment.apps/gatekeeper-controller-manager   3/3     3            3           26sNAME                                                       DESIRED   CURRENT   READY   AGEreplicaset.apps/gatekeeper-audit-76bc76bfb6                1         1         1       26sreplicaset.apps/gatekeeper-controller-manager-5c6c56cccf   3         3         3       26s
  • Now we have all the resources, Pod, and Services running in the gatekeeper-system namespace. The next step is to define policies. In this example, I will create a policy using Rego that denies all pod creation. The first step is to define ConstraintTemplate and Constraint CRD by using Rego.

In the code above once, the count reaches greater than 0(1> 0), policy violation will occur, and the message(msg: msg) will be displayed to the user.

  • Create the template
kubectl create -f template.yamlconstrainttemplate.templates.gatekeeper.sh/k8salwaysdeny created
  • Verify its create successfully
kubectl get constrainttemplatesNAME            AGEk8salwaysdeny   22s> kubectl get crd |grep -i denyk8salwaysdeny.constraints.gatekeeper.sh              2021-11-01T04:22:28Z
  • Next step is to create the constraint which uses this template

Note: Make sure the kind matches the kind defined in the constraint template.

Next, create the constraint and verify it

> kubectl create -f constraints.yamlk8salwaysdeny.constraints.gatekeeper.sh/deny-pod-creation created> kubectl get k8salwaysdenyNAME                AGEdeny-pod-creation   36s
  • Verify if the policy is working as expected
> kubectl run nginx --image=nginxError from server ([deny-pod-creation] Pod creation denied): admission webhook "validation.gatekeeper.sh" denied the request: [deny-pod-creation] Pod creation denied
  • Check the violation under the status section:
> kubectl describe k8sAlwaysDeny deny-pod-creationName:         deny-pod-creationNamespace:Labels:       <none>Annotations:  <none>API Version:  constraints.gatekeeper.sh/v1beta1Kind:         K8sAlwaysDenyMetadata:Creation Timestamp:  2021-11-01T04:26:52ZGeneration:          1Managed Fields:API Version:  constraints.gatekeeper.sh/v1beta1Fields Type:  FieldsV1fieldsV1:f:status:Manager:      gatekeeperOperation:    UpdateTime:         2021-11-01T04:26:52ZAPI Version:  constraints.gatekeeper.sh/v1beta1Fields Type:  FieldsV1fieldsV1:f:spec:.:f:match:.:f:kinds:f:parameters:.:f:message:Manager:         kubectl-createOperation:       UpdateTime:            2021-11-01T04:26:52ZResource Version:  1908UID:               c19da666-371b-4d84-be71-8db885e0f643Spec:Match:Kinds:API Groups:Kinds:PodParameters:Message:  Pod creation deniedStatus:Audit Timestamp:  2021-11-01T04:27:51ZBy Pod:Constraint UID:       c19da666-371b-4d84-be71-8db885e0f643Enforced:             trueId:                   gatekeeper-audit-76bc76bfb6-wsclzObserved Generation:  1Operations:auditstatusConstraint UID:       c19da666-371b-4d84-be71-8db885e0f643Enforced:             trueId:                   gatekeeper-controller-manager-5c6c56cccf-7jsjfObserved Generation:  1Operations:webhookConstraint UID:       c19da666-371b-4d84-be71-8db885e0f643Enforced:             trueId:                   gatekeeper-controller-manager-5c6c56cccf-8558vObserved Generation:  1Operations:webhookConstraint UID:       c19da666-371b-4d84-be71-8db885e0f643Enforced:             trueId:                   gatekeeper-controller-manager-5c6c56cccf-8wlxbObserved Generation:  1Operations:webhookTotal Violations:  17Violations:Enforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                gatekeeper-audit-76bc76bfb6-wsclzNamespace:           gatekeeper-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                gatekeeper-controller-manager-5c6c56cccf-7jsjfNamespace:           gatekeeper-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                gatekeeper-controller-manager-5c6c56cccf-8558vNamespace:           gatekeeper-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                gatekeeper-controller-manager-5c6c56cccf-8wlxbNamespace:           gatekeeper-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                coredns-558bd4d5db-9pwptNamespace:           kube-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                coredns-558bd4d5db-bk8w2Namespace:           kube-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                etcd-my-kind-cluster-control-planeNamespace:           kube-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                kindnet-hbhv6Namespace:           kube-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                kindnet-tnh79Namespace:           kube-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                kindnet-xg4l4Namespace:           kube-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                kube-apiserver-my-kind-cluster-control-planeNamespace:           kube-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                kube-controller-manager-my-kind-cluster-control-planeNamespace:           kube-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                kube-proxy-46bnbNamespace:           kube-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                kube-proxy-6x9zgNamespace:           kube-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                kube-proxy-7mtwwNamespace:           kube-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                kube-scheduler-my-kind-cluster-control-planeNamespace:           kube-systemEnforcement Action:  denyKind:                PodMessage:             Pod creation deniedName:                local-path-provisioner-547f784dff-g6qwdNamespace:           local-path-storageEvents:                  <none>

Advantages

  • You can express complex policy
  • Support Kubernetes and other systems like CI/CD, microservices, etc
  • Community support

Disadvantage

  • You need to learn a new language(rego) to write policies
  • The policy is complex and hard to read

Kyverno

One of the major challenges with OPA is you need to learn a new language rego to enforce the policy. In the case of Kyverno, you don’t need to learn a new language, but it’s specific to Kubernetes.

Installing Kyverno

kubectl create -f https://raw.githubusercontent.com/kyverno/kyverno/release-1.5/definitions/release/install.yamlnamespace/kyverno createdcustomresourcedefinition.apiextensions.k8s.io/clusterpolicies.kyverno.io createdcustomresourcedefinition.apiextensions.k8s.io/clusterpolicyreports.wgpolicyk8s.io createdcustomresourcedefinition.apiextensions.k8s.io/clusterreportchangerequests.kyverno.io createdcustomresourcedefinition.apiextensions.k8s.io/generaterequests.kyverno.io createdcustomresourcedefinition.apiextensions.k8s.io/policies.kyverno.io createdcustomresourcedefinition.apiextensions.k8s.io/policyreports.wgpolicyk8s.io createdcustomresourcedefinition.apiextensions.k8s.io/reportchangerequests.kyverno.io createdserviceaccount/kyverno-service-account createdclusterrole.rbac.authorization.k8s.io/kyverno:admin-policies createdclusterrole.rbac.authorization.k8s.io/kyverno:admin-policyreport createdclusterrole.rbac.authorization.k8s.io/kyverno:admin-reportchangerequest createdclusterrole.rbac.authorization.k8s.io/kyverno:customresources createdclusterrole.rbac.authorization.k8s.io/kyverno:generatecontroller createdclusterrole.rbac.authorization.k8s.io/kyverno:leaderelection createdclusterrole.rbac.authorization.k8s.io/kyverno:policycontroller createdclusterrole.rbac.authorization.k8s.io/kyverno:userinfo createdclusterrole.rbac.authorization.k8s.io/kyverno:webhook createdclusterrolebinding.rbac.authorization.k8s.io/kyverno:customresources createdclusterrolebinding.rbac.authorization.k8s.io/kyverno:generatecontroller createdclusterrolebinding.rbac.authorization.k8s.io/kyverno:leaderelection createdclusterrolebinding.rbac.authorization.k8s.io/kyverno:policycontroller createdclusterrolebinding.rbac.authorization.k8s.io/kyverno:userinfo createdclusterrolebinding.rbac.authorization.k8s.io/kyverno:webhook createdconfigmap/kyverno createdconfigmap/kyverno-metrics createdservice/kyverno-svc createdservice/kyverno-svc-metrics createddeployment.apps/kyverno createdWarning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudgetpoddisruptionbudget.policy/kyverno created
  • Create the pod which enforce the policy that all the pod must have labels app.kubernetes.io/name
  • Create the policy
kubectl create -f policy.yamlclusterpolicy.kyverno.io/require-labels created
  • Verify if it’s working as expected
> kubectl run nginx --image nginxError from server: admission webhook "validate.kyverno.svc-fail" denied the request:resource Pod/default/nginx was blocked due to the following policiesrequire-labels:check-for-labels: 'validation error: label ''app.kubernetes.io/name'' is required.Rule check-for-labels failed at path /metadata/labels/app.kubernetes.io/name/'

Advantage

  • Kubernetes style
  • Easy to read

Disadvantages

  • As it’s not using any programming language, writing complex policies may be difficult
  • Relatively new

Conclusion

Whatever policy engine you choose entirely depends upon your requirement and your use case, but it’s essential to implement policies to secure your Kubernetes cluster.

AWS Community Builder, Ex-Redhat, Author, Blogger, YouTuber, RHCA, RHCDS, RHCE, Docker Certified,4XAWS, CCNA, MCP, Certified Jenkins, Terraform Certified, 1XGCP