Six tools to Simplify Kubernetes Journey — Day 5 — Kubewatch — A watcher for Kubernetes

As per the official github link kubewatch is a Kubernetes watcher that publishes notifications to Slack/hipchat/mattermost/flock channels. It watches the cluster for resource changes and notifies them through webhooks.

Basically kubewatch looks for events like pod/deployment/confimap creation/deletion and sends notification to selected channels like

  • slack
  • hipchat
  • mattermost
  • flock
  • webhook
  • smtp

Installation

In this blog, we will use helm to install kubewatch and use slack as the notification medium. But before setting up helm and slack, you must need to have your running Kubernetes cluster. Please check the link on how to configure the Kubernetes cluster using kind.

Configuring Slack

  • Go to your slack workspace, and you have the option to create a new channel, or you can use an existing channel. For this demo, I am going to use an existing channel slacktest(but creating a new channel is pretty easy, you need to click on Create a new channel).
  • Once you have a slack channel, the next step is to get a slack token to integrate with Kubewatch. To create a slack token, go to https://api.slack.com/apps/new and create a new app. Give your App Name, for eg: kubewatch, and select your Development Slack Workspace for eg.100daysofdevops in this case.
  • In the next step we need to define our token scope. In this case we are using chat:write as scope which gives permission to post messages in approved channels & conversations.
  • Now scroll down and click on Oauth & Permissions and note Bot User OAuth Access token.
  • One more step you need to perform is to invite the Bot to the channel

/invite @BOT_NAME

/invite @kubewatch

Using helm

Once you have helm installed in your cluster(if you want more information about helm and install it, please follow this link).

  • Add the bitnami repo.
helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories
  • To verify it
helm repo list                                          
NAME URL
bitnami https://charts.bitnami.com/bitnami
  • Next, we will run a repo update to make sure all the charts are updated and in sync.
helm repo update
Update Complete. ⎈Happy Helming!⎈
  • Search for kubewatch chart
helm search repo kubewatcNAME             	CHART VERSION	APP VERSION	DESCRIPTION                                       bitnami/kubewatch	3.2.17       	0.1.0      	Kubewatch is a Kubernetes watcher that currentl...h
  • As we know all helm chart comes with a default set of values but in this case, we want to modify these values according to our requirement.
helm show values bitnami/kubewatch > ~/kubewatch.yaml
  • Now open this file and modify few parameters. First, let start with slack and add the channel to send notification and add token we have created during the slack section.
slack:
enabled: true
# Slack channel to notify
channel: "slacktest"
# Slack bots token. Create using: https://my.slack.com/services/new/bot
# and invite the bot to your channel using: /join @botname
token: "xoxb-XXXXXXX"
  • In the next step, we need to define what I want to monitor. So, in this case, I want to monitor only the events from the default namespace(namespaceToWatch: “default”) and the resources to watch(deployment, services, pod). If you want to monitor any other resources, changes the parameter to true(for eg. replicaset: false to replicaset: true).
# namespace to watch, leave it empty for watching all.
namespaceToWatch: "default"
# Resources to watch
resourcesToWatch:
deployment: true
replicationcontroller: false
replicaset: false
daemonset: false
services: true
pod: true
job: false
persistentvolume: false
  • One important parameter you need to set is rbac to true. If you set it to a default value to false, then the service account this helm chart creates doesn’t have access to list Kubernetes resources such as pod, deployments, services, etc.
rbac:
create: true
  • With all the configuration in place, its time to deploy the helm chart with our customized values.
helm install my-kubewatch bitnami/kubewatch --values ~/kubewatch.yam
NAME: my-kubewatchLAST DEPLOYED: Sat Oct 23 08:35:34 2021NAMESPACE: defaultSTATUS: deployedREVISION: 1TEST SUITE: NoneNOTES:CHART NAME: kubewatchCHART VERSION: 3.2.17APP VERSION: 0.1.0
** Please be patient while the chart is being deployed **
To verify that kubewatch has started, run:l
  • If you now execute the command shown at the last command’s output, you will see the my-kubewatch pod created in the default namespace.
kubectl get deploy -w --namespace default my-kubewatch
NAME READY UP-TO-DATE AVAILABLE AGE
my-kubewatch 0/1 1 0 11s
my-kubewatch 1/1 1 1 16s

Testing

  • Try to create any pod in the default namespace
kubectl run nginx2 --image=nginx              
pod/nginx2 created
  • If you go to the slack channel, you will see a notification like this
  • You can also tail the logs of your pod to verify these notifications
kubectl logs my-kubewatch-5fd7dbf69d-8hjs==> Config file exists...time="2021-10-23T15:35:50Z" level=info msg="Starting kubewatch controller" pkg=kubewatch-podtime="2021-10-23T15:35:50Z" level=info msg="Starting kubewatch controller" pkg=kubewatch-deploymenttime="2021-10-23T15:35:50Z" level=info msg="Processing add to deployment: default/my-kubewatch" pkg=kubewatch-deploymenttime="2021-10-23T15:35:50Z" level=info msg="Processing add to pod: default/my-kubewatch-5fd7dbf69d-8hjsk" pkg=kubewatch-podtime="2021-10-23T15:35:50Z" level=info msg="Kubewatch controller synced and ready" pkg=kubewatch-deploymenttime="2021-10-23T15:35:50Z" level=info msg="Kubewatch controller synced and ready" pkg=kubewatch-podtime="2021-10-23T15:36:03Z" level=info msg="Processing add to pod: default/nginx2" pkg=kubewatch-podtime="2021-10-23T15:36:03Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-podtime="2021-10-23T15:36:03Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod2021/10/23 15:36:03 Message successfully sent to channel C02JMBPQZP0 at 1635003363.0007002021/10/23 15:36:04 Message successfully sent to channel C02JMBPQZP0 at 1635003363.000800time="2021-10-23T15:36:16Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod2021/10/23 15:36:16 Message successfully sent to channel C02JMBPQZP0 at 1635003376.000900k

Wrapping Up

Kubewatch is a powerful tool and sends a notification for any events happening in your Kubernetes cluster. But please configure it wisely, as too many notifications in the busy cluster might lead to notification fatigue, and you will start missing the important ones.

AWS Community Builder, Ex-Redhat, Author, Blogger, YouTuber, RHCA, RHCDS, RHCE, Docker Certified,4XAWS, CCNA, MCP, Certified Jenkins, Terraform Certified, 1XGCP