Six tools to Simplify Kubernetes Journey — Day 5 — Kubewatch — A watcher for Kubernetes

As per the official github link kubewatch is a Kubernetes watcher that publishes notifications to Slack/hipchat/mattermost/flock channels. It watches the cluster for resource changes and notifies them through webhooks.

Basically kubewatch looks for events like pod/deployment/confimap creation/deletion and sends notification to selected channels like

• slack
• hipchat
• mattermost
• flock
• webhook
• smtp

Installation

In this blog, we will use helm to install kubewatch and use slack as the notification medium. But before setting up helm and slack, you must need to have your running Kubernetes cluster. Please check the link on how to configure the Kubernetes cluster using kind.

Configuring Slack

• Go to your slack workspace, and you have the option to create a new channel, or you can use an existing channel. For this demo, I am going to use an existing channel slacktest(but creating a new channel is pretty easy, you need to click on Create a new channel).

• Once you have a slack channel, the next step is to get a slack token to integrate with Kubewatch. To create a slack token, go to https://api.slack.com/apps/new and create a new app. Give your App Name, for eg: kubewatch, and select your Development Slack Workspace for eg.100daysofdevops in this case.

• In the next step we need to define our token scope. In this case we are using chat:write as scope which gives permission to post messages in approved channels & conversations.

• Now scroll down and click on Oauth & Permissions and note Bot User OAuth Access token.

• One more step you need to perform is to invite the Bot to the channel

/invite @BOT_NAME

/invite @kubewatch

Using helm

Once you have helm installed in your cluster(if you want more information about helm and install it, please follow this link).

• Add the bitnami repo.

helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories

• To verify it

helm repo list                                          
NAME          URL                                                   
bitnami       https://charts.bitnami.com/bitnami

• Next, we will run a repo update to make sure all the charts are updated and in sync.

helm repo update
Update Complete. ⎈Happy Helming!⎈

• Search for kubewatch chart

helm search repo kubewatc
NAME             	CHART VERSION	APP VERSION	DESCRIPTION                                       
bitnami/kubewatch	3.2.17       	0.1.0      	Kubewatch is a Kubernetes watcher that currentl...h

• As we know all helm chart comes with a default set of values but in this case, we want to modify these values according to our requirement.

helm show values bitnami/kubewatch > ~/kubewatch.yaml

• Now open this file and modify few parameters. First, let start with slack and add the channel to send notification and add token we have created during the slack section.

slack:
  enabled: true
  # Slack channel to notify
  channel: "slacktest"
  # Slack bots token. Create using: https://my.slack.com/services/new/bot
  # and invite the bot to your channel using: /join @botname
  token: "xoxb-XXXXXXX"

• In the next step, we need to define what I want to monitor. So, in this case, I want to monitor only the events from the default namespace(namespaceToWatch: “default”) and the resources to watch(deployment, services, pod). If you want to monitor any other resources, changes the parameter to true(for eg. replicaset: false to replicaset: true).

# namespace to watch, leave it empty for watching all.
namespaceToWatch: "default"
# Resources to watch
resourcesToWatch:
  deployment: true
  replicationcontroller: false
  replicaset: false
  daemonset: false
  services: true
  pod: true
  job: false
  persistentvolume: false

• One important parameter you need to set is rbac to true. If you set it to a default value to false, then the service account this helm chart creates doesn’t have access to list Kubernetes resources such as pod, deployments, services, etc.

rbac:
  create: true

• With all the configuration in place, its time to deploy the helm chart with our customized values.

helm install my-kubewatch bitnami/kubewatch --values ~/kubewatch.yam

NAME: my-kubewatch
LAST DEPLOYED: Sat Oct 23 08:35:34 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: kubewatch
CHART VERSION: 3.2.17
APP VERSION: 0.1.0

** Please be patient while the chart is being deployed **

To verify that kubewatch has started, run:l

• If you now execute the command shown at the last command’s output, you will see the my-kubewatch pod created in the default namespace.

kubectl get deploy -w --namespace default my-kubewatch
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
my-kubewatch   0/1     1            0           11s
my-kubewatch   1/1     1            1           16s

Testing

• Try to create any pod in the default namespace

kubectl run nginx2 --image=nginx              
pod/nginx2 created

• If you go to the slack channel, you will see a notification like this

• You can also tail the logs of your pod to verify these notifications

kubectl logs my-kubewatch-5fd7dbf69d-8hjs
==> Config file exists...
time="2021-10-23T15:35:50Z" level=info msg="Starting kubewatch controller" pkg=kubewatch-pod
time="2021-10-23T15:35:50Z" level=info msg="Starting kubewatch controller" pkg=kubewatch-deployment
time="2021-10-23T15:35:50Z" level=info msg="Processing add to deployment: default/my-kubewatch" pkg=kubewatch-deployment
time="2021-10-23T15:35:50Z" level=info msg="Processing add to pod: default/my-kubewatch-5fd7dbf69d-8hjsk" pkg=kubewatch-pod
time="2021-10-23T15:35:50Z" level=info msg="Kubewatch controller synced and ready" pkg=kubewatch-deployment
time="2021-10-23T15:35:50Z" level=info msg="Kubewatch controller synced and ready" pkg=kubewatch-pod
time="2021-10-23T15:36:03Z" level=info msg="Processing add to pod: default/nginx2" pkg=kubewatch-pod
time="2021-10-23T15:36:03Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod
time="2021-10-23T15:36:03Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod
2021/10/23 15:36:03 Message successfully sent to channel C02JMBPQZP0 at 1635003363.000700
2021/10/23 15:36:04 Message successfully sent to channel C02JMBPQZP0 at 1635003363.000800
time="2021-10-23T15:36:16Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod
2021/10/23 15:36:16 Message successfully sent to channel C02JMBPQZP0 at 1635003376.000900k

Wrapping Up

Kubewatch is a powerful tool and sends a notification for any events happening in your Kubernetes cluster. But please configure it wisely, as too many notifications in the busy cluster might lead to notification fatigue, and you will start missing the important ones.