Day 14–101 Days of DevOps — A python script to check if the website is up

Welcome to Day 14 of 101 Days of DevOps. The topic for today is a Python script to check if the website is up.

To view the complete course, please check the below url.

101 Days of DevOps

For more info, register via the below link

Create an Account

YouTube Channel link

Prashant Lakhera

This is one of the common tasks we encountered as a part of our daily job, where we need to verify if our website is up. With a few lines of code and the Python module's help, we can verify it.

https://www.youtube.com/watch?v=hJBk0H5jPfg

In this tutorial, we are going to use the following three modules

• Requests
• BeautifulSoup4
• smtplib

Requests is an elegant and simple HTTP library for Python built for human beings. In simple terms, it’s a module that allows us to download files and webpages from the web easily. We are first going to define our url using the variable producturl and then check the website status. If the website status code is not 200, then we are going to receive an email. Sending the email part we are going to tackle in the latter part of the code.

NOTE: Requests is a 3rd party module, so you need to install it; you can do it via pip3 install requests.

import requests
producturl="http://100daysofdevops.com/contact-us/"
res = requests.get(producturl, timeout=5)
if res.status_code != 200:
    <To Be Done>

So far, so good, but the issue I have faced on my website is returning status code 200, but some users still complain that the webpage doesn’t load. To fix that issue, we are going to use another module called beautiful soup4. Similar to the request, it’s a 3rd party module, so you first need to install it(pip3 install beautifulsoup4)

BeautifulSoup4: Beautiful Soup is a library that makes it easy to scrape information from web pages. It sits atop an HTML or XML parser, providing Pythonic idioms for iterating, searching, and modifying the parse tree. So the way you see the web page in your browser is its html page; html basically tells your browser how to make the webpage look(its an oversimplified explanation; it contains many other things like css and javascript, to name a few).

BeautifulSoup is a third-party module, so we first need to install it, and we can do it with the help of pip.

pip3 install beautifulsoup4

• Once installed, we can use it in our code, and let’s call a beautiful soup module, which will return the object.

mport bs4
soup = bs4.BeautifulSoup(res.text,'html.parser’)

In the next step, we are going to use the select method and pass css selector. To view the html for any webpage, basically right-click on the webpage and click on View page source. HTML is basically text with angle(<>) bracket thing called HTML element. You also have a developer tool to open that tool in chrome press F12. You can hover over to the text and click on inspect element to view its source. That will basically bring the HTML element, which is useful when we need to pull specific information from our website.

Below I am trying to copy css selector for the content in my webpage WELCOME.

elems = soup.select('#post-67 > div > div > header > h2')

Now to get the first element, we will pass [0], and to get the string out of it, let call text. Then we will use try/except block to search for particular text(in this case, WELCOME), and if that is not present, we will receive an email. Sending the email part we are going to tackle in the next part of the code.

try:
    if elems[0].text != "Welcome":
        <To be done>
except Exception as e:
    <To be done>

smtplib: smtplib creates a Simple Mail Transfer Protocol client session object that will help us send emails. In this case, we are going to use gmail to send a mail.

To authenticate to a google account, you first need to follow some steps.

1. Turn off “Less secure app access.”

Less secure apps & your Google Account

2. Create a Sign in with App Passwords

Sign in with App Passwords

Would you please follow these two articles and let me know if you are stuck on any issues in the comment section?

Once that is done, you need to follow the series of steps

• Import smtplib module

import smtplib

• Create an SMTP session

s = smtplib.SMTP('smtp.gmail.com', 587)

• For security purposes, start TLS

s.starttls()

• Authenticate to google

s.login(email_add, password)

NOTE: Please make sure here provide your app password create using https://support.google.com/accounts/answer/185833?hl=en

• The message that needs to be sent

subject = "!!!!!! 100 days of devops site is down"
    body = "Check the status of 100 days of devops"
    message = f'Subject: {subject}\n\n {body}

• Finally, sending an email

# s.sendmail("sender_email", "reciever_email", message)
s.sendmail(email_add, email_add, message)

• Terminating the session

s.quit()

• I will convert the above code for sending email into a function

def send_email():
    s = smtplib.SMTP('smtp.gmail.com', 587)
    s.starttls()
    s.login(email_add, password)
    subject = "!!!!!! 100 days of devops site is down"
    body = "Check the status of 100 days of devops"
    message = f'Subject: {subject}\n\n {body}'
    s.sendmail(email_add, email_add, message)
    s.quit()

• Now use it in the above code, where we have mentioned to be <To be done>. Our code so far looks like

import requests
import bs4
import smtplib


def send_email():
    s = smtplib.SMTP('smtp.gmail.com', 587)
    s.starttls()
    s.login(email_add, password)
    subject = "!!!!!! 100 days of devops site is down"
    body = "Check the status of 100 days of devops"
    message = f'Subject: {subject}\n\n {body}'
    s.sendmail(email_add, email_add, message)
    s.quit()


producturl="http://100daysofdevops.com/contact-us/"

res = requests.get(producturl, timeout=5)
if res.status_code != 200:
    send_email()

soup = bs4.BeautifulSoup(res.text,'html.parser')

elems = soup.select('#post-67 > div > div > header > h2')
try:
    if elems[0].text != "Welcome":
        send_email()
except Exception as e:
    send_email()

• We need to add few more changes. The email address and password we are hardcoding in our code is bad security practice. Rather than hardcoding in our code, we will export it as an environment variable and use it on our code. To do that

export EMAIL_ADD ="abc@gmail.com"
export EMAIL_PASS="password"

• Now to use it in our code, we are going to import the os module and use the os.environment.get

import os

email_add = os.environ.get('EMAIL_ADD')
password = os.environ.get('EMAIL_PASS')

• The final code will look like this

import requests
import bs4
import smtplib
import os

email_add = os.environ.get('EMAIL_ADD')
password = os.environ.get('EMAIL_PASS')

def send_email():
    s = smtplib.SMTP('smtp.gmail.com', 587)
    s.starttls()
    s.login(email_add, password)
    subject = "!!!!!! 100 days of devops site is down"
    body = "Check the status of 100 days of devops"
    message = f'Subject: {subject}\n\n {body}'
    s.sendmail(email_add, email_add, message)
    s.quit()


producturl="http://100daysofdevops.com/contact-us/"

res = requests.get(producturl, timeout=5)
if res.status_code != 200:
    send_email()

soup = bs4.BeautifulSoup(res.text,'html.parser')

elems = soup.select('#post-67 > div > div > header > h2')
try:
    if elems[0].text != "Welcome":
        send_email()
except Exception as e:
    send_email()

GitHub link: https://github.com/100daysofdevops/100daysofdevops/blob/master/python_script_to_check_website_status/script_to_check_website_status.py

I am looking forward to you guys joining the amazing journey.

• Twitter: 100daysofdevops
• Facebook: https://www.facebook.com/groups/795382630808645/
• Medium: https://medium.com/@devopslearning
• GitHub: https://github.com/100daysofdevops/100daysofdevops
• Slack: https://join.slack.com/t/100daysofdevops/shared_invite/zt-au03logz-YfDUp_FJF4rAUeDEbgWmsg
• Reddit: r/101DaysofDevops
• Meetup: https://www.meetup.com/100daysofdevops/