Parsing log using Bash Script

Check the updated DevOps Course.

Course Registration link:

Create an Account

Course Link:

101 Days of DevOps

YouTube link:

Prashant Lakhera

This is one of the common tasks which we use to perform on daily basis where we search for a particular date log

Scenario: Search for all the logs of date Apr 12

Apr 12 13:22:07 docker NetworkManager[657]: <info>  [1523553727.1687] policy: set 'enp0s3' (enp0s3) as default for IPv6 routing and DNS
Apr 12 13:22:31 docker systemd: Started Session 2 of user root.
Apr 12 13:22:31 docker systemd-logind: New session 2 of user root.
Apr 12 13:22:31 docker systemd: Starting Session 2 of user root

• Location of log files in common Linux System(/var/log)
• We are interested in /var/log/messages
• messages log file contain general system messages
• syslog/rsyslog generally routed all general system messages to log file or the logs which are not configured to be routed to any log file

Now our first challenge is to get the date in the same format as we have in the /var/log/messages file

Apr 12

If you run the date command, the format is different and its not going to solve our purpose

# date
Thu Apr 12 13:46:29 EDT 2018

Run date with help option

date --help

We will see something like this

%b   locale's abbreviated month name (e.g., Jan)
%d   day of month (e.g., 01)

TADA that resolve our issue

# date +%b\ %d
Apr 12

Now to parse the log file we are going to use grep or technically egrep

'egrep' means 'grep -E'
-E, --extended-regexp     PATTERN is an extended regular expression (ERE)

First try to run egrep in combination with date command we build earlier

# egrep "$(date +"%b %d")" messages
Apr 12 13:21:47 docker chronyd[627]: Selected source 173.255.215.209
Apr 12 13:21:47 docker chronyd[627]: System clock wrong by 424033.939728 seconds, adjustment started
Apr 12 13:21:47 docker chronyd[627]: System clock was stepped by 424033.939728 seconds
Apr 12 13:21:47 docker systemd: Time has been changed
Apr 12 13:21:49 docker chronyd[627]: Source 146.71.77.36 replaced with 216.229.0.50

Let’s put together everything in the bash script

#!/bin/bash
logfile=/var/log/messages
mydatexpr=`date +%b\ %d`
for log in $logfile
do
egrep "$mydatexpr" $log
done

lakhera2018/bash_scripts

Make this script executable

chmod +x logparsing.sh

Run it

# ./logparsing.sh
Apr 12 13:21:47 docker chronyd[627]: Selected source 173.255.215.209
Apr 12 13:21:47 docker chronyd[627]: System clock wrong by 424033.939728 seconds, adjustment started
Apr 12 13:21:47 docker chronyd[627]: System clock was stepped by 424033.939728 seconds
Apr 12 13:21:47 docker systemd: Time has been changed

This script looks good if we are dealing with a single file but what would be the case when we need to search for a particular date in multiple files, with simple changes in the script we will make it more flexible

#!/bin/bash
logfile1=/var/log/messages
logfile2=/var/log/secure
mydatexpr=`date +%b\ %d`

for log in $logfile{1,2}
do
egrep "$mydatexpr" $log
done

lakhera2018/bash_scripts

Again make it executable and run it

# chmod +x logparsingmultiple.sh

Script looks good and doing its job but it’s difficult to read the output

Let’s make minor tweak to the script to make it more readable

#!/bin/bash
logfile1=/var/log/messages
logfile2=/var/log/secure
mydatexpr=`date +%b\ %d`
for log in $logfile{1,2}
do
echo $log BEGIN
egrep "$mydatexpr" $log
echo $log END
done

lakhera2018/bash_scripts

When you execute this, output will be more readable

/var/log/messages END
/var/log/secure BEGIN
Apr 12 13:22:31 docker sshd[1239]: Accepted password for root from 192.168.0.2 port 58423 ssh2
Apr 12 13:22:31 docker sshd[1239]: pam_unix(sshd:session): session opened for user root by (uid=0)
/var/log/secure END

Now our basic script works let’s try to harden this script, by adding the condition to see if file exist

#!/bin/bash
logfile1=/var/log/messages
logfile2=/var/log/notexist
mydatexpr=`date +%b\ %d`
for log in $logfile{1,2}
do
if [ -e $log ]
then
echo $log BEGIN
egrep "$mydatexpr" $log
echo $log END
else
echo "$log file doesn't exist"
fi
done

In this case output will look like this

/var/log/messages END
/var/log/notexist file doesn't exist