100 Days of DevOps — Day 41-Real-Time Apache Log Analysis using Amazon Kinesis and Amazon Elasticsearch Service

What is Amazon Kinesis

What is AWS ElasticSearch Service

Step1: Setup ElasticSearch Cluster

Go to https://us-west-2.console.aws.amazon.com/es/ --> Create a new domain
* Choose Development and testing or based on your requirement
* Give your Elasticsearch domain name(I need to change it to hundreddaysofdevops as numeral 100 is not accepted by AWS) 
* Choose all the default options
* This is my test cluster and that why I am choosing these wide open options, these are definitely not recommended for PRD environment
* Network configuration(Public access for your environment choose particular VPC)
* Access policy: Your policy must be restricted

Step2: Setup firehose delivery Pipeline, this will continuously insert logs to ElasticSearch Cluster

Go to https://us-west-2.console.aws.amazon.com/firehose --> Create Delivery Stream
* Give your Delivery stream some name and keep all the options as default
* Source record transformation: Enabled and Choose the lambda function
* Select Destination: Choose Amazon ElasticSearch Service
* Domain: Should be auto-populated with ElasticSearch Name
* Index: Give your index some name(eg: mytestindex)
* Type(eg: apache)
* Backup mode: Create a new S3 bucket
NOTE: Backup Mode is to prevent any data loss,firehose store the data in S3 bucket

Step3: Send data to Firehose Delivery Stream

# sudo yum install –y https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-latest.amzn1.noarch.rpm# cat /etc/aws-kinesis/agent.json
service aws-kinesis-agent start

Step4: Visualize the data using Kibana



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Prashant Lakhera

Prashant Lakhera


AWS Community Builder, Ex-Redhat, Author, Blogger, YouTuber, RHCA, RHCDS, RHCE, Docker Certified,4XAWS, CCNA, MCP, Certified Jenkins, Terraform Certified, 1XGCP