100 Days of DevOps — Day 83-Introduction to Splunk
To view the updated DevOps course(101DaysofDevOps)
Course Registration link: https://www.101daysofdevops.com/register/
Course Link: https://www.101daysofdevops.com/courses/101-days-of-devops/
YouTube link: https://www.youtube.com/user/laprashant/videos
Welcome to Day 83 of 100 Days of DevOps, Focus for today is Introduction to Splunk
As per wiki “Splunk (the product) captures, indexes and correlates (near)real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards, and visualizations”
Built-In
- Front End: Python(CherryPy)
- Backend: C/C++
Downloading and Installing Splunk
https://www.splunk.com/en_us/download.html
wget -O splunk-6.5.2–67571ef4b87d-linux-2.6-x86_64.rpm ‘https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=6.5.2&product=splunk&filename=splunk-6.5.2-67571ef4b87d-linux-2.6-x86_64.rpm&wget=true'# rpm -ivh splunk-6.5.2–67571ef4b87d-linux-2.6-x86_64.rpmwarning: splunk-6.5.2–67571ef4b87d-linux-2.6-x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 653fb112: NOKEYPreparing… ################################# [100%]Updating / installing…1:splunk-6.5.2–67571ef4b87d ################################# [100%]complete
Starting Splunk
# pwd/opt/splunk/bin# ./splunk start — accept-licenseSplunk> All batbelt. No tights.Checking prerequisites…Checking http port [8000]: openChecking mgmt port [8089]: openChecking appserver port [127.0.0.1:8065]: openChecking kvstore port [8191]: openChecking configuration… Done.Creating: /opt/splunk/var/run/splunk/appserver/i18nCreating: /opt/splunk/var/run/splunk/appserver/modules/static/cssCreating: /opt/splunk/var/run/splunk/uploadCreating: /opt/splunk/var/spool/splunkCreating: /opt/splunk/var/spool/dirmoncacheCreating: /opt/splunk/var/lib/splunk/authDbCreating: /opt/splunk/var/lib/splunk/hashDbChecking critical directories… DoneChecking indexes…Validated: _audit _internal _introspection _telemetry _thefishbucket history main summaryDoneNew certs have been generated in ‘/opt/splunk/etc/auth’.Checking filesystem compatibility… DoneChecking conf files for problems…DoneChecking default conf files for edits…Validating installed files against hashes from ‘/opt/splunk/splunk-6.5.2–67571ef4b87d-linux-2.6-x86_64-manifest’All installed files intact.DoneAll preliminary checks passed.Starting splunk server daemon (splunkd)…Generating a 1024 bit RSA private key………++++++…………………………++++++writing new private key to ‘privKeySecure.pem’— — -Signature oksubject=/CN=ip-XXXX/O=SplunkUserGetting CA Private Keywriting RSA keyDone[ OK ]Waiting for web server at http://127.0.0.1:8000 to be available… DoneIf you get stuck, we’re here to help.Look for answers here: http://docs.splunk.comThe Splunk web interface is at http://XXXX:8000
Enabling it at boot time
# ./splunk enable boot-startInit script installed at /etc/init.d/splunk.Init script is configured to run at boot.# To disable it#./splunk disable boot-start# To run Splunk as a specific user
#./splunk enable boot-start -user splunkuser
Once it’s up and running you will see screen like this
Default username: admin
password: changeme
Adding Linux Logs to Splunk(local host)
Click on Add Data → Monitor → Files and Directories
Set Source Type → Input Settings → Review → Done(For the first time go with default settings)(We have added /var/log/messages from the local box)
Finally, you will see data like this
Setting up Splunk to receive logs from a remote machine
Forwarders: These are the agents installed on the client side and used to send data to Splunk Server Indexer(Indexer: Where Splunk stores data in the form of indexes)
Splunk Provide two kind of forwarders
- Universal Forwarder
- Heavy Forwarder
Splunk Architecture
- Search Head: Used for Visualization and connects to indexer to fetch the data. The user usually logs in to search head to search and visualize data.
There are two other components
- Deployment Servers: It manages all Splunk configuration/servers(indexer,forwarder..) from one machine. Let say I want to modify one file and push it to all servers that task I can do with the help of deployment servers.
- Licensing Server: Manage and monitor license usage
Under Setting Links → Forwarding and receiving
Under Receive data → Add New
Restart Splunk Server
# ./splunk stopStopping splunkd…Shutting down. Please wait, as this may take a few minutes... [ OK ]Stopping splunk helpers…[ OK ]Done.# ./splunk startSplunk> All batbelt. No tights.Checking prerequisites…Checking http port [8000]: openChecking mgmt port [8089]: openChecking appserver port [127.0.0.1:8065]: openChecking kvstore port [8191]: openChecking configuration… Done.Checking critical directories… DoneChecking indexes…Validated: _audit _internal _introspection _telemetry _thefishbucket history main summaryDoneChecking filesystem compatibility… DoneChecking conf files for problems…DoneChecking default conf files for edits…Validating installed files against hashes from ‘/opt/splunk/splunk-6.5.2–67571ef4b87d-linux-2.6-x86_64-manifest’All installed files intact.DoneAll preliminary checks passed.Starting splunk server daemon (splunkd)…Done[ OK ]Waiting for web server at http://127.0.0.1:8000 to be available.. DoneIf you get stuck, we’re here to help.Look for answers here: http://docs.splunk.comThe Splunk web interface is at http://XXXX:8000#To check splunk status# ./splunk statussplunkd is running (PID: 5711).splunk helpers are running (PIDs: 5719 5727 5832 5889).
We can also restart Splunk from WebUI
Settings → Server controls → Restart Splunk
Important files in Splunk
# Log File Location/opt/splunk/var/log# Config file location/opt/splunk/etc
On the client side
# wget -O splunkforwarder-6.5.2–67571ef4b87d-linux-2.6-x86_64.rpm ‘https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=6.5.2&product=universalforwarder&filename=splunkforwarder-6.5.2-67571ef4b87d-linux-2.6-x86_64.rpm&wget=true'# rpm -ivh splunkforwarder-6.5.2–67571ef4b87d-linux-2.6-x86_64.rpmwarning: splunkforwarder-6.5.2–67571ef4b87d-linux-2.6-x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 653fb112: NOKEYPreparing… ########################################### [100%]1:splunkforwarder ########################################### [100%]
After Installing RPM, start the splunk forwarder
# pwd/opt/splunkforwarder/bin# ./splunk start — accept-licenseThis appears to be your first time running this version of Splunk.Splunk> All batbelt. No tights.Checking prerequisites…Checking mgmt port [8089]: openCreating: /opt/splunkforwarder/var/lib/splunkCreating: /opt/splunkforwarder/var/run/splunkCreating: /opt/splunkforwarder/var/run/splunk/appserver/i18nCreating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/cssCreating: /opt/splunkforwarder/var/run/splunk/uploadCreating: /opt/splunkforwarder/var/spool/splunkCreating: /opt/splunkforwarder/var/spool/dirmoncacheCreating: /opt/splunkforwarder/var/lib/splunk/authDbCreating: /opt/splunkforwarder/var/lib/splunk/hashDbNew certs have been generated in ‘/opt/splunkforwarder/etc/auth’.Checking conf files for problems…DoneChecking default conf files for edits…Validating installed files against hashes from ‘/opt/splunkforwarder/splunkforwarder-6.5.2–67571ef4b87d-linux-2.6-x86_64-manifest’All installed files intact.DoneAll preliminary checks passed.Starting splunk server daemon (splunkd)…Done[ OK ]
Now to enable Splunk forwarder at boot time
# ./splunk enable boot-startInit script installed at /etc/init.d/splunk.Init script is configured to run at boot.
It’s always a good idea to change the default password(changeme)
# ./splunk edit user admin -password test1234 -role admin -auth admin:changemeUser admin edited.
Now to add the forwarder
# ./splunk add forward-server 172.31.8.8:9997 -auth admin:Redhat12Added forwarding to: 172.31.8.8:9997
Now you can define which file you want to monitor(eg:/var/log/secure)
# ./splunk add monitor /var/log/secureAdded monitor of ‘/var/log/secure’.
Looking forward from you guys to join this journey and spend a minimum an hour every day for the next 100 days on DevOps work and post your progress using any of the below medium.
- Twitter: @100daysofdevops OR @lakhera2015
- Facebook: https://www.facebook.com/groups/795382630808645/
- Medium: https://medium.com/@devopslearning
- Slack: https://devops-myworld.slack.com/messages/CF41EFG49/
- GitHub Link:https://github.com/100daysofdevops
Reference
