100 Days of AWS — Day 8— Pushing System logs and Custom Metrics using CloudWatch agent

Solution

What are the Cloudwatch logs?

Scenario1: Perform the step manually

Step1: Setup IAM Role(CloudWatch agent installed on EC2 instance, require permission to put data and that why we need to setup this role)

$  wget https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip$ unzip AmazonCloudWatchAgent.zip$ sudo ./install.shcreate group cwagent, result: 0create user cwagent, result: 0
$  sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard================================================================= Welcome to the Amazon CloudWatch Agent Configuration Manager ==                                                              == CloudWatch Agent allows you to collect metrics and logs from == your host and send them to CloudWatch. Additional CloudWatch == charges may apply.                                           =================================================================On which OS are you planning to use the agent?1. linux2. windows3. darwindefault choice: [1]:1Trying to fetch the default region based on ec2 metadata...Are you using EC2 or On-Premises hosts?1. EC22. On-Premisesdefault choice: [1]:1Which user are you planning to run the agent?1. root2. cwagent3. othersdefault choice: [1]:1Do you want to turn on StatsD daemon?1. yes2. nodefault choice: [1]:1Which port do you want StatsD daemon to listen to?default choice: [8125]What is the collect interval for StatsD daemon?1. 10s2. 30s3. 60sdefault choice: [1]:What is the aggregation interval for metrics collected by StatsD daemon?1. Do not aggregate2. 10s3. 30s4. 60sdefault choice: [4]:Do you want to monitor metrics from CollectD? WARNING: CollectD must be installed or the Agent will fail to start1. yes2. nodefault choice: [1]:Do you want to monitor any host metrics? e.g. CPU, memory, etc.1. yes2. nodefault choice: [1]:Do you want to monitor cpu metrics per core?1. yes2. nodefault choice: [1]:Do you want to add ec2 dimensions (ImageId, InstanceId, InstanceType, AutoScalingGroupName) into all of your metrics if the info is available?1. yes2. nodefault choice: [1]:Do you want to aggregate ec2 dimensions (InstanceId)?1. yes2. nodefault choice: [1]:Would you like to collect your metrics at high resolution (sub-minute resolution)? This enables sub-minute resolution for all metrics, but you can customize for specific metrics in the output json file.1. 1s2. 10s3. 30s4. 60sdefault choice: [4]:Which default metrics config do you want?1. Basic2. Standard3. Advanced <--Make sure you choose advance then only you will see(Memory,Swap,Disk...)4. Nonedefault choice: [1]:3Current config as follows:{"agent": {"metrics_collection_interval": 60,"run_as_user": "root"},"metrics": {"aggregation_dimensions": [["InstanceId"]],"append_dimensions": {"AutoScalingGroupName": "${aws:AutoScalingGroupName}","ImageId": "${aws:ImageId}","InstanceId": "${aws:InstanceId}","InstanceType": "${aws:InstanceType}"},"metrics_collected": {"collectd": {"metrics_aggregation_interval": 60},"cpu": {"measurement": ["cpu_usage_idle","cpu_usage_iowait","cpu_usage_user","cpu_usage_system"],"metrics_collection_interval": 60,"resources": ["*"],"totalcpu": false},"disk": {"measurement": ["used_percent","inodes_free"],"metrics_collection_interval": 60,"resources": ["*"]},"diskio": {"measurement": ["io_time","write_bytes","read_bytes","writes","reads"],"metrics_collection_interval": 60,"resources": ["*"]},"mem": {"measurement": ["mem_used_percent"],"metrics_collection_interval": 60},"netstat": {"measurement": ["tcp_established","tcp_time_wait"],"metrics_collection_interval": 60},"statsd": {"metrics_aggregation_interval": 60,"metrics_collection_interval": 10,"service_address": ":8125"},"swap": {"measurement": ["swap_used_percent"],"metrics_collection_interval": 60}}}}Are you satisfied with the above config? Note: it can be manually customized after the wizard completes to add additional items.1. yes2. nodefault choice: [1]:1Do you have any existing CloudWatch Log Agent (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html) configuration file to import for migration?1. yes2. nodefault choice: [2]:Do you want to monitor any log files?1. yes2. nodefault choice: [1]:Log file path:/var/log/secureLog group name:default choice: [secure]Log stream name:default choice: [{instance_id}]Log Group Retention in days1. -12. 13. 34. 55. 76. 147. 308. 609. 9010. 12011. 15012. 18013. 36514. 40015. 54516. 73117. 182718. 3653default choice: [1]:7Do you want to specify any additional log files to monitor?1. yes2. nodefault choice: [1]:1Log file path:/var/log/messagesLog group name:default choice: [messages]Log stream name:default choice: [{instance_id}]Log Group Retention in days1. -12. 13. 34. 55. 76. 147. 308. 609. 9010. 12011. 15012. 18013. 36514. 40015. 54516. 73117. 182718. 3653default choice: [1]:7Do you want to specify any additional log files to monitor?1. yes2. nodefault choice: [1]:2Saved config file to /opt/aws/amazon-cloudwatch-agent/bin/config.json successfully.Current config as follows:{"agent": {"metrics_collection_interval": 60,"run_as_user": "root"},"logs": {"logs_collected": {"files": {"collect_list": [{"file_path": "/var/log/secure","log_group_name": "secure","log_stream_name": "{instance_id}","retention_in_days": 30},{"file_path": "/var/log/messages","log_group_name": "messages","log_stream_name": "{instance_id}","retention_in_days": 30}]}}},"metrics": {"aggregation_dimensions": [["InstanceId"]],"append_dimensions": {"AutoScalingGroupName": "${aws:AutoScalingGroupName}","ImageId": "${aws:ImageId}","InstanceId": "${aws:InstanceId}","InstanceType": "${aws:InstanceType}"},"metrics_collected": {"collectd": {"metrics_aggregation_interval": 60},"cpu": {"measurement": ["cpu_usage_idle","cpu_usage_iowait","cpu_usage_user","cpu_usage_system"],"metrics_collection_interval": 60,"resources": ["*"],"totalcpu": false},"disk": {"measurement": ["used_percent","inodes_free"],"metrics_collection_interval": 60,"resources": ["*"]},"diskio": {"measurement": ["io_time","write_bytes","read_bytes","writes","reads"],"metrics_collection_interval": 60,"resources": ["*"]},"mem": {"measurement": ["mem_used_percent"],"metrics_collection_interval": 60},"netstat": {"measurement": ["tcp_established","tcp_time_wait"],"metrics_collection_interval": 60},"statsd": {"metrics_aggregation_interval": 60,"metrics_collection_interval": 10,"service_address": ":8125"},"swap": {"measurement": ["swap_used_percent"],"metrics_collection_interval": 60}}}}Please check the above content of the config.The config file is also located at /opt/aws/amazon-cloudwatch-agent/bin/config.json.Edit it manually if needed.Do you want to store the config in the SSM parameter store?1. yes2. nodefault choice: [1]:1What parameter store name do you want to use to store your config? (Use 'AmazonCloudWatch-' prefix if you use our managed AWS policy)default choice: [AmazonCloudWatch-linux]Trying to fetch the default region based on ec2 metadata...Which region do you want to store the config in the parameter store?default choice: [us-east-1]Which AWS credential should be used to send json config to parameter store?1. ASIATGNGBM4JE3EICGM5(From SDK)2. Otherdefault choice: [1]:Successfully put config to parameter store AmazonCloudWatch-linux.Program exits now.
$ sudo amazon-linux-extras install collectd
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s****** processing amazon-cloudwatch-agent ******/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:/opt/aws/amazon-cloudwatch-agent/bin/config.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config defaultSuccessfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmpStart configuration validation.../opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default2022/04/12 02:57:26 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp ...Valid Json input schema.I! Detecting run_as_user...No csm configuration found.Configuration validation first phase succeeded/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.tomlConfiguration validation second phase succeededConfiguration validation succeededamazon-cloudwatch-agent has already been stoppedCreated symlink from /etc/systemd/system/multi-user.target.wants/amazon-cloudwatch-agent.service to /etc/systemd/system/amazon-cloudwatch-agent.service.Redirecting to /bin/systemctl restart amazon-cloudwatch-agent.service
$ sudo  /bin/systemctl restart amazon-cloudwatch-agent.service
# sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:<ssm parmeter name> -s# In the above case it was AmazonCloudWatch-linux, so our command will look like this# sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-linux -s

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Prashant Lakhera

Prashant Lakhera

4.9K Followers

AWS Community Builder, Ex-Redhat, Author, Blogger, YouTuber, RHCA, RHCDS, RHCE, Docker Certified,4XAWS, CCNA, MCP, Certified Jenkins, Terraform Certified, 1XGCP