Configuring and Working with Cloudwatch Logs

Below details how to setup Cloudwatch log agent on the EC2 instances

# yum update -y

#yum install -y awslogs ( this command may not work for RH, CenOS, use below steps)

# curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 47998  100 47998    0     0   419k      0 --:--:-- --:--:-- --:--:--  422k

# ls
awslogs-agent-setup.py

# python ./awslogs-agent-setup.py  --region us-east-1
Launching interactive setup of CloudWatch Logs agent ...

Step 1 of 5: Installing pip ...DONE
Step 2 of 5: Downloading the latest CloudWatch Logs agent bits ... DONE
Step 3 of 5: Configuring AWS CLI ...
AWS Access Key ID [None]: <aws-access-key>
AWS Secret Access Key [None]:<aws-seceret-access-key>
Default region name [us-east-1]:
Default output format [None]:

Step 4 of 5: Configuring the CloudWatch Logs Agent ...
Path of log file to upload [/var/log/messages]:
Destination Log Group name [/var/log/messages]:

Choose Log Stream name:
  1. Use EC2 instance id.
  2. Use hostname.
  3. Custom.
Enter choice [1]:

Choose Log Event timestamp format:
  1. %b %d %H:%M:%S    (Dec 31 23:59:59)
  2. %d/%b/%Y:%H:%M:%S (10/Oct/2000:13:55:36)
  3. %Y-%m-%d %H:%M:%S (2008-09-08 11:52:54)
  4. Custom
Enter choice [1]: 3

Choose initial position of upload:
  1. From start of file.
  2. From end of file.
Enter choice [1]: 1
More log files to configure? [Y]: n

Step 5 of 5: Setting up agent as a daemon ...DONE


------------------------------------------------------
- Configuration file successfully saved at: /var/awslogs/etc/awslogs.conf
- You can begin accessing new log events after a few moments at https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logs:
- You can use 'sudo service awslogs start|stop|status|restart' to control the daemon.
- To see diagnostic information for the CloudWatch Logs Agent, see /var/log/awslogs.log
- You can rerun interactive setup using 'sudo python ./awslogs-agent-setup.py --region us-east-1 --only-generate-config'
------------------------------------------------------
#

Check your cloudwatch dashboard to see if any logs are coming in your stream.

Now, how to collect the logs from cloudwatch. On your local system or desktop, install awslogs from github

https://github.com/jorgebastida/awslogs

# pip install awslogs
/usr/local/aws/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
You are using pip version 7.0.3, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting awslogs
/usr/local/aws/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading awslogs-0.5.0.tar.gz
Collecting boto3>=1.2.1 (from awslogs)
  Downloading boto3-1.3.1-py2.py3-none-any.whl (113kB)
    100% |████████████████████████████████| 114kB 2.8MB/s
Collecting termcolor>=1.1.0 (from awslogs)
  Downloading termcolor-1.1.0.tar.gz
Requirement already satisfied (use --upgrade to upgrade): python-dateutil>=2.4.0 in /usr/local/aws/lib/python2.6/site-packages (from awslogs)
Requirement already satisfied (use --upgrade to upgrade): argparse>=1.1.0 in /usr/local/aws/lib/python2.6/site-packages (from awslogs)
Requirement already satisfied (use --upgrade to upgrade): futures<4.0.0,>=2.2.0 in /usr/local/aws/lib/python2.6/site-packages (from boto3>=1.2.1->awslogs)
Requirement already satisfied (use --upgrade to upgrade): jmespath<1.0.0,>=0.7.1 in /usr/local/aws/lib/python2.6/site-packages (from boto3>=1.2.1->awslogs)
Requirement already satisfied (use --upgrade to upgrade): botocore<1.5.0,>=1.4.1 in /usr/local/aws/lib/python2.6/site-packages (from boto3>=1.2.1->awslogs)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5 in /usr/local/aws/lib/python2.6/site-packages (from python-dateutil>=2.4.0->awslogs)
Requirement already satisfied (use --upgrade to upgrade): ordereddict==1.1 in /usr/local/aws/lib/python2.6/site-packages (from botocore<1.5.0,>=1.4.1->boto3>=1.2.1->awslogs)
Requirement already satisfied (use --upgrade to upgrade): simplejson==3.3.0 in /usr/local/aws/lib/python2.6/site-packages (from botocore<1.5.0,>=1.4.1->boto3>=1.2.1->awslogs)
Requirement already satisfied (use --upgrade to upgrade): docutils>=0.10 in /usr/local/aws/lib/python2.6/site-packages (from botocore<1.5.0,>=1.4.1->boto3>=1.2.1->awslogs)
Building wheels for collected packages: awslogs, termcolor
  Running setup.py bdist_wheel for awslogs
  Stored in directory: /root/.cache/pip/wheels/21/ca/1d/09b0bc28e47edd432789b3670bb7cd0d116ccfa54a83ab42a5
  Running setup.py bdist_wheel for termcolor
  Stored in directory: /root/.cache/pip/wheels/de/f7/bf/1bcac7bf30549e6a4957382e2ecab04c88e513117207067b03
Successfully built awslogs termcolor
Installing collected packages: boto3, termcolor, awslogs
Successfully installed awslogs-0.5.0 boto3-1.3.1 termcolor-1.1.0

Once the tool is installed. run the below commands to get the output from cloudwatch.

awslogs get /var/log/messages --start='1h ago' | grep xenbus
awslogs get /var/log/messages --start='45 minutes'



awslogs get /var/log/messages --start='2 days ago' --no-color (this is to avoid the coloring and wild characters when you start writing to other file)


aws logs get-log-events --log-group-name /var/log/messages --log-stream-name i-097fb8cbe37dfb658 --output text 
Have Fun.