How can i send the data in IIS logs to Amazon CloudWatch logs so that i can monitor the performance of my website.
One of the things that I am trying to monitor is the average request size of my web request. I know that IIS logs have the data about the size of web request(BytesRecv, ByteSent) and I can have CloudWatch logs read my IIS log files but What i cannot figure out is a way to tell CloudWatch logs that BytesRecv, ByteSent should be treated as 2 datapoints.
I don't think the CloudWatch Logs service has that capability. When it ingests logs like IIS, you can create simple filters to match something, like 404 errors, and then you can create datapoints on the number of those errors in a given time period. However, I haven't found a way to extract data from logs directly in CloudWatch.
I believe the solution to this problem would be to use Amazon Kinesis to get the log files out of CloudWatch and then process them with EMR to get those data points and then put that information into S3. A lot easier said than done, I know. I think the toughest part of this would be writing your EMR logic and then putting that data into some kind of consolidated format to write to S3. I'd recommend asking for help around that area.
Another option would be to have Amazon Kinesis drop the log files in S3, then trigger an Amazon Lambda action when those logs files are uploaded. The Lambda function could then parse those log files, extract the information you need, put it into some kind of json, xml, etc and write that to S3. The hard part here is writing the lambda function. This link describes how to use lambda to parse CloudTrail logs written to S3, so you could probably follow a lot of that logic to do this.
http://docs.aws.amazon.com/lambda/latest/dg/wt-cloudtrail-events-adminuser.html
If you can get this info in IIS logs you can share them to cloudwatch logs
you can send logs via EC2Config Service or SSM Agent more details are documented in this Post.
Then you can use existing filters to your log group or create custom filter to extract the fields that you want from the logs -> so it's custom log metric based on log filters. e.g.
[serverip, method, uri, query, port, dash, clientip, useragent, status, zero1, zero2, millis]
or some specific filters.
So you can now either use filters as mentioned above or Log Insight queries for creating dashboards.
