I'm using the Amazon Kinesis Agent to tail my nginx web access logs with a view to pushing the access log to Kinesis - ultimately to be used within Kinesis Analytics.
The agent is running and is tailing my access log, then pushing records up to Kinesis. I can see records arriving in Kinesis so there connectivity is good.
The issue I'm having related to the formatter options - specifically regarding "LOGTOJSON".
Here is the contents of /etc/aws-kinesis/agent.json:
{
"cloudwatch.emitMetrics": false,
"kinesis.endpoint": "kinesis.eu-west-1.amazonaws.com",
"flows": [
{
"filePattern": "/var/log/nginx/proxy-access*",
"kinesisStream": "removed-xxx",
"partitionKeyOption": "RANDOM",
"dataProcessingOptions": {
"optionName": "LOGTOJSON",
"logFormat": "COMBINEDAPACHELOG"
}
]
}
Here is the section of the nginx config that I am using to output logs in the apache combined log format:
log_format combined_apache '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';
access_log logs/proxy-access.log combined_apache;
Here is a few lines of output from the nginx access log:
172.17.0.1 - - [09/Sep/2016:11:04:27 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
172.17.0.1 - - [09/Sep/2016:11:04:27 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
172.17.0.1 - - [09/Sep/2016:11:04:27 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
172.17.0.1 - - [09/Sep/2016:11:04:27 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
And here is the output from the Kinesis agent. You can see that records are being "parsed" but not "transformed".
2016-09-09 11:03:20.574+0000 18b332249e3f (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 0 records parsed (0 bytes), and 0 records sent successfully to destinations. Uptime: 60041ms
2016-09-09 11:03:20.574+0000 18b332249e3f (FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2016-09-09 11:03:50.572+0000 18b332249e3f (FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2016-09-09 11:03:50.572+0000 18b332249e3f (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 0 records parsed (0 bytes), and 0 records sent successfully to destinations. Uptime: 90038ms
2016-09-09 11:04:20.572+0000 18b332249e3f (FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*]: Tailer Progress: Tailer has parsed 398 records (36218 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2016-09-09 11:04:20.572+0000 18b332249e3f (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 398 records parsed (36218 bytes), and 0 records sent successfully to destinations. Uptime: 120038ms
2016-09-09 11:04:50.572+0000 18b332249e3f (FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*]: Tailer Progress: Tailer has parsed 533 records (48503 bytes), transformed 0 records, skipped 0 records, and has successfully sent 500 records to destination.
2016-09-09 11:04:50.572+0000 18b332249e3f (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 533 records parsed (48503 bytes), and 500 records sent successfully to destinations. Uptime: 150038ms
2016-09-09 11:05:20.571+0000 18b332249e3f (FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*]: Tailer Progress: Tailer has parsed 533 records (48503 bytes), transformed 0 records, skipped 0 records, and has successfully sent 500 records to destination.
Kinesis Analytics detects the incoming data as a CSV stream rather than a JSON stream. When I look at the raw stream content in the AWS console, it appears like this:
172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:59 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:59 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
So it looks as though the agent is sending each record without any kind of transformation and delimiting each line with a comma.
I can't see where I'm going wrong here. The nginx access log looks like its being written in the correct format, but the Kinesis agent doesn't transform it to JSON - for unknown reasons.
Any pointers would be appreciated.
