my english is not good ,sorry ...
these days ,i always test td-agent on AWS EC2(Amazon Linux AMI release 2014.03). input log from nginx and output log to S3 is OK, this very good!
then i want praser log to json ,when i made the source format "nginx" ,it says
" 2015-03-31 00:12:24 +0800 [warn]: pattern not match: "95.218.58.120 - [2015-03-30T16:11:40+08:00] \"GET /provider.txt HTTP/1.1\" 200 726 \"-\" \"FreePP 2.9.5.159 (iPhone; iPhone OS 8.1.3; ar_SA) [65075070]\" "
i think mybe my nginx log is not default.then,i made regexp for this ,
format /^(?<remote>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: (?<path>[^\"]*)+\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"?$/
and my log is:
178.152.203.39 - [2015-03-31T00:18:10+08:00] "GET /provider.txt HTTP/1.1" 200 726 "-" "Dalvik/1.6.0 (Linux; U; Android 4.3; GT-I9300 Build/JSS15J)"
restart td-agent ,td-agent logs seems OK.but no output found ,and my cpu 100%,any one can help me ? many thanks !!!
top - 00:26:17 up 4 days, 1:10, 2 users, load average: 0.20, 0.35, 0.57
Tasks: 82 total, 3 running, 79 sleeping, 0 stopped, 0 zombie
Cpu(s): 48.3%us, 2.2%sy, 0.0%ni, 47.5%id, 0.0%wa, 0.3%hi, 1.7%si, 0.0%st
Mem: 7401596k total, 6612324k used, 789272k free, 151200k buffers
Swap: 0k total, 0k used, 0k free, 5911168k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4724 td-agent 20 0 222m 23m 2800 R 96.5 0.3 0:11.70 ruby
473 nginx 20 0 119m 12m 1668 S 3.7 0.2 5:58.61 nginx
2015-03-31 00:26:04 +0800 [info]: process finished code=0
2015-03-31 00:26:05 +0800 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2015-03-31 00:26:05 +0800 [info]: starting fluentd-0.10.61
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-mongo' version '0.7.6'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-s3' version '0.5.6'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-td' version '0.10.26'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.0'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-03-31 00:26:05 +0800 [info]: gem 'fluentd' version '0.10.61'
2015-03-31 00:26:05 +0800 [info]: using configuration file: <ROOT>
<source>
type tail
format /^(?<remote>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: (?<path>[^\"]*)+\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"?$/
time_format %d/%b/%YT%H:%M:%S%z
path /var/log/nginx/provider.access.log
log_level debug
pos_file /var/log/td-agent/access.log.pos
tag s3.nginx.access
</source>
<match s3.nginx.access>
type file
path /var/log/td-agent/nginxAccess
time_slice_format %Y%m%d
time_slice_wait 10m
format single_value
</match>
</ROOT>
2015-03-31 00:26:05 +0800 [info]: adding source type="tail"
2015-03-31 00:26:05 +0800 [info]: adding match pattern="s3.nginx.access" type="file"
2015-03-31 00:26:05 +0800 [info]: following tail of /var/log/nginx/provider.access.log
