fluent and webhdfs filename with 197001011

Question

I run td-agent on ubuntu 14.04 with the follow configuration:


    <source>
     type tail
     format json
     path /path/tomcat/logs/file-input.log
     tag bhc.hdfs
     pos_file /var/td-agent/file.pos
     </source>

    <match bhc.hdfs>
     type webhdfs
     port 50070
     host my.host.name
     path /hdfs/path/file.${hostname}.%Y%m%d.log
     username user
     flush_interval 10s
     output_include_time false
     output_include_tag false
     output_data_type json
    </match>

Log source files in directory /path/tomcat/logs/file-input.log contain only a structured json data. Ntp daemon is installed and running but when td-agent creates file in hdfs date on filename is 19700101. What's wrong?

score 1 · Accepted Answer · answered Mar 12 '15 at 01:49

1

Fluentd records has time, and webhdfs plugin creates files with that records' timestamp, not current time.

tail plugin uses field named as time for time of record in default. If your log data have any other time information field, you can specify it with time_key and time_format.

See also: http://docs.fluentd.org/articles/in_tail

answered Mar 12 '15 at 01:49

Satoshi Tagomori

126
2

Thanks @Satoshi, but why it's necessary to configure ntp daemon? – critical Mar 12 '15 at 07:47
@critical NTP daemon is just for getting right time from server's clock. – Satoshi Tagomori Mar 17 '15 at 12:04

fluent and webhdfs filename with 197001011

1 Answers1