Logs are not sent to elasticsearch using fluentd. The log is written in one line

Question

The Java module writes a single-line log:

{"timestamp":"2020-09-29 10:46:18.761","level":"INFO","message":"status: OK","logger":"thrift.handler.CheckTokenThriftHandler","system":"-","service":"AUTH BE","conf_item":"-"}{"timestamp":"2020-09-29 10:49:11.338","level":"INFO","message":"status: OK","logger":"thrift.handler.CheckTokenThriftHandler","system":"-","service":"AUTH BE","conf_item":"-"}...

Each new event is pasted to the end of the line. In the fluentd settings I use the standard parser:

[PARSER]
    Name         java-system
    Format       json
    Time_Key     time
    Time_Format  %Y-%m-%d %H:%M:%S.%L
    Time_Keep    On

If I split the events in the log into lines, the agent sends everything to elastic without any problems. If everything is in one line, it doesn't send anything. It doesn't write anything informative in the log.

Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.0 tag=gw_system
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.0:es.0
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.1 tag=calc-system
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.1:es.3
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.2 tag=push-system
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.2:es.2
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.3 tag=gw_integration
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.3:es.1
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [ info] [sp] stream processor started
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.0] file=/opt/java_services/gateway/log/system.log read=35484 lines=0
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.1] file=/opt/java_services/calculator/log/system.log read=336548 lines=0
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.2] file=/opt/java_services/push/log/system_temp.log read=12302 lines=0
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.3] file=/opt/java_services/gateway/log/integration.log promote to TAIL_EVENT
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.0] file=/opt/java_services/gateway/log/system.log promote to TAIL_EVENT
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.1] file=/opt/java_services/calculator/log/system.log promote to TAIL_EVENT
Sep 29 15:11:41  td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.2] file=/opt/java_services/push/log/system_temp.log promote to TAIL_EVENT

Wrote a separate parser

[PARSER]
    Name    java_push_system
    Format  regex
    Regex   (?<date>(?<={"timestamp":")(\d+\-\d+\-\d+\s+\d+:\d+:\d+.\d+))",(?<message>[^}]*)
    Time_Key    time
    Time_Format %Y-%m-%d %H:%M:%S.%L
    Time_Keep    On

On the site regex101.com my string is parsed, but the agent still does not send anything until it is split into separate lines.

/etc/td-agent-bit/conf]# cat push-system*
[FILTER]
    Name   record_modifier
    Match  push-system
    Record hostname ${HOSTNAME}
    Record environment DEV
    Record path /opt/java_services/push/log/system.log
[INPUT]
    Name   tail
    Tag    push-system
    Parser java-system
    Path   /opt/java_services/push/log/system.log
    DB     /etc/td-agent-bit/system-push.db
    Buffer_Max_Size 32MB
    Buffer_Chunk_Size 8MB
[OUTPUT]
    Name        es
    Match       push-system
    Host        10.*
    Port        9200
    HTTP_User   *
    HTTP_Passwd *
    Index       push

Help to understand please!

In your first case, the JSON is ill-formed e.g. `{...}{...}...`. Objects are not separated by commas and should be enclosed in brackets `[]` to be a well-formed JSON. You may verify this using an online JSON validator e.g. https://jsonlint.com/. — Azeem, Sep 30 '20 at 11:23
Are you sure that you're using `fluentd` and not `fluent-bit`? The configuration and logs show that it's `fluent-bit` but it is tagged as `fluentd`. You might want to update that. — Azeem, Sep 30 '20 at 11:28
@Azeem yes, this is fluent-bit. I dont't know, how change tag. — Aleksandr Bryanskiy, Sep 30 '20 at 11:37
Please [edit](https://stackoverflow.com/posts/64131832/edit) your question, remove `fluentd` and add `fluent-bit`. — Azeem, Sep 30 '20 at 11:38
Awesome! Can you please add the error logs you get in the `regex` case? And, you need to add the elasticsearch output plugin's configuration as well. — Azeem, Sep 30 '20 at 11:45
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.0] file=/opt/java_services/gateway/log/system.log read=35484 lines=0 Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.1] file=/opt/java_services/calculator/log/system.log read=336548 lines=0 Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.2] file=/opt/java_services/push/log/system_temp.log read=12302 lines=0 — Aleksandr Bryanskiy, Sep 30 '20 at 12:10
`[FILTER] Name record_modifier Match push-system Record hostname ${HOSTNAME} Record environment DEV Record path /opt/java_services/push/log/system.log [INPUT] Name tail Tag push-system Parser java-system Path /opt/java_services/push/log/system.log DB /etc/td-agent-bit/system-push.db Buffer_Max_Size 32MB Buffer_Chunk_Size 8MB [OUTPUT] Name es Match push-system Host 10.XX.XX.XXX Port 9200 HTTP_User *** HTTP_Passwd *** Index push` — Aleksandr Bryanskiy, Sep 30 '20 at 12:14
Please add this information in your question. Do you see any error logs on console? And, did you try routing the events to stdout? — Azeem, Sep 30 '20 at 12:17
No errors are visible. As if the parser doesn't detect the string. What standard output are we talking about? I restart the service like this. All output is attached in the message. systemctl restart td-agent-bit; tail -f /var/log/messages| grep -i td-agent-bit — Aleksandr Bryanskiy, Sep 30 '20 at 12:29
It seems like you're trying to use `regex` as the bulk logs parser, right? With regex, there would be multiple matches, please look at this https://rubular.com/r/9JqzkRmS79Mdtu. I'm not sure how this would be handled. Maybe, you'd achieve this by writing a custom plugin if one doesn't exist already. — Azeem, Sep 30 '20 at 12:30
Standard Output: https://docs.fluentbit.io/manual/pipeline/outputs/standard-output — Azeem, Sep 30 '20 at 12:32

Logs are not sent to elasticsearch using fluentd. The log is written in one line

0 Answers0