1

I have a pretty simple application which send tcp log to promtail. (So promtail doesn't gaver data from file)

The problem is that the last log sent to protail isn't stored in loki until another log comes.

Let's say that I logged "foo", I'll see nothing in grafana. Then if I log "bar" to promtail, I'll only see foo in grafana, then if I log another "foo", I'll see only "foo" and "bar"...

I have this architecture :

Dockerised C++ app which use spdlog lib to log through tcp; Dockerised promtail which only gaver from tcp; Dockerised loki and grafana both are vanilla versions

My log format is the following : <165>4 %Y-%m-%dT%H:%M:%S.%eZ MY_APPLICATION MY_PROJECT - 1 - %v, The %v is composed of various str that I build in one before loging.

Promtail doesn't show any errors. If there are some network nerds, when I ctrl+c (end) my app, the log bufferised in promtail is finaly stored in Loki and visible in grafana. I might think that my logs lack a kind of "end of log" sign to have this to happen...

Thanks for the help, any suggestion or even question will be appreciated ! (Sorry for my not so good english)

Here is some code to reproduce the problem :

C++ app

static std::shared_ptr<spdlog::logger> tcp_logger_ptr;

void init_logger() {
    spdlog::init_thread_pool(4096, 1);
    auto tcp_sink_config = spdlog::sinks::tcp_sink_config(172.17.0.1, 1514);
    tcp_logger_ptr = spdlog::create_async<spdlog::sinks::tcp_sink_mt>("TCP_logger", tcp_sink_config);
    tcp_logger_ptr->set_pattern(PROMTAIL_LOG_PATTERN);
}

Then to log

void tcp_log(spdlog::level::level_enum log_type, const char* log_message) {
    tcp_logger_ptr->log(log_type, log_message);
}

Promtail docker config

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:1514
      idle_timeout: 60s
      label_structured_data: yes
      labels:
        job: "syslog"
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'
markalex
  • 8,623
  • 2
  • 7
  • 32
lou lou
  • 11
  • 2

1 Answers1

0

Ok thanks everyone for the help ! (lol)

I fugured it out, and it is pretty disappointing to be honnest... The answer was the idle_timeout: 60s

It just says to promtail that if it has nothing to do for 60s, it flush what it got stored (my log in this case). So you see it comming, with an idle timeout of 0.1 the fonctionning is pretty good.

lou lou
  • 11
  • 2