0

Using this config for logstash's output. It's using /tmp/logstash-gcs as a local folder. Send to GCS when file become 1024 kbytes.

input {
  beats {
    port => 5044
  }
}

filter {}

output {
  google_cloud_storage {
    bucket => "gcs-bucket-name"
    json_key_file => "/secrets/service_account/credentials.json"
    temp_directory => "/tmp/logstash-gcs"
    log_file_prefix => "logstash_gcs"
    max_file_size_kbytes => 1024
    output_format => "json"
    date_pattern => "%Y-%m-%dT%H:00"
    flush_interval_secs => 2
    gzip => false
    gzip_content_encoding => false
    uploader_interval_secs => 60
    include_uuid => true
    include_hostname => true
  }
}

After restart machine, can it continuous do the job schedule without any data loss? Does it has a queue feature to manage as pub/sub?

iooi
  • 453
  • 2
  • 10
  • 23
  • What is your input? If you restart the logstash machine you can lose data while the service was out depending on your input. – leandrojmp Jan 05 '21 at 00:23
  • I added input. It's filebeat. – iooi Jan 05 '21 at 01:04
  • Filebeat tracks what was the last line it sent to logstash, if logstash fails it will keep retrying until logstash is back and logstash will starting output those events, so you should not expect any data loss. – leandrojmp Jan 05 '21 at 01:30
  • I deployed a logstash service on kubernetes. It's using a PVC to mount the `/tmp/logstash-gcs` path to a storage. After the pod restarted, the file can be found there, but didn't been upload to gcs yet: `sh-4.2$ ls -l /tmp/logstash-gcs/`, `-rw-r--r-- 1 logstash logstash 875 Jan 5 01:45 logstash_gcs_logstash-log-pipeline-0_2021-01-05T01:00.part000.d31cb8f9-2e88-9dff-7acd-1ca258ef7681.log`. After waited for 60 secends, it still didn't been uploaded. The size is below 1024kb, but is it normal? When can it been uploaded? – iooi Jan 05 '21 at 02:34
  • I do not use this output, but looking in the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-google_cloud_storage.html#_improvements_todo_list) you have this information in a improvements list: "There’s no recover method, so if logstash/plugin crashes, files may not be uploaded to GCS." Maybe this happens when restarting as well. – leandrojmp Jan 05 '21 at 02:42
  • I checked it. It' clear now. So maybe the best way to let those files can be uploaded to gcs, need to beat them by `filebeat` again. Thank you very much. – iooi Jan 05 '21 at 02:57
  • I understand you found the solution for your question. It would be great if you can post the solution as an answer so other users can benefit. – marian.vladoi Jan 05 '21 at 10:53

1 Answers1

0

About solution, there are 2 ways.

  • Use filebeat to beat those tmp files again.
  • Set grace period seconds to insure logstash has enough time to send last task when machine down.
iooi
  • 453
  • 2
  • 10
  • 23