0

I wonder if you can configure logstash in the following way:

Background Info:

  • Every day I get a xml file pushed to my server, which should be parsed.

  • To indicate a complete file transfer afterwards I get an empty .ctl (custom file) transfered to the same folder.

  • The files both have the following name schema 'feedback_{year}{yearday}_UTC{hoursminutesseconds}_51.{extention}' (e.g. feedback_16002_UTC235953_51.xml). So they have the same file name but one is with .xml and the other is a .ctl file.

Question:

Is there a way to configure logstash to wait parsing the xml file until the according .ctl file is present?

EDIT: Is there maybe a way to archiev that with filebeat?

EDIT2: It would also be enough to be able to configure logstash in a way that it will wait x minutes before starting to process a new file, if that is easier.

Thanks for any help in advance

d.a.d.a
  • 1,296
  • 1
  • 12
  • 28
  • Hi @d.a.d.a how did you manage to solve the issue? I have a similar use case in which I have to wait for a file being fully written in a specific directory before ingesting it – bruce_k Jun 14 '23 at 14:38
  • Oh wow it's been ages since this. I am sorry but I actually don't recall how we solved this. Probably like suggested by @alfredocambera suggested. Maybe also something like this https://discuss.elastic.co/t/wait-for-file-input-plugin-90-seconds/217627 – d.a.d.a Jun 15 '23 at 18:05

1 Answers1

0

Your problem is that you don't want to start the parser before the file transfer hasn't been completed. So, why don't push the data to a file (file-complete.xml) when you find your flag file (empty.ctl)?

Here is the possible logic for a script and runs using crontab:

  • if empty.ctl exists:
    • Clear file-complete.xml
    • Add the content of file.xml to file-complete.xml.
    • Remove empty.ctl

This way, you'd need to parse the data from file-complete.xml. I think is simpler to debug and configure.

Hope it helps,

alfredocambera
  • 3,155
  • 34
  • 29