Logstash filename as ElasticSearch index

Question

I am using a folder as input:

input {
  file {
    path => "C:/ProjectName/Uploads/*"
    start_position => "beginning"
    sincedb_path => "dev/null"
  }
}

and as output:

output {
  elasticsearch {
    hosts => "localhost"
    index => "manual_index_name" # want filename here
    document_type => "_doc"
  }     
}

I want the index in elasticsearch to be the name of the file being indexed. I've tried variations of this answer with no success as I am not clear on what it is doing: https://stackoverflow.com/a/40156466/6483906

score 3 · Accepted Answer · edited May 22 '18 at 20:01

3

You'll need to use a grok filter to find the last portion of the filename:

filter {
  grok {
     match => ["path", "Uploads/%{GREEDYDATA:index_name}" ]
  }
}

and then just use the portion in your index name index => "%{index_name}"

edited May 22 '18 at 20:01

Harsha Laxman

481
1
6
21

answered May 22 '18 at 18:10

Alcanzar

16,985
6
42
59

1

I follow the same pattern, but the index I got is just %{index_name} as a plain string. Could you please give a hint on how to solve it? – Annie H. Jun 07 '20 at 16:31
Same thing happen for me – Shino Lex Mar 21 '22 at 14:41

Logstash filename as ElasticSearch index

1 Answers1

Linked