9

I'm using Logstash 1.4.1 together with ES1.01 and would like to replace already indexed documents based on a calculated checksum. I'm currently using the "fingerprint" filter in Logstash which creates a "fingerprint" field based on a specified algorithm. Now - what I want to accomplish is that ES replaces an already existing document based on an identical fingerprint value.

Say, for example that I have a document with a fingerprint-field value of "2c9a6802e10fbcff36177e0b88993f90868fa6fa". Now - if a document with an identical fingerprint value is about to be indexed, I want it to replace the old document already present in the index.

I've tried to add the following to the "elasticsearch-template.json" template file which I assume is used by the Logstash ES-output plugin:

...
  "mappings" : {
    "_default_" : {
       "_id" : {"index": "not_analyzed", "store" : false, "path" : "fingerprint" },
       "_all" : {"enabled" : true},
       "dynamic_templates" : [ {
...

but it doesn't work. What am I doing wrong here?

Cheers

Kodo
  • 541
  • 1
  • 6
  • 17

2 Answers2

16

I would use the document_id parameter in your logstash elasticsearch output section:

document_id

Value type is string
Default value is nil

The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id

I believe the entry should be something like this:

document_id => "%{fingerprint}"

It uses logstash's sprintf format to replace a string with the contents of a field:

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#sprintf

A J
  • 2,508
  • 21
  • 26
John Petrone
  • 26,943
  • 6
  • 63
  • 68
3

Assuming the fingerprint is getting set as the _id, you may be hitting an issue with logstash's daily index management and not using the timestamp from your data.

Ensure that you have your timestamp set from the input data, so you are guaranteed the document goes to the correct daily index:

http://logstash.net/docs/1.4.2/filters/date

If my guess is correct, you should see that your duplicate documents have different @timestamp and are in different daily indexes.

ppearcy
  • 2,732
  • 19
  • 21
  • Hi! Hmmmm, I don't think that is the issue. The timestamp is set correctly but somehow the "_id" field is not set with the value present in the "fingerprint" field. I thought by amending the logstash/es template file I would be able to set the "path" attribute of the "_id" field to "fingerprint" but so far I have been unable to make this work... Thanks for your suggestion though:) – Kodo Jul 31 '14 at 07:06