0

I am using Elastic Search with a TTL for each index, and I have 1 index per day.

Now the problem is: I get my logs to expire , say 1 day(which I specify in the default.json) after the indexing, but what I want is the logs should expire relative to the timestamp of the logs(which I parse and replace the timestamp with) and not the time when the logs were indexed.

How do I do this? Need expert help please.

Thanks

user2359303
  • 261
  • 2
  • 7
  • 15

1 Answers1

0

I think you can do it by enabling the timestamp field and configuring its path in the mapping to point to a field in your documents. This would be the mapping:

{
    "tweet" : {
        "_timestamp" : {
            "enabled" : true,
            "path" : "post_date"
        }
    }
}

Haven't tried it myself but theoretically it should work, give it a try and let me know how it went!

javanna
  • 59,145
  • 14
  • 144
  • 125
  • How do I specify the path? I am having the timestamp in a field in the logstash export.... Do i need something like path:"@fields:ts", where ts is a field containing the time. – user2359303 May 16 '13 at 11:47
  • When i simply use "path":"ts" or "path":"@fields:ts", i get the error: Failed to parse doc to extract timestamp – user2359303 May 16 '13 at 11:50
  • That seems the way to go. Look at your log file and see what the parsing error is exactly. – javanna May 16 '13 at 11:55
  • My timestamp is of the form 2009-11-15T14:12:12.699Z. Maybe this creates a problem....when I use "path":"@timestamp", it should work ryt? – user2359303 May 16 '13 at 12:03
  • could you give me your mail id? I think u r the only one on SO who can help me with ES :P – user2359303 May 16 '13 at 12:12
  • There are quite some other experts around, I think it's more a matter of asking clear questions and doing some work yourself too if you want to get great answers. – javanna May 16 '13 at 12:51
  • I have already tried with all the fields possible..but I get a parser exception. Even specifying the data format does not help – user2359303 May 16 '13 at 12:56