0

Info:

Filebeat is installed on machine from where logs will be read and sent to elastic search server. From test machine, using elasticsearch-dsl, I am reading logs and writing it to file.

Problem:

Orig Log from machine :

[Timestamp][INFO] AAAAAA
[Timestamp][INFO] BBBBBB
[Timestamp][INFO] CCCCCC

After searching and writing logs to output file :

[Timestamp][INFO] CCCCCC
[Timestamp][INFO] AAAAAA
[Timestamp][INFO] BBBBBB

How to keep the sequence of log intact or as it is?

Code :

from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search, Q, Index
import time
#Make Connection
es =  Elasticsearch(["100.16.13.222:9200"])

#Create Index Object
ind = Index("filebeat-*",using=es)
#Clear Cache
ind.clear_cache()
#Create Search object for this index
sear = ind.search()

#Create query
sear =  sear.query("match",host="WIN-LK9FS7568K4").query("match",tags="old_log")
res = sear.execute(ignore_cache=True)
print int(res.hits.total)

with open("a.txt","w") as fh:
    for i in sear.scan():
        fh.write(i.message+"\n")
Dinesh Pundkar
  • 4,160
  • 1
  • 23
  • 37

1 Answers1

1

You need to sort your logs by timestamp. Change your search code to this:

sear =  sear.sort('timestamp')
            .query("match",host="WIN-LK9FS7568K4")
            .query("match",tags="old_log")

Of course you need to change timestamp to match your timestamp field.

Val
  • 207,596
  • 13
  • 358
  • 360
  • Thanks for help !!! One more question. How to search using "offset"? I have tried sear.sort("offset") but it is not sorting – Dinesh Pundkar Apr 14 '17 at 11:54
  • 1
    This is described [here](https://elasticsearch-dsl.readthedocs.io/en/latest/search_dsl.html#pagination), simply using the Python slicing API. – Val Apr 14 '17 at 12:00