0

I'd like to import a text file in Elasticsearch. The text file contains 3 values per line. After spending several hours of struggling, I didn't get it done. Help is greatly appreciated.

Elasticsearch 5.4.0 with Logstash installed.

Sample data:

username email hash
username email hash
username email hash
username email hash
username email hash

also built a python script but its too slow:

import requests
import json
from elasticsearch import Elasticsearch
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

i = 1
with open("my2") as fileobject:
    for line in fileobject:
        username, email, hash = line.strip('\n').split(' ')
        body = {"username": username, "email": email, "password": hash}
        es.index(index='dbs', doc_type='db1', id=i, body=body)
        i += 1

edit: Thanks its work but i guess my filter is bad because i want it to look like this:

    {
  "_index": "logstash-2017.06.01",
  "_type": "db",
  "_id": "AVxinqK5XRvft8kN7Q6M",
  "_version": 1,
  "_score": null,
  "_source": {
    "username": "Marlb0ro",
    "email": "Marlb0ro@site.com",
    "hash": "123456",
}

and it put the data like this:

    {
  "_index": "logstash-2017.06.01",
  "_type": "logs",
  "_id": "AVxinqK5XRvft8kN7Q6M",
  "_version": 1,
  "_score": null,
  "_source": {
    "path": "C:/Users/user/Desktop/user/log.txt",
    "@timestamp": "2017-06-01T07:46:22.488Z",
    "@version": "1",
    "host": "DESKTOP-FNGSJ6C",
    "message": "username email password",
    "tags": [
      "_grokparsefailure"
    ]
  },
  "fields": {
    "@timestamp": [
      1496303182488
    ]
  },
  "sort": [
    1496303182488
  ]
}
Marlb0ro
  • 15
  • 1
  • 6
  • If you spent several hours, you probably have bootstraped some logstash configuration, which you could show. – Val Jun 01 '17 at 03:51
  • came up with this pattern: %{WORD:username} %{WORD:email} %{WORD:hash} but i dont understand how to set it up in a config file – Marlb0ro Jun 01 '17 at 06:25
  • you might want to have a look at this question: https://stackoverflow.com/questions/31884094/logstash-grok-filter-for-custom-logs/32132737#32132737 – Val Jun 01 '17 at 07:02
  • help me please i read all the logstash docs and i cant do it also builet a python script but its too slow – Marlb0ro Jun 01 '17 at 07:22

1 Answers1

2

Simply put this in a file called grok.conf:

input {
        file {
                path => "/path/to/your/file.log"
                start_position => beginning
                sincedb_path => "/dev/null"
        }
}
filter {
        grok {
                match => {"message" => "%{WORD:username} %{WORD:email} %{WORD:hash}" }
        }
}
output {
        elasticsearch {
                hosts => ["localhost:9200"]
        }
}

Then run Logstash with bin/logstash -f grok.conf and you should be ok.

Val
  • 207,596
  • 13
  • 358
  • 360
  • Thanks that work but i think my filter is bad can u look at my edit? i couldnt fit it inside the comment – Marlb0ro Jun 01 '17 at 07:57