4

I got a brand new empty installation of Elastic Search 6.1.1 running on port 9200 of my CentOS7 host. I have this "es.json" file with data sample ready to be inserted in the ES.

I also have a log file in which each line is a json chunk. I could use both to populate my ES database.

How do I insert this and other data to ES?

I didnt find good explanations on the documentation and on the internet. for some reason the docs does not make it very clear

Victor Ferreira
  • 6,151
  • 13
  • 64
  • 120
  • I am pretty sure that you have to create an index for each of these data sets you have since indexes in ES 6 can only have a single mapping type now. Did you create an index? – Dr G. Dec 22 '17 at 20:00
  • Take a look at [Index](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html) and [Bulk](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html) APIs. Also consider whether to define mappings or let ES generate [dynamic mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapping.html) for you. You could always use the [Reindex](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html) API if you need to change mappings in the future. – dcd018 Dec 22 '17 at 20:01
  • This helps: https://stackoverflow.com/a/65213529/3357884 – Mahan Dec 09 '20 at 09:02

2 Answers2

8

Use the --data-binary flag in curl to bulk import from a JSON file.

curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/{index}/{type}/_bulk?pretty' --data-binary @es.json

Data can be posted to one of the endpoints - /_bulk, /{index}/_bulk or {index}/{type}/_bulk. When {index} or {index}/{type} is provided, they will be used by default on bulk items that don’t provide them explicitly.

Content-Type: application/x-ndjson stands for Newline delimited JSON.

Before importing the JSON file, you might want to define mappings yourself or let Elasticsearch generate mappings dynamically during import. If you don't want Elasticsearch to generate mappings dynamically during import, refer to this doc to define mappings yourself.

References:

Bless
  • 5,052
  • 2
  • 40
  • 44
  • URL is given in the answer (`localhost:9200/{index}/{type}/_bulk?pretty`). Replace `{index}` and `{type}` with the required values. Eg.: `localhost:9200/my_index/my_type/_bulk?pretty` – Bless Dec 22 '17 at 22:59
1

You can use elasticsearch_loader for loading json files into elasticsearch (2.X, 5.X, 6.X).

You can download it with pip:

pip install elasticsearch-loader

And then you will be able to load json files into elasticsearch by issuing:

elasticsearch_loader --index incidents --type incident json file1.json file2.json

Disclaimer: I'm the author of elasticsearch_loader

MosheZada
  • 2,189
  • 1
  • 14
  • 17