how to feed data to Elasticseach as Integer using Python?

Question

i am using this python script to feed my data to elasticsearch 6.0. How can i store the variable Value with type float in Elasticsearch? I can't use the metric options for the visualization in Kibana, because all the data is stored automatically as string

from elasticsearch import Elasticsearch

Device=""
Value=""
for key, value in row.items():  
    Device = key
    Value = value
    print("Dev",Device,  "Val:", Value)                     
    doc = {'Device':Device, 'Measure':Value ,  'Sourcefile':filename}
    print('   doc:    ', doc)
    es.index(index=name, doc_type='trends', body=doc)

Thanks

EDIT:

After the advice of @Saul, i could fix this problem with the following code:

import os,csv
import time
from elasticsearch import Elasticsearch
#import pandas as pd
import requests

Datum = time.strftime("%Y-%m-%d_")
path = '/home/pi/Desktop/Data'


os.chdir(path)
name = 'test'
es = Elasticsearch() 

    #'Time': time ,
#url = 'http://localhost:9200/index_name?pretty'
doc = {
  "mappings": {
    "doc": { 
      "properties": { 
        "device":    { "type": "text"  }, 
        "measure":     { "type": "text"  }, 
        "age":      { "type": "integer" },  
        "created":  {
          "type":   "date", 
          "format": "strict_date_optional_time||epoch_millis"
        }
      }
    }
  }
}
#headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
#r = requests.post(url, data=json.dumps(data), headers=headers)
r= es.index(index=name, doc_type='trends', body=doc)

print(r)

Possible solution is to create a mapping in the index to define the data type for each attribute or field in the document. This is the official documentation https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html. — Saul Rosales, Dec 01 '17 at 22:58

Tim · Answer 1 · 2017-12-03T10:48:07.810

0

Elasticsearch defines field types in the index mapping. It looks like you probably have dynamic mapping enabled, so when you send data to Elasticsearch for the first time, it makes an educated guess about the shape of your data and the field types.

Once those types are set, they are fixed for that index, and Elasticsearch will continue to interpret your data according to those types no matter what you do in your python script.

To fix this you need to either:

Define the index mapping before you load any data. This is the better option as it gives you complete control over how your data is interpreted. https://www.elastic.co/guide/en/elasticsearch/reference/6.0/mapping.html
Make sure that, the first time you send data into the index, you use the correct data types. This will rely dynamic mapping generation, but it will typically do the right thing.

Defining the index mapping is the best option. It's common to do that once off, in Kibana or with curl, or if you create a lot of indices, with a template. However if you want to use python, you should look at the create or put_mapping functions on IndicesClient

edited Dec 03 '17 at 10:48

answered Dec 01 '17 at 23:44

Tim

6,406
22
34

i found the link already but i couldn't implement it in python. i didn't find any example for mapping ES using Python on WWW `PUT name { "mappings": { "doc": { "properties": { "Device": { "type": "text" }, "Measure": { "type": "integer" }, "created": { "type": "date", "format": "strict_date_optional_time||epoch_millis" } } } } }` – AhmyOhlin Dec 02 '17 at 04:29
Adding information like "oh, I tried that, but I ran into this other problem that I didn't mention" goes against the purpose of this site. Your job is to ask a question and include all relevant information, and then we answer it. We can't guess what you've already tried and why it didn't work for you. I'll update my answer. – Tim Dec 03 '17 at 10:42
thanks i will take your advice in my next questrions – AhmyOhlin Dec 04 '17 at 09:32

Saul Rosales · Accepted Answer · 2017-12-04T04:22:37.593

You need to send a HTTP Post request using python request, as follows:

url = "http://localhost:9200/index_name?pretty”
data = {
  "mappings": {
    "doc": { 
      "properties": { 
        "title":    { "type": "text"  }, 
        "name":     { "type": "text"  }, 
        "age":      { "type": "integer" },  
        "created":  {
          "type":   "date", 
          "format": "strict_date_optional_time||epoch_millis"
        }
      }
    }
  }
}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(data), headers=headers)

Please replace index_name in the URL with the name of the index you are defining in to elasticsearch engine.

If you want to delete the index before creating it again, please do as follows:

url = "http://localhost:9200/index_name”
data = { }
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.delete(url, data=json.dumps(data), headers=headers)

please replace index_name in the URL with your actual index name. After deleting the index, create it again with the first code example above including the mappings that you would need. Enjoy.

Thank you for the help. I change the index name and i executed it but i cant see any index in the plugin head ES or in Kibana: when i `print r i got ` — AhmyOhlin, Dec 04 '17 at 10:19
PS: Iam working with csv file, not json file.should i do some change here as well ? or it doesnt matters because it belongs to request syntaxe? — AhmyOhlin, Dec 04 '17 at 10:28
@AhmyOhlin, first question, you need to check status of the indexes with Index API, "localhost:9200/_cat/indices?v&pretty" this will show you the indexes status. Regarding to the cvs files, you need to change the data format to meet elastic search JSON format. — Saul Rosales, Dec 05 '17 at 18:49

how to feed data to Elasticseach as Integer using Python?

2 Answers2