Elasticsearch type object for a string

Question

Here is a simplified version of my mapping:

{
    "mappings": {
        "user-event": {
            "_all": {
               "enabled": true
            },
            "properties": {
                "account_id": {
                    "type": "string",
                    "index": "not_analyzed"
                },
                "name": {
                    "type": "string"
                },
                "payload": {
                    "type": "string",
                    "index": "not_analyzed"
                },
            }
        },
    }
}

Currently payload is a string, and this will be a stringified JSON payload, for instance: "{\"foo\": \"bar\"}"

Is it possible to change the payload type to JSON so that ES automatically parses the string and so I can do query on payload?

If not, do I have to write raw JSON into ES to be able to query it?

Yes, it's possible, but you need to modify the way you're sending your data to ES: How do you send your documents to ES? — Val, Jan 05 '17 at 06:39
It's a downstream service I do not control. I write to a kafka stream, and that gets published into ES. How so do I need to modify what I write? — Kousha, Jan 05 '17 at 06:41
Can you modify the documents before writing to Kafka? Another way is to do it after Kafka, are you using Logstash to consume Kafka and send to ES? — Val, Jan 05 '17 at 06:50
@Val; so I do have a question about that. I have a `org.apache.avro.specific.SpecificRecordBase` and this is the object I serialize and send to Kafka. Given avro does not have a generic `object` type, the `payload` type is set to `string`. So how can I write a json to kafka? — Kousha, Jan 05 '17 at 06:56
That leads to my second question: how are you consuming kafka? — Val, Jan 05 '17 at 06:58
You mean the consumer who then writes to ES? If so again that's downstream and I do not know. That service is not going to change. I believe they just write whatever is sent to kafka. So I need to write JSON to kafka to get that into ES. — Kousha, Jan 05 '17 at 06:59
You can then use a complex [Avro Record type](http://avro.apache.org/docs/1.8.1/spec.html#schema_complex) in order to encore your payload field. No need to store it as a JSON string. Some example here: http://stackoverflow.com/questions/28163225/how-to-define-avro-schema-for-complex-json-document — Val, Jan 05 '17 at 07:14
This works even if `payload` is not a of defined key/value pair? This payload could have any key/value of the user's chose — Kousha, Jan 05 '17 at 07:16
Well if you use Avro that's because you ideally want to keep some order in your data. If you let anything creep into your data, Avro is not really the best bet. What version of ES are you using? — Val, Jan 05 '17 at 07:20
Until you upgrade to ES 5 where you can use pipeline processors in order to transform your docs before they get indexed, you could use the deprecated [transform](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/mapping-transform.html) feature in order to deserialize your JSON payload. That should work, although I've never tried it. — Val, Jan 05 '17 at 07:25
Is [your other question](http://stackoverflow.com/questions/41474464/elasticsearch-possible-to-search-through-string-json) related? — Val, Jan 05 '17 at 07:28

score 0 · Accepted Answer · answered Jan 05 '17 at 06:44

ES don't have mapping type JSON, here all available types. But you can change your mapping and use nested type for it, after that you have to re-index all your data and during this re-index you have to post not JSON string but JSON object, something like this:

curl -XPUT localhost:9200/yourIndex/yourType/1 -d '{
    "account_id" : 1,
    "name" : "Admin",
    "payload": {"foo": "bar"}
}'

And everything will work!

Elasticsearch type object for a string

1 Answers1