How to update Mapping

Question

How can I update index mapping to include the following field doc_as_upsert : true

My logstash ingesting cloudtrail logs from s3 is showing the following on the log Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2018.10.08", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x251f932>], :response=>{"index"=>{"_index"=>"cloudtrail-2018.10.08", "_type"=>"doc", "_id"=>"t2mmVWYBVQr-RbWuAQIS", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [requestParameters.disableApiTermination]", "caused_by"=>{"type"=>"json_parse_exception", "reason"=>"Current token (START_OBJECT) not of boolean type\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@133a6c; line: 1, column: 1509]"}}}}}

Mapping is dynamic and very long so can't fit it all here but here is what I cat fit

{
  "cloudtrail-2018.10.08": {
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "message_field": {
              "path_match": "message",
              "match_mapping_type": "string",
              "mapping": {
                "norms": false,
                "type": "text"
              }
            }
          },
          {
            "string_fields": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "fields": {
                  "keyword": {
                    "ignore_above": 256,
                    "type": "keyword"
                  }
                },
                "norms": false,
                "type": "text"
              }
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "keyword"
          },
          "geoip": {
            "dynamic": "true",
            "properties": {
              "ip": {
                "type": "ip"
              },
              "latitude": {
                "type": "half_float"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "half_float"
              }
            }
          }
        }
      },
      "doc": {
        "dynamic_templates": [
          {
            "message_field": {
              "path_match": "message",
              "match_mapping_type": "string",
              "mapping": {
                "norms": false,
                "type": "text"
              }
            }
          },
          {
            "string_fields": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "fields": {
                  "keyword": {
                    "ignore_above": 256,
                    "type": "keyword"
                  }
                },
                "norms": false,
                "type": "text"
              }
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "keyword"
          },
          "additionalEventData": {
            "properties": {
              "configRuleArn": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "configRuleInputParameters": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "configRuleName": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "managedRuleIdentifier": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "notificationJobType": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "vpcEndpointId": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "apiVersion": {
            "type": "date"
          },
          "awsRegion": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "errorCode": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "errorMessage": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "eventID": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "eventName": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "eventSource": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "eventType": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "eventVersion": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }

Here am get the following errors when I try update the mapping with these

PUT cloudtrail-*/_mapping/_doc
{
  "properties": {
    "doc_as_upsert": true
    }
  }

I get error

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Expected map for property [fields] on field [doc_as_upsert] but got a class java.lang.String"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Expected map for property [fields] on field [doc_as_upsert] but got a class java.lang.String"
  },
  "status": 400
}

The error you see suggests that Elasticsearch failed to parse the JSON you send. It could be you are sending data in format it does not expect. If it is possible please post your mapping and the JSON document ES is complaining about. — Nikolay Vasiliev, Oct 09 '18 at 07:05
Thanks for the updated info in the question. Do you want Elasticsearch to do "upsert" to any document you send for indexing by default? I'm afraid this can't be done. As Terek showed below, `doc_as_upsert` is a property of `_update` API, not of `PUT _mapping`. Please reply if I didn't understand you right. — Nikolay Vasiliev, Oct 11 '18 at 11:25

Tarek Essam · Answer 1 · 2018-10-09T09:46:24.590

0

doc_as_upsert is a flage you use to tell elasticsearch that you want to update the document with the content of doc as the upsert value. it has nothing to do with update index mapping.

assume you want to update document of id 1 and index test (update the name).

POST test/_doc/1/_update
{
    "doc" : {
    "name" : "new_name"
    },
"doc_as_upsert" : true
}

edited Oct 09 '18 at 09:46

answered Oct 08 '18 at 22:35

Tarek Essam

3,602
2
12
21

will the following work as well ? ` PUT cloudtrail-*/_mapping/_doc { "properties": { "doc_as_upsert": true } } } ` – tripleb Oct 09 '18 at 08:29
In the mapping you specify the datatype of the field for the Elasticsearch to be able to correctly index that field. So do not specify a value in the mapping just set the datatype to be a "boolean". – Tarek Essam Oct 09 '18 at 08:54
still getting error, this is what I ran ` PUT cloudtrail-*/_mapping/_doc { "properties": { "doc_as_upsert": { "type": "true" } } } ` and error ` { "error": { "root_cause": [ { "type": "mapper_parsing_exception", "reason": "No handler for type [true] declared on field [doc_as_upsert]" } ], "type": "mapper_parsing_exception", "reason": "No handler for type [true] declared on field [doc_as_upsert]" }, "status": 400 } ` – tripleb Oct 09 '18 at 09:05
I thought you wanted to add a new field and insert its mapping. I edit the answer. Thats is how doc_as_upsert work. If you want anything else tell me. – Tarek Essam Oct 09 '18 at 09:47

How to update Mapping

1 Answers1