Why after set mapping, index return nothing?

Question

I am using Elasticsearch 7.12.0 , Logstash 7.12.0, Kibana 7.12.0 on Windows 10 x64. Logstash config file logistics.conf

input {
  jdbc {
    jdbc_driver_library => "D:\\tools\\postgresql-42.2.16.jar"
    jdbc_driver_class => "org.postgresql.Driver"
    jdbc_connection_string => "jdbc:postgresql://localhost:5433/ld"
    jdbc_user => "xxxx"
    jdbc_password => "sEcrET"
    schedule => "*/5 * * * *"
    statement => "select * from inventory_item_report();"
    }
}

filter {
  uuid {
    target => "uuid"
  }
}

output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "localdist"
    document_id => "%{uuid}"
    doc_as_upsert => "true"
    }
}

Run logstash

logstash -f logistics.conf

If I don't set mapping explicit, the query

GET /localdist/_search
{
  "query": {
    "match_all": {}
  }
}

return many result.

My mappings

POST localdist/_mapping
{
  
}

DELETE /localdist

PUT /localdist
{
  
}

POST /localdist
{
  
}

PUT localdist/_mapping
{
  "properties": {
    "unt_cost": {
      "type": "double"
    },
    "ii_typ": {
      "type": "keyword"
    },
    "qty_uom_id": {
      "type": "keyword"
    },
    "prod_id": {
      "type": "keyword"
    },
    "root_cat_id": {
      "type": "keyword"
    },
    "uom": {
      "type": "keyword"
    },
    "product_name": {
      "type": "text"
    },
    "ii_id": {
      "type": "keyword"
    },
    "wght_uom_id": {
      "type": "keyword"
    },
    "iid_seq_id": {
      "type": "long"
    },
    "avai_diff": {
      "type": "double"
    },
    "invt_change_typ": {
      "type": "keyword"
    },
    "ccy": {
      "type": "keyword"
    },
    "exp_date": {
      "type": "date"
    },
    "req_amt": {
      "type": "text"
    },
    "pur_cost": {
      "type": "double"
    },
    "tot_pri": {
      "type": "long"
    },
    "own_pid": {
      "type": "keyword"
    },
    "doc_type": {
      "type": "keyword"
    },
    "ii_date": {
      "type": "date"
    },
    "fac_id": {
      "type": "keyword"
    },
    "shipment_type_id": {
      "type": "keyword"
    },
    "lot_id": {
      "type": "keyword"
    },
    "phy_invt_id": {
      "type": "keyword"
    },
    "facility_name": {
      "type": "text"
    },
    "amt_ohand_diff": {
      "type": "double"
    },
    "reason_id": {
      "type": "keyword"
    },
    "cat_id": {
      "type": "keyword"
    },
    "qty_ohand_diff": {
      "type": "double"
    },
    "@timestamp": {
      "type": "date"
    }
  }
}

run query

GET /localdist/_search
{
  "query": {
    "match_all": {}
  }
}

return nothing.

How to fix it, how to make explicit mappings works correctly?

What is POST mapping doing for you? – Tushar Shahi Jun 01 '21 at 11:20 — Tushar Shahi, Jun 01 '21 at 11:20
I use Logstash JDBC put index to Elasticsearch. – Vy Do Jun 01 '21 at 13:21 — Vy Do, Jun 01 '21 at 13:21

score 1 · Accepted Answer · answered Jun 01 '21 at 13:06

1

If I got you right, you are indexing via logstash. Elastic then create the index if missing, indexes the documents, and tries to guess the mapping for your documents based on the very first documents.

TL;DR: You are DELETING your index containing the data by yourself.

With

DELETE /localdist

you are deleting the whole index including all data. After that, by issuing

PUT /localdist
{
  
}

you are re-creating the previously deleted index which is empty again. And at the end, you are setting the index mapping with

PUT localdist/_mapping
{
  "properties": {
    "unt_cost": {
      "type": "double"
    },
    "ii_typ": {
      "type": "keyword"
    },
    ...

Now, as you have an empty elastic index with a mapping set, start the logstash pipeline again. If your documents are matching the index mapping, the docs should start to appear very quickly.

answered Jun 01 '21 at 13:06

ibexit

3,465
1
11
25

If Logstash pipeline has 15 fields, I explicit declaring mapping with 5 fields, what happens with other 10 fields, Please tell me the behavior in this case.? – Vy Do Jun 01 '21 at 13:21
If the fields mapping is not defined yet, elastic will try an "educated guess" based on the field value: https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapping.html – ibexit Jun 01 '21 at 14:05
I have mistake in function sql, when it call once, the result of JDBC pipeline return nearly nothing. – Vy Do Jun 02 '21 at 09:16
Cool, glad to hear that! Have fun! – ibexit Jun 02 '21 at 15:20

Why after set mapping, index return nothing?

1 Answers1