Import nested entities in Solr DIH

Question

I need configure nested entities which are imported via DIH. Between user and address is one to many cardinality (user has many addresses).

This is our import definition in data-config.xml

<document>
    <entity name="user" query="...">
           <field column="id" name="id" />
           <field column="code" name="code" />
        // next fields of user

        <entity name="address" child="true" query="..." where="user_id=user.id">
           <field column="id" name="id" />
           <field column="city" name="city" />
           // ... next fields of address
        </entity>
    </entity>
</document>

and schema.xml configuration like this:

// user fields
<field name="id" type="long" indexed="true" stored="true" />
<field name="name" type="string" indexed="true" stored="true" />
<field name="code" type="string" indexed="true" stored="true" />
// ...

// address fields
<field name="address" type="string" indexed="true" multiValued="true"  stored="true" />
<field name="address.id" type="long" indexed="true" stored="true" />
<field name="address.city" type="string" indexed="false" stored="true" />
// ...

This solution cause that no address object is imported. Thank you for any advice.

EDIT: I also found many warning logs Error creating document : SolrInputDocument(fields: [user_id=122, ... _version_=1671840418228076544, _root_=00924553002], children: [SolrInputDocument(fields: [address_id=1, _root_=00924553002, _version_=1671840418228076544]), SolrInputDocument(fields: [address_id=20, _root_=00924553002, _version_=1671840418228076544])])

EDIT 2: Default logs in application hide error. I checked server logs on machine, and I found this error: org.apache.solr.common.SolrException: [doc=null] missing required field: code at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:245)

In schema.xml I have this field set as identifier of user, but children doesn't have this field. Then I tried add code field as alias for ID, and I found another error which tell there is missing value in required fields which are used in parent object (user). This conditions were applied also on nested objects. So I tried also remove these condition from these fields. After this import ran, but when I execute select, all objects are on same level. Solr imported it as flat.

This is expected select result:

{
  "reponse": [
    {
      "id": 1,
      "code": "tsdx242-234",
      "first_name": "Michael",
      "last_name": "Sprox",
      "addresses": [
        {
          "id": 44,
          "city": "Paris",
          "street": "Champs-Elysees"
        },
        {
          "id": 24,
          "city": "Budapest",
          "street": "Akácfa utca"
        }
      ]
    },
    {
      "id": 2,
      "code": "xx45982-114",
      "first_name": "Petra",
      "last_name": "Jurka",
      "addresses": [
        {
          "id": 31,
          "city": "Vienna",
          "street": "Karlsplatz"
        },
        {
          "id": 44,
          "city": "Paris",
          "street": "Champs-Elysees"
        }
      ]
    }
  ]
}

But it gives this, mixed users and addresses in one lovel:

{
  "response": [
    {
      "id": 44,
      "city": "Paris",
      "street": "Champs-Elysees"
    },
    {
      "id": 24,
      "city": "Budapest",
      "street": "Akácfa utca"
    },
    {
      "id": 31,
      "city": "Vienna",
      "street": "Karlsplatz"
    },
    {
      "id": 1,
      "code": "tsdx242-234",
      "first_name": "Michael",
      "last_name": "Sprox"
    },
    {
      "id": 2,
      "code": "xx45982-114",
      "first_name": "Petra",
      "last_name": "Jurka"
    }
  ]
}

check this https://stackoverflow.com/questions/24920305/defining-nested-entities-in-solr-data-import-handler — Abhijit Bashetti, Jul 10 '20 at 12:24
https://stackoverflow.com/questions/40838090/configuration-nested-entity-using-dih-in-solr — Abhijit Bashetti, Jul 10 '20 at 12:24
@AbhijitBashetti not helps, but I found something in logs, so I edited question. Check it please. — Denis Stephanov, Jul 10 '20 at 14:33
hey , you can find exact error in ..\solr-7.6.0\server\logs\solr.log , can you plz edit your question with that error log. — Mansi Joshi, Jul 11 '20 at 10:49
@Denis Stephanov your error shows that `code` fields is require in your schema and when solr perform indexing it missing from database. so if that field is not required then simple set `required="false"`. hope this help!! — Mansi Joshi, Jul 11 '20 at 17:57
@MansiJoshi yes, I edited it and remove required condition where it failed, but problem is that after this, solr import data in flat model. It didn't create array field with nested objects as I expected — Denis Stephanov, Jul 11 '20 at 18:20
@Denis Stephanov can you plz share your current output and expected output. — Mansi Joshi, Jul 12 '20 at 08:38
@Denis Stephanov https://lucene.apache.org/solr/guide/8_1/indexing-nested-documents.html can you check `Schema Configuration` from this link for `root` fields and other some ponits. — Mansi Joshi, Jul 12 '20 at 16:43

score 0 · Answer 1 · answered Jul 10 '20 at 12:34

0

Try to rewrite it to someting like this

<document>
        <entity name="user" query="...">
               <field column="id" name="id" />
            // next fields of user
    
            <entity name="second_entity_name" pk="user_id" query="SELECT ... FROM where user_id = '${user.id}'" >
               <field column="id" name="id" />
               <field column="city" name="city" />
               // ... next fields of address
            </entity>
        </entity>
    </document>

answered Jul 10 '20 at 12:34

Oyeme

11,088
4
42
65

Hi, this no helped, but I found some warning logs, so I added it into question. – Denis Stephanov Jul 10 '20 at 14:29
You can expand logs to see more detailed info, it might be that mappings type is wrongly set – Oyeme Jul 10 '20 at 17:47
Check EDIT 2 please. I found possible reason of error. – Denis Stephanov Jul 11 '20 at 17:48
Try to do mapping slowly by mapping one field to one and set require to false. Looks like you have wrong mapping – Oyeme Jul 11 '20 at 21:20
I tried and solr imported data but wrong. Check edit 2, import pass but data were imported in flat model, so there were not hierarchy and all objects were on same level – Denis Stephanov Jul 11 '20 at 22:27

Import nested entities in Solr DIH

1 Answers1