Solr Delta Import Query is not working

Question

I am trying to import data from Mongodb to Solr6.0. Full import is executing properly but delta import is not working. When I execute delta import I get below result.

Requests: 0 , Fetched: 0 , Skipped: 0 , Processed: 0

My data config file queries are as below

query=""

deltaQuery="db.getCollection('customer').find({'jDate':{$gt:'${dih.last_index_time}'}},{'_id' :1});"
deltaImportQuery="db.getCollection('customer').find({'_id':'${dataimporter.delta.id}'})"

the whole data-config.xml

<?xml version="1.0"?>
<dataConfig>
<dataSource name="MyMongo" type="MongoDataSource" database="test_db" />
<document name="import">
 <!-- if query="" then it imports everything -->
     <entity  processor="MongoEntityProcessor"
             query=""
             deltaQuery="db.getCollection('customer').find({'jDate':{$gt: '${dih.last_index_time}'}},{'_id' :1});"
             deltaImportQuery="db.getCollection('customer').find({'_id':'${dataimporter.delta.id}'})"
             collection="customer"   
             datasource="MyMongo"
             transformer="MongoMapperTransformer" name="sample_entity">

               <!--  If mongoField name and the field declared in schema.xml are same than no need to declare below.
                     If not same than you have to refer the mongoField to field in schema.xml
                    ( Ex: mongoField="EmpNumber" to name="EmployeeNumber"). -->                                              

           <field column="_id"  name="id"/>  
           <field column="CustID" name="CustID" mongoField="CustID"/>       

           <field column="CustName" name="CustName" mongoField="CustName"/>       
            <field column="jDate" name="jDate" mongoField="jDate"/>
            <field column="dob" name="dob" mongoField="dob"/>          
       </entity>
 </document>
</dataConfig>

I tried with hardcoded values, but it still not worked like below

     query=""
     deltaQuery="db.getCollection('customer').find({'jDate':{$gt: new Date(1480581295000)}},{'_id' :1});"

deltaImportQuery="db.getCollection('customer').find({'_id':ObjectId('34234234dfsd34534524234ee')})"

And then I tried doing the below thing, but still no success

query=""
         deltaQuery="{'jDate':{$gt: new Date(1480581295000)}}"
        deltaImportQuery="{'_id':ObjectId(34234234dfsd34534524234ee)}"

Anybody, Please if you can.

@TMBT Please help me out in this.I have this new question suggested by you. — Kamini, Nov 30 '16 at 05:48
Yes.After full import , I have added one new document in MongoDB and then ran delta import — Kamini, Nov 30 '16 at 06:10
just curious whys is your query=""(empty). should it not be the same as you had when doing full import? — root, Nov 30 '16 at 06:16
Also is the 'new' keyword really necessary in {$gt: new '${dih.last_index_time}'}} — root, Nov 30 '16 at 06:52
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/129403/discussion-between-kamini-and-root545). — Kamini, Nov 30 '16 at 06:57
@Kamini, I am facing a similar problem too. Can you please show how did you resolve the issue? — Mallikarjun Sannapyati, Nov 21 '19 at 00:06
It's possible to have the same problem with the following issue: [https://stackoverflow.com/questions/58965037/solr-mongo-docdb-delta-import-query-is-not-working/60668539#60668539](https://stackoverflow.com/questions/58965037/solr-mongo-docdb-delta-import-query-is-not-working/60668539#60668539) — Mihai Stancu, Mar 13 '20 at 10:17

nagendra patod · Answer 1 · 2017-12-26T13:05:17.530

can you replace {'_id':'${dataimporter.delta.id}'} to {'_id':'${dataimporter.delta._id}'} in deltaImportQuery.

deltaImportQuery : (Only used in delta-import) .There is a namespace ${dih.delta.column-name} which can be used in this query. e.g: select * from tbl where id=${dih.delta.id} Solr1.4

as above line mentioned that we can only use column name in delta namespace.

Same problem also faced during delta import from mysql in question Solr delta import Query exception

reference:
https://wiki.apache.org/solr/DataImportHandler https://wiki.apache.org/solr/DataImportHandler#Configuration_in_data-config.xml

score 0 · Answer 2 · edited Feb 11 '19 at 15:07

0

Can you try

deltaQuery as db.getCollection('customer').find({'jDate':{$gt:**ISODate(**'${dih.last_index_time}'**)**}},{'_id' :1});

edited Feb 11 '19 at 15:07

Robert Columbia

6,313
15
32
40

answered Feb 11 '19 at 15:05

Venkata Madhu

93
1
14

Solr Delta Import Query is not working

2 Answers2

Linked