0

Background: 2 out of 3 of our Zookeeper machines snapped which caused my SOLR system to collapse. I added new Zookeeper machines and updated each SOLR machine config with the new Zookeeper machine.

After this, I started the SOLR and used admin page to query *:* which returned a different number every time I queried the pool.

So, I purged all the records in SOLR cloud and ran a batch-job to populate all the data again from Oracle to SOLR. (Everything looked good).

Problem: I have a daily batchjob which updates the SOLR with DELTA(Inserts + Updates) from Oracle.

Since this instance. The number in SOLR pool are not matching the DELTA(insert + update). For ex: even though 1000 records were updated or inserted on a day, SOLR counts differ by more than 10000.

The numbers returned by *:* are not matching. We have tried purging the records multiple time. Things look good when we insert the record for the first time after purge but as soon as updates start happening, numbers don't match.

There are no Duplicate Records. and If I query for a specific record we get the correct record but facet numbers are wrong too.

Is the index file is corrupted?

user1324887
  • 632
  • 3
  • 11
  • 32
  • 1. How many Solr instances for the cluster? 2. How many shards? 3. How do you do indexing? Through SolrJ or anything else? – s.xie Mar 03 '17 at 02:50
  • 25 instances in the cluster, 3 Shards. I have a Spring batch job which reads the records and updated them in `SOLR` – user1324887 Mar 03 '17 at 02:54
  • did zk track the proper status of all the instances? – s.xie Mar 03 '17 at 03:00
  • @s.xie, yes it did. and the new zk also have all the instances. cloud looks good. – user1324887 Mar 03 '17 at 03:04
  • Nothing can ring a bell from these facts. A few suggestions: 1. solr config commit strategy. 2. check solr logs, if replication is properly done. 3. maybe rebuild solr on exiting zk ensemble? – s.xie Mar 03 '17 at 03:13
  • @s.xie, just following up on this again. found that there are duplicate documents in our shards with different `_version_`. I ran the solr queries with `distrib=false` on 2 different nodes from 2 shards and they both returned 2 documents with different `_version_`. any insights? – user1324887 Mar 13 '17 at 18:19

1 Answers1

0

Try optimizing your index. I was also facing the same issue and optimizing the index fixed it.

      curl http://hostname:8980/solr/<core>/update?optimize=true

Some more info on optimize:

http://wiki.apache.org/solr/SolrPerformanceFactors#Optimization_Considerations

PS: Note that an optimize is expensive. You should not run it more than once daily.

kpahwa
  • 723
  • 4
  • 7
  • I have tried optimizing the index. That did not change anything so far. – user1324887 Mar 03 '17 at 18:53
  • just following up on this again. found that there are duplicate documents in our shards with different `_version_`. I ran the solr queries with `distrib=false` on 2 different nodes from 2 shards and they both returned 2 documents with different `_version_`. any insights? – user1324887 Mar 13 '17 at 18:20