3

I have been trying to implement a SolrCloud, and everything works fine until I try to create a collection with 6 shards. My setup is as follows:

  • 5 virtual servers, all running Ubuntu 14.04, hosted by a single company across different data centers
  • 3 servers running ZooKeeper 3.4.6 for the ensemble
  • 2 servers, each running Solr 5.1.0 server (Jetty)
  • The Solr instances each have a 2TB, ext4 secondary disk for the indexes, mounted at /solrData/Indexes. I set this value in solrconfig.xml via <dataDir>/solrData/Indexes</dataDir>, and uploaded it to the ZooKeeper ensemble. Note that these secondary disks are neither NAS nor NFS, which I know can cause problems. The solr user owns /solrData.

All the intra-server communication is via private IP, since all are hosted by the same company. I'm using iptables for firewall, and the ports are open and all the servers are communicating successfully. Config upload to ZooKeeper is successful, and I can see via the Solr admin interface that both nodes are available.

The trouble starts when I try to create a collection using the following command:

http://xxx.xxx.xxx.xxx:8983/solr/admin/collections?action=CREATE&name=coll1&maxShardsPerNode=6&router.name=implicit&shards=shard1,shard2,shard3,shard4,shard5,shard6&router.field=shard&async=4444

Via the Solr UI logging, I see that multiple index creation commands are issued simultaneously, like so:

6/25/2015, 7:55:45 AM WARN SolrCore [coll1_shard2_replica1] Solr index directory '/solrData/Indexes/index' doesn't exist. Creating new index...
6/25/2015, 7:55:45 AM WARN SolrCore [coll1_shard1_replica2] Solr index directory '/solrData/Indexes/index' doesn't exist. Creating new index...

Ultimately the task gets reported as complete, but in the log, I have locking errors:

Error creating core [coll1_shard2_replica1]: Lock obtain timed out: SimpleFSLock@/solrData/Indexes/index/write.lock
SolrIndexWriter was not closed prior to finalize(),​ indicates a bug -- POSSIBLE RESOURCE LEAK!!!
Error closing IndexWriter

If I look at the cloud graph, maybe a couple of the shards will have been created, others are closed or recovering, and if I restart Solr, none of the cores can fire up.

Now, I know what you're going to say: follow this SO post and change solrconfig.xml locking settings to this:

<unlockOnStartup>true</unlockOnStartup>
<lockType>simple</lockType>

I did that, and it had no impact whatsoever. Hence the question. I'm about to have to release a single Solr instance into production, which I hate to do. Does anybody know how to fix this?

Community
  • 1
  • 1
LandonC
  • 889
  • 1
  • 16
  • 28
  • So you are trying to create 6 shards spread across two servers? I am not sure if the command you are using is correct. Can you try this query instead and see if it works. http://1.1.1.1:8983/solr/admin/collections?action=CREATE&name=yourcollectionname&numShards=6&replicationFactor=1&maxShardsPerNode=3&createNodeSet=1.1.1.1:8983_solr,2.2.2.2:8983_solr&collection.configName=yourconfigname I do not see createNodeSet param in your command which is used to specify which severs to spread the shards across. – jay Jun 25 '15 at 21:11
  • @jay Thanks, but no luck: all the exceptions are the same. – LandonC Jun 28 '15 at 20:07

1 Answers1

1

Based on the log entry you supplied, it looks like Solr may be creating the data (index) directory for EACH shard in the same folder. 

 Solr index directory '/solrData/Indexes/index' doesn't exist. Creating new index...

This message was shown for two different collections and it references the same location. What I usually do, is change my Solr Home to a different directory, under which all collection "instance" stuff will be created. Then I manually edit the core.properties for each shard to specify the location of the index data.

nick_v1
  • 1,654
  • 1
  • 18
  • 29
  • I'm sure you're right about all the shards using the same directory, but I never saw anything in the documentation indicating that each shard should have it's own directory. Is that what you're saying? That seems to fly in the face of the solrconfig.xml file kept under ZooKeeper having a single dataDir property. – LandonC Jun 28 '15 at 20:04
  • The configs will be stored in zookeeper, but you still need an instanceDir. Additionally, the data directories need to be different for each shard. – nick_v1 Jun 28 '15 at 23:20
  • OK, I totally missed that. Is there a reference in the documentation for instanceDir, and the concept of directory/shard? I want to make sure I can guide my coworkers to the right place in my absence. Thanks! – LandonC Jun 29 '15 at 02:17
  • If this answer was helpful kindly mark it as the answer. Thanks – nick_v1 Jun 29 '15 at 17:25
  • 2
    Manually editing core.properties and setting each shard to its own data directory is the answer, despite the fact that this information is provided _nowhere_ in the Solr documentation. – LandonC Jul 07 '15 at 22:03