0

I recently started implementing solr-cloud on AWS EC2 for search applications. I have created 2 AWS Ec2 instances with the following configurations ---

  1. EC2 Type - t2.medium
  2. ram - 4GB
  3. Disk Space - 8GB
  4. OS - ubuntu 18.04

For the 2 EC2 instances, I have created a security group which allows all inbound traffic. NACL has default settings that allows all inbound traffic as well.

Steps Followed to install Apache Solr -

  1. ssh into ec2 :
ssh -i "pem_file" ubuntu@ec2-public-ipv4-address
  1. cd to /opt directory
  2. run --> sudo apt-update
  3. run --> sudo apt-get openjdk-11
  4. Check java -version
  5. run --> wget https://archive.apache.org/dist/lucene/solr/8.3.0/solr-8.3.0.tgz
  6. run --> tar -xvzf solr-8.3.0.tgz
  7. export SOLR_HOME=/opt/solr-8.3.0
  8. Add /opt/solr-8.3.0 to Path environment variable
  9. Update the sudo vim /etc/hosts file with the hosts -- a. public-ip-v4-address-of-ec2 solr-node-1
  10. Started Solr using the following command --> sudo bin/solr start -c -p 8983 -h solr-node-1 -force
  11. Checked the opened ports using --> sudo lsof -i -P -n | grep LISTEN
  12. Created collections, shards and replicas using ---> bin/solr create -c travasko -d sample_techproducts_configs -n travasko_configs -shards 2 -rf 2 -p 8983

I repeated the same process on the other EC2 machine and ran solr on it. Now, to use the data import handler in solr, I edited the following files:

  1. solrconfig.xml
<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
    <lst name="defaults">
      <str name="config">data-config.xml</str>
    </lst>
</requestHandler>
  1. data-config.xml
<dataConfig>
<dataSource type="JdbcDataSource" 
            driver="com.mysql.jdbc.Driver"
            url="jdbc:mysql://examplerds.cuhj86yfdpid.us-east-1.rds.amazonaws.com:3306/TRAVASKODB1" 
            user="examplerds" 
            password="examplerds#123"/>
<document>
  <entity name="MOMENTS"  
    pk="MOMENT_ID"
    query="SELECT MOMENT_ID,MOMENT_TEXT FROM MOMENTS"
    deltaImportQuery="SELECT MOMENT_ID,MOMENT_TEXT FROM MOMENTS WHERE MOMENT_ID='${dih.delta.MOMENT_ID}'"
    deltaQuery="SELECT MOMENT_ID FROM MOMENTS  WHERE LAST_MODIFIED > '${dih.last_index_time}'"
    >
     <field column="MOMENT_ID" name="MOMENT_ID"/>
     <field column="MOMENT_TEXT" name="MOMENT_TEXT"/>       
  </entity>
</document>
</dataConfig>
  1. managed_schema
<schema name="MOMENTS" version="1.5">
    <field name="_version_" type="long" indexed="true" stored="true"/>
    <field name="MOMENT_ID" type="integer" indexed="true" stored="true" required="true" multiValued="false" /> 
    <field name="MOMENT_TEXT" type="string" indexed="true" stored="true" multiValued="false" />
</schema>
  1. Downloaded mysql jdbc using the following command: 
wget -q "http://search.maven.org/remotecontent?filepath=mysql/mysql-connector-java/5.1.32/mysql-connector-java-5.1.32.jar" -O mysql-connector-java.jar
  1. Add to solrconfig.xml:
<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />

<lib dir="${solr.install.dir:../../../..}/dist/" regex="mysql-connector-java.jar" />
  1. After editing the files above, I uploaded them to the solr-cloud using the following zookeper command -->
bin/solr zk -n travasko_config -z solr-node-1:9983 cp /opt/solr-8.3.0/server/solr/configsets/_default/conf/managed-schema zk:/configs/travasko_config/managed-schema
  1. I then checked all the above files in the solr-cloud and could notice the changes i added.
  2. The current issue is that when I select the collection I created above, and click on Dataimport, It throws an error as below ---> 
The solrconfig.xml file for this index does not have an operational DataImportHandler defined!

Note: The AWS RDS and EC2 instances are in the same VPC sharing the same Security Group.

So why is solrconfig.xml file throwing an error during dataimport ? What am i missing here?

Vinit Kumar
  • 55
  • 10
  • 1
    When you're starting Solr in "cloud mode" with `-c` the configs are stored in Zookeeper - See [Using Zookeeper to manage configuration files](https://lucene.apache.org/solr/guide/8_3/using-zookeeper-to-manage-configuration-files.html). In effect you've probably not made your changes visible to Solr, since they're not available before being uploaded to Zookeeper. In addition - be _very_ careful about exposing Solr on public ips without additional security; it's not meant to be publicly available. – MatsLindh Apr 07 '20 at 07:03
  • @MatsLindh, I edited my answer above. I ran the zookeeper commands to upload the files. Besides, at this point I wanted to configure solr and hence was not focusing on the security. Once, I successfully, implement the basic model, I will definitely incorporate that. – Vinit Kumar Apr 07 '20 at 07:26
  • According to your `zk` command you've only uploaded the managed-schema? You want to upload the complete configset: https://stackoverflow.com/questions/50238528/solr-index-does-not-have-an-operational-dataimporthandler-defined – MatsLindh Apr 07 '20 at 07:30
  • @MatsLindh I actually added the entire configset. The above zookeeper command was for my latest update to the manaed_schema file. Forgot to mention that. – Vinit Kumar Apr 07 '20 at 07:32

1 Answers1

1

The solution to the above issue was basically setting the java system property for solr versions greater than 8.2.0 as below: 

-Denable.dih.dataConfigParam=true

This parameter can be set either in solr.in.cmd or solr.in.sh which can be found inside the directory below: , 

/opt/solr-8.3.0/bin

If, /opt/solr-8.3.0 is the installation directory of solr.

The other method was to pass this parameter as command line parameter while starting solr as below: 

sudo bin/solr start -c -p 8983 -h solr-node-1 -Denable.dih.dataConfigParam=true -force

solr-node-1 is the public IPv4 address of the AWS Ec2 instance on which solr is configured.

Vinit Kumar
  • 55
  • 10