0

I am using Voltdb. And my use case is to import data from kafka to voltdb. I am using below command : Command: kafkaloader test --brokers <>:2181, --topic kafkavoltdb

In deployment.xml file the configuration is: 

 <security enabled="false" provider="hash"/>
 <import>
  <configuration type="kafka" enabled="true" format="csv">
   <property name="topics">kafkavoltdb</property>
    <property name="procedure">TEST.insert</property>
         <property name="brokers">brokers:6667</property>
           </configuration> 
             </import>

I am not able to fetch data from kafka to voltdb and the kafkaloader commands hungs up and not throwing any error. The logs showing : Failed to get Kafka partition info org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata. Note: i am using apache kafka (HDP version 3.0 ,Kerberos security cluster) Kindly help me with solution.

Ashish Mathur
  • 33
  • 4
  • Try setting offset to earliest before start consuming – Ryuzaki L Mar 13 '19 at 09:31
  • Kafka with Kerberos is currently not supported by VoltDB. This is planned for a future version. – Andrew Mar 13 '19 at 15:04
  • Hi Andrew, Just to understand your comment, if my hadoop cluster is kerberised where my kafka and hive services are running, then voltdb will not be able to integrate with my hadoop cluster. My second question, if I have separate kafka cluster which is not kerberized and my hadoop cluster is kerberized where my hive service is running, can voltdb connet to hadoop via kafka. – Ashish Mathur Mar 14 '19 at 13:18
  • You can export directly to a kerberized hadoop cluster using standard export settings in the configuration file. The only thing that is not supported is Kerberized Kafka. This is planned for an upcoming release. Full Disclosure: I work at VoltDB. I recommend this section of the docs for how to export to a kerberized hadoop cluster: https://docs.voltdb.com/UsingVoltDB/ExportToHttp.php#ExportHttpKerberos – Andrew Mar 14 '19 at 14:14
  • Thanks for the info Andrew. The other option i have for injecting streaming data into voltdb is through spark .Is connectivity between spark and Voltdb possible.Be informed the my apache spark is also kerberized.I want to import streaming data into voltdb through spark – Ashish Mathur Mar 18 '19 at 11:08

0 Answers0