I am trying to configure a Flume (1.6 from Cloudera CDH-5.11.0) agent to use an IBM WebSphere MQ 9.1.1.0 as Source.
As there's no official documentation, I followed these resources (despite they comes from older versions):
- http://wpcertification.blogspot.com/2015/08/configure-flume-to-use-ibm-mq-as-jms.html
- https://community.cloudera.com/t5/Data-Ingestion-Integration/Trying-to-integrate-ibm-mq-as-a-flume-source/td-p/5279
I proceeded this way:
Installed MQ 9.1.1.0 client and configured the JMSAdmin.conf
INITIAL_CONTEXT_FACTORY=com.sun.jndi.fscontext.RefFSContextFactory PROVIDER_URL=file:/C:/temp/jmsbindings SECURITY_AUTHENTICATION=none
Created the .bindings file:
- DEF CF(flumeConnectionFactory) QMGR(myQueueManager) HOSTNAME(myHostName) PORT(1414) CHANNEL(myChannelName) TRANSPORT(CLIENT)
- Extracted the java client (it is both available as groups of jars from the Windows client or as a single uberjar from IBM Support Portal). I tried both paths, however, right now I extracted these single jars:
- com.ibm.mq.jar
- com.ibm.mq.headers.jar
- com.ibm.mq.jmqi.jar
- com.ibm.mq.pcf.jar
- com.ibm.mq.mqjms.jar
- fcontext.jar
- jms.jar
- providerutil.jar
- Loaded jars and .bindings file on my flume host
- Prepared this source configuration:
test_IBM_MQ.sources.amq_ibm.type = jms test_IBM_MQ.sources.amq_ibm.initialContextFactory = com.sun.jndi.fscontext.RefFSContextFactory test_IBM_MQ.sources.amq_ibm.connectionFactory=flumeConnectionFactory test_IBM_MQ.sources.amq_ibm.providerURL = file:///etc/flume/conf test_IBM_MQ.sources.amq_ibm.destinationName = myDestination test_IBM_MQ.sources.amq_ibm.destinationType = QUEUE
When I finally start my Flume agent, I get this error:
java.lang.ClassCastException: javax.naming.Reference cannot be cast to javax.jms.ConnectionFactory
Followed by this little Stacktrace
Source amq_ibm has been removed due to an error during configuration java.lang.ClassCastException: javax.naming.Reference cannot be cast to javax.jms.ConnectionFactory at org.apache.flume.source.jms.JMSSource.doConfigure(JMSSource.java:231) at org.apache.flume.source.BasicSourceSemantics.configure(BasicSourceSemantics.java:65) at org.apache.flume.source.AbstractPollableSource.configure(AbstractPollableSource.java:65) at org.apache.flume.conf.Configurables.configure(Configurables.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:326) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:101) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:141) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Inspecting the Flume source, the exploding line is the following
connectionFactory = (ConnectionFactory) initialContext.lookup(connectionFactoryName);
I tried to follow this old related question
However, some of the jars the suggest to load do not exist anymore, since the upgrade to jms2.0 from IBM mq v8:
I am starting to suspect that's there is something which Flume 1.6 which doesn't like JMS2.0
Any help? Thanks
