0

I'm trying to create stream of data from multiple sources and load it into hdfs.

For proof of concept i'm using single node hadoop instalation and Spring-XD on a same machine. Hadoop was deployed with Ambari and reports to be functional and healthy (HDP 2.0). Spring-XD is installed in /var/spring-xd with environment variable $XD_HOME set to /var/spring-xd/xd; both server and client of Spring run with --hadoopDistro hdp20 flag. However i can only get those messages in log file so far:

18:00:06,346  INFO DeploymentsPathChildrenCache-0 Configuration.deprecation:840 - fs.default.name is deprecated. Instead, use fs.defaultFS
18:00:06,692  WARN DeploymentsPathChildrenCache-0 util.NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18:00:07,944  INFO DeploymentsPathChildrenCache-0 logging.ClasspathLoggingApplicationListener:55 - Application failed to start with classpath: [file:/var/spring-xd/xd/modules/processor/scripts/, file:/var/spring-xd/xd/config/, file:/var/spring-xd/xd/, file:/var/spring-xd/xd/lib/activation-1.1.jar, file:/var/spring-xd/xd/lib/amqp-client-3.2.4.jar, file:/var/spring-xd/xd/lib/aopalliance-1.0.jar, file:/var/spring-xd/xd/lib/args4j-2.0.16.jar, file:/var/spring-xd/xd/lib/asm-3.1.jar, file:/var/spring-xd/xd/lib/aspectjrt-1.6.6.jar, file:/var/spring-xd/xd/lib/aspectjweaver-1.6.6.jar, file:/var/spring-xd/xd/lib/avro-1.7.5.jar, file:/var/spring-xd/xd/lib/cglib-2.2.1-v20090111.jar, file:/var/spring-xd/xd/lib/cloudfoundry-connector-0.9.2.jar, file:/var/spring-xd/xd/lib/com.ibm.jbatch-tck-spi-1.0.jar, file:/var/spring-xd/xd/lib/commons-beanutils-1.7.0.jar, file:/var/spring-xd/xd/lib/commons-beanutils-core-1.8.0.jar, file:/var/spring-xd/xd/lib/commons-cli-1.2.jar, file:/var/spring-xd/xd/lib/commons-codec-1.7.jar, file:/var/spring-xd/xd/lib/commons-collections-3.2.1.jar, file:/var/spring-xd/xd/lib/commons-compress-1.4.1.jar, file:/var/spring-xd/xd/lib/commons-configuration-1.6.jar, file:/var/spring-xd/xd/lib/commons-daemon-1.0.13.jar, file:/var/spring-xd/xd/lib/commons-dbcp-1.2.2.jar, file:/var/spring-xd/xd/lib/commons-digester-1.8.jar, file:/var/spring-xd/xd/lib/commons-el-1.0.jar, file:/var/spring-xd/xd/lib/commons-fileupload-1.2.1.jar, file:/var/spring-xd/xd/lib/commons-httpclient-3.1.jar, file:/var/spring-xd/xd/lib/commons-io-2.4.jar, file:/var/spring-xd/xd/lib/commons-jexl-2.1.1.jar, file:/var/spring-xd/xd/lib/commons-lang-2.6.jar, file:/var/spring-xd/xd/lib/commons-math-2.1.jar, file:/var/spring-xd/xd/lib/commons-net-3.3.jar, file:/var/spring-xd/xd/lib/commons-pool-1.3.jar, file:/var/spring-xd/xd/lib/core-0.9.2.jar, file:/var/spring-xd/xd/lib/curator-client-2.4.1.jar, file:/var/spring-xd/xd/lib/curator-framework-2.4.1.jar, file:/var/spring-xd/xd/lib/curator-recipes-2.4.1.jar, file:/var/spring-xd/xd/lib/ehcache-core-2.3.0.jar, file:/var/spring-xd/xd/lib/freemarker-2.3.15.jar, file:/var/spring-xd/xd/lib/groovy-all-2.2.2.jar, file:/var/spring-xd/xd/lib/guava-14.0.1.jar, file:/var/spring-xd/xd/lib/guice-3.0.jar, file:/var/spring-xd/xd/lib/guice-servlet-3.0.jar, file:/var/spring-xd/xd/lib/hibernate-validator-4.3.1.Final.jar, file:/var/spring-xd/xd/lib/hsqldb-2.3.0.jar, file:/var/spring-xd/xd/lib/jackson-annotations-2.3.0.jar, file:/var/spring-xd/xd/lib/jackson-core-2.3.2.jar, file:/var/spring-xd/xd/lib/jackson-core-asl-1.9.13.jar, file:/var/spring-xd/xd/lib/jackson-databind-2.3.2.jar, file:/var/spring-xd/xd/lib/jackson-mapper-asl-1.9.13.jar, file:/var/spring-xd/xd/lib/javax.batch-api-1.0.jar, file:/var/spring-xd/xd/lib/javax.inject-1.jar, file:/var/spring-xd/xd/lib/javax.mail-1.5.0.jar, file:/var/spring-xd/xd/lib/jboss-logging-3.1.0.CR2.jar, file:/var/spring-xd/xd/lib/jcl-over-slf4j-1.7.6.jar, file:/var/spring-xd/xd/lib/jersey-guice-1.9.jar, file:/var/spring-xd/xd/lib/jersey-test-framework-grizzly2-1.9.jar, file:/var/spring-xd/xd/lib/jettison-1.2.jar, file:/var/spring-xd/xd/lib/joda-time-2.1.jar, file:/var/spring-xd/xd/lib/jolokia-core-1.1.5.jar, file:/var/spring-xd/xd/lib/jolokia-jmx-1.1.5.jar, file:/var/spring-xd/xd/lib/jolokia-jvm-1.1.5.jar, file:/var/spring-xd/xd/lib/jolokia-spring-1.1.5.jar, file:/var/spring-xd/xd/lib/json-path-0.8.1.jar, file:/var/spring-xd/xd/lib/json-simple-1.1.jar, file:/var/spring-xd/xd/lib/json-smart-1.1.1.jar, file:/var/spring-xd/xd/lib/kite-data-core-0.12.1.jar, file:/var/spring-xd/xd/lib/kryo-2.22.jar, file:/var/spring-xd/xd/lib/lettuce-2.3.3.jar, file:/var/spring-xd/xd/lib/log4j-1.2.17.jar, file:/var/spring-xd/xd/lib/mysql-connector-java-5.1.23.jar, file:/var/spring-xd/xd/lib/netty-3.7.0.Final.jar, file:/var/spring-xd/xd/lib/objenesis-1.3.jar, file:/var/spring-xd/xd/lib/opencsv-2.3.jar, file:/var/spring-xd/xd/lib/paranamer-2.3.jar, file:/var/spring-xd/xd/lib/parquet-avro-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-column-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-common-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-encoding-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-format-1.0.0.jar, file:/var/spring-xd/xd/lib/parquet-generator-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-hadoop-1.2.5.jar, file:/var/spring-xd/xd/lib/postgresql-9.2-1002-jdbc4.jar, file:/var/spring-xd/xd/lib/slf4j-api-1.7.6.jar, file:/var/spring-xd/xd/lib/slf4j-log4j12-1.7.6.jar, file:/var/spring-xd/xd/lib/snakeyaml-1.12.jar, file:/var/spring-xd/xd/lib/snappy-java-1.0.5.jar, file:/var/spring-xd/xd/lib/spring-amqp-1.3.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-aop-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-batch-admin-manager-1.3.0.M1.jar, file:/var/spring-xd/xd/lib/spring-batch-admin-resources-1.3.0.M1.jar, file:/var/spring-xd/xd/lib/spring-batch-core-3.0.0.M3.jar, file:/var/spring-xd/xd/lib/spring-batch-infrastructure-3.0.0.M3.jar, file:/var/spring-xd/xd/lib/spring-batch-integration-3.0.0.M3.jar, file:/var/spring-xd/xd/lib/spring-beans-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-boot-1.0.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-boot-actuator-1.0.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-boot-autoconfigure-1.0.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-context-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-context-support-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-core-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-data-commons-1.6.2.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-data-redis-1.2.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-expression-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-hateoas-0.9.0.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-integration-amqp-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-core-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-event-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-file-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-ftp-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-groovy-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-http-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-jmx-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-redis-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-scripting-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-jdbc-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-messaging-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-plugin-core-0.8.0.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-rabbit-1.3.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-retry-1.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-service-connector-0.9.2.jar, file:/var/spring-xd/xd/lib/spring-tx-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-web-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-webmvc-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-xd-analytics-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-batch-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-dirt-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-hadoop-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-module-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-module-spi-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-rest-domain-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-tuple-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/tomcat-el-api-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-embed-core-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-embed-logging-juli-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-jdbc-7.0.42.jar, file:/var/spring-xd/xd/lib/tomcat-jsp-api-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-juli-7.0.42.jar, file:/var/spring-xd/xd/lib/tomcat-servlet-api-7.0.35.jar, file:/var/spring-xd/xd/lib/uuid-3.2.jar, file:/var/spring-xd/xd/lib/validation-api-1.0.0.GA.jar, file:/var/spring-xd/xd/lib/xmlenc-0.52.jar, file:/var/spring-xd/xd/lib/xmlpull-1.1.3.1.jar, file:/var/spring-xd/xd/lib/xpp3_min-1.1.4c.jar, file:/var/spring-xd/xd/lib/xstream-1.4.4.jar, file:/var/spring-xd/xd/lib/xz-1.0.jar, file:/var/spring-xd/xd/lib/zookeeper-3.4.6.jar, file:/var/spring-xd/xd/lib/hdp20/avro-1.7.5.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-annotations-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-auth-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-distcp-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-hdfs-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-core-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-jobclient-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-shuffle-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-streaming-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-api-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-client-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-server-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-server-nodemanager-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/jetty-util-6.1.26.jar, file:/var/spring-xd/xd/lib/hdp20/protobuf-java-2.5.0.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-2.0.0.RC2-hdp20.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-batch-2.0.0.RC2-hdp20.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-core-2.0.0.RC2-hdp20.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-store-2.0.0.RC2-hdp20.jar]
18:00:07,944  INFO DeploymentsPathChildrenCache-0 logging.AutoConfigurationReportLoggingInitializer:101 - 

Error starting ApplicationContext. To display the auto-configuration report enabled debug logging (start with --debug)


18:00:07,946 ERROR DeploymentsPathChildrenCache-0 cache.PathChildrenCache:550 - 
java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storeWriter': Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanInitializationException: failed to initialize; nested exception is java.net.ConnectException: Call From CentOS/10.0.0.4 to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at org.springframework.xd.dirt.server.ContainerRegistrar.deployStreamModule(ContainerRegistrar.java:448)
    at org.springframework.xd.dirt.server.ContainerRegistrar.onChildAdded(ContainerRegistrar.java:347)

I'm still able to successfuly execute such commands as 

hadoop fs ls /

from xd-shell and get results succesfuly.

I'm aware of Spring-XD API, but would like to stay within XD-Shell for now. Any help would be greatly appreciated.

Andrei Stefan
  • 51,654
  • 6
  • 98
  • 89
JagdCrab
  • 635
  • 2
  • 9
  • 22

3 Answers3

0

It's not able to connect to localhost:8020, i.e. the name node service. Most likely your hadoop installation is not started correctly.

Enno Shioji
  • 26,542
  • 13
  • 70
  • 109
  • Sorry, i forgot to mention. "hadoop fs" commands are still works fine from xd-shell. I added it to the question. – JagdCrab May 06 '14 at 21:12
  • @JagdCrab: Hm that does seem weird. If you do telnet localhost 8020, does it connect then? – Enno Shioji May 06 '14 at 21:17
  • telnet doesn't like it and returns "Unknown host" however if i make http request through browser on 10.0.0.4:8020 (which is internal ip of this machine. It's also the only machine in network) i get proper reply that it's Hadoop IPC port and i should use web-interface ports for http requests. – JagdCrab May 06 '14 at 21:29
  • It's weird that it can't resolve localhost, you prob. want to fix that. In the meanwhile, if you can change the config to point to 10.0.0.4 instead of localhost, it'll prob. at least be able to connect. – Enno Shioji May 06 '14 at 21:46
  • 4
    Open a browser on that machine to http://localhost:50070, that should give you the DFS health screen. On top it should tell you what address the NameNode is configured for. If it is not 'localhost:8020' then you need to modify $XD_HOME/xd/config/servers.yml and set the spring:hadoop:fsUri to hdfs:// replacing with what was on that DFS health screen. There is already a commented out entry in the servers.yml, just modify that. – Thomas Risberg May 07 '14 at 12:30
0

check firewall rules and run some standard hadoop commands.

XD is running Zookeeper which could be blocked

Tim Spann
  • 517
  • 2
  • 6
0
  1. Test with Spring XD Single Node - it's a little easier and no zookeeper
  2. Make sure you have permissions setup in HDFS for access, I've seen that a problem before.
  3. If Hadoop is running in a virtual machine on your desktop and spring xd is on your main machine, sometimes there's IP/DNS issues that will block it.
Tim Spann
  • 517
  • 2
  • 6