JDBCHDFS job failed with Filesystem closed exception

Question

We tried following jdbchdfs job in current environment

Our environment is:
spring-xd version :1.2.1
hadoop distribution : PHD3.0

job create rcd_prod --definition "jdbchdfs --driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver --url='jdbc:sqlserver://hostname:49552;database=databasename' --username=User --password=xxxxxx --testOnBorrow=false --delimiter=^ --sql='select * from tablename' --fsUri=hdfs://HOSTNAME-HA --fileExtension=csv --directory=/PATH/TO/DIR/ --fileName=XYZ --partitions=1 --makeUnique=true"

Getting error on execution of jdbchdfs job

Following Error

java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:795)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2005)
    at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1136)
    at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1132)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1132)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1423)
    at org.springframework.xd.batch.item.hadoop.AbstractHdfsItemWriter.initializeCounterIfNecessary(AbstractHdfsItemWriter.java:66)
    at org.springframework.xd.batch.item.hadoop.HdfsTextItemWriter.write(HdfsTextItemWriter.java:60)
    at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
    at com.sun.proxy.$Proxy94.write(Unknown Source)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:175)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:151)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:274)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:199)
    at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:75)
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletSte

What is causing this exception and how can we fix this?

Please, suggest where lacking while submitting question ! so, that it is more readable ! — NEO, Sep 11 '15 at 13:31

Ali · Answer 1 · 2016-01-24T14:44:23.500

0

This problem is usually encountered when you're running multiple jdbchdfs jobs. The thing is core Hadoop library closes the connections for the entire JVM - so any jobs running parallel and have opened HDFS files may close each other's file system connection.

This has been fixed from 1.3.1.RC.

http://github.com/spring-projects/spring-xd/pull/1805

edited Jan 24 '16 at 14:44

answered Oct 10 '15 at 17:03

Ali

69
8

Thanks for reply. We have two parallel job both running at same time using cron. But only it executs one job and other completly failed. Do we have any way so that our both job run parallel without fail.and one more thing i observed if i ran diffrent job so my both jobs fails on first run of cron.Looking forward about how to avoid failure of job on each run. – NEO Oct 13 '15 at 08:00
can you run in distributed mode? try deploying to different containers in that way you will have 2 separate JVMs. – Ali Oct 13 '15 at 12:21
Spring XD team just fixed this - you may see it resolved in the next release. https://github.com/spring-projects/spring-xd/pull/1805 – Ali Oct 14 '15 at 07:13
Thanks for fixed link. I have deployed to diffrent containers using following commands : job deploy hdfsjob1 --properties "module.jdbchdfs.criteria=host.equals('host1.acme.com')" job deploy hdfsjob2 --properties "module.jdbchdfs.criteria=host.equals('host2.acme.com')" – NEO Oct 14 '15 at 08:02
While running multiple parallel job. job failure event is normal in this version. is that possible to have different xd-shell admin-ui account ? so, in that way we may run development and production batch job without interruption ? Any help on it would be much appreciated ? – NEO Oct 16 '15 at 22:06
what do you mean by different xd-shell accounts? how does your setup look like? do you have a common xd-cluster for production/development jobs? if yes, then you may create container groups, and then deploying to the container you can provide deployment properties - similar to the way you gave criteria for host name. This way you will have separate groups for dev and prod and know at deployment time what group are you deploying your job. – Ali Oct 17 '15 at 10:48
We are using spring xd distributed runtime architecture. install via ambari plugin. – NEO Oct 18 '15 at 16:48

JDBCHDFS job failed with Filesystem closed exception

1 Answers1