Spring XD: pipe (>) from file source to batch job fails (IllegalArgumentException: Unable to convert provided JSON to Map)

Question

I am trying to pipe the output (file contents) of a Spring XD file source to a simple batch job. This fails with the following exception. It appears that XD is trying to incorrectly use the file contents as JSON job parameters when launching my job. As you may notice, the file I am using to test is a .gitignore.

02:01:11,568 1.1.0.RC1  INFO DeploymentsPathChildrenCache-0 module.ModuleDeployer - Deployed ResourceConfiguredModule [name=file, type=source, group=mystream1, index=0 @5c617835]
02:01:11,571 1.1.0.RC1 DEBUG task-scheduler-9 support.DefaultListableBeanFactory - Returning cached instance of singleton bean 'integrationEvaluationContext'
02:01:11,573 1.1.0.RC1 DEBUG task-scheduler-9 support.DefaultListableBeanFactory - Returning cached instance of singleton bean 'output'
02:01:11,575 1.1.0.RC1  INFO DeploymentSupervisor-0 server.StreamDeploymentListener - Deployment status for stream 'mystream1': DeploymentStatus{state=deployed}
02:01:11,577 1.1.0.RC1  INFO DeploymentSupervisor-0 server.StreamDeploymentListener - Stream Stream{name='mystream1'} deployment attempt complete
02:01:11,577 1.1.0.RC1 DEBUG task-scheduler-9 local.LocalMessageBus$7 - outbound.job:testjob1 received message: GenericMessage [payload=C:\Users\me\Desktop\testfiles\.gitignore, headers={timestamp=1423465271577, id=de52e134-2dbc-454f-b27d-b032e89e6254, contentType=text/plain}]
02:01:11,578 1.1.0.RC1 DEBUG task-scheduler-1 local.LocalMessageBus$7 - inbound.job:testjob1 received message: GenericMessage [payload=C:\Users\me\Desktop\testfiles\.gitignore, headers={timestamp=1423465271577, id=de52e134-2dbc-454f-b27d-b032e89e6254, contentType=text/plain}]
02:01:11,581 1.1.0.RC1 DEBUG task-scheduler-1 job.JobLaunchRequestTransformer - JobParameters are provided as 'String'. Convertering to Spring Batch JobParameters...
02:01:11,584 1.1.0.RC1 DEBUG task-scheduler-1 support.DefaultListableBeanFactory - Returning cached instance of singleton bean 'errorChannel'
02:01:11,588 1.1.0.RC1 ERROR task-scheduler-1 handler.LoggingHandler - org.springframework.integration.transformer.MessageTransformationException: ; nested exception is org.springframework.messaging.MessageHandlingException: ; nested exception is java.lang.IllegalArgumentException: Unable to convert provided JSON to Map<String, Object>
    at org.springframework.integration.transformer.MessageTransformingHandler.handleRequestMessage(MessageTransformingHandler.java:74)
    at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:99)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.integration.monitor.SimpleMessageHandlerMetrics.handleMessage(SimpleMessageHandlerMetrics.java:107)
    at org.springframework.integration.monitor.SimpleMessageHandlerMetrics.invoke(SimpleMessageHandlerMetrics.java:87)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
    at com.sun.proxy.$Proxy123.handleMessage(Unknown Source)
    at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
    at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:277)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:239)
    at sun.reflect.GeneratedMethodAccessor77.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.integration.monitor.DirectChannelMetrics.monitorSend(DirectChannelMetrics.java:114)
    at org.springframework.integration.monitor.DirectChannelMetrics.doInvoke(DirectChannelMetrics.java:98)
    at org.springframework.integration.monitor.DirectChannelMetrics.invoke(DirectChannelMetrics.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
    at com.sun.proxy.$Proxy120.send(Unknown Source)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
    at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:95)
    at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:248)
    at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:171)
    at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:119)
    at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:105)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
    at org.springframework.integration.endpoint.PollingConsumer.handleMessage(PollingConsumer.java:74)
    at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:219)
    at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:55)
    at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:149)
    at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:146)
    at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:298)
    at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
    at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
    at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:292)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: org.springframework.messaging.MessageHandlingException: ; nested exception is java.lang.IllegalArgumentException: Unable to convert provided JSON to Map<String, Object>
    at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:78)
    at org.springframework.integration.transformer.AbstractMessageProcessingTransformer.transform(AbstractMessageProcessingTransformer.java:64)
    at org.springframework.integration.transformer.MessageTransformingHandler.handleRequestMessage(MessageTransformingHandler.java:68)
    ... 59 more
Caused by: java.lang.IllegalArgumentException: Unable to convert provided JSON to Map<String, Object>
    at org.springframework.xd.dirt.plugins.job.ExpandedJobParametersConverter.getJobParametersForJsonString(ExpandedJobParametersConverter.java:169)
    at org.springframework.xd.dirt.plugins.job.JobLaunchRequestTransformer.toJobLaunchRequest(JobLaunchRequestTransformer.java:128)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:112)
    at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:129)
    at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
    at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:342)
    at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:88)
    at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:131)
    at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:330)
    at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:164)
    at org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:276)
    at org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:142)
    at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:75)
    ... 61 more
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'C': was expecting ('true', 'false' or 'null')
 at [Source: C:\Users\me\Desktop\testfiles\.gitignore; line: 1, column: 2]
    at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1419)
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:508)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2300)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1459)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:683)
    at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3105)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3051)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2175)
    at org.springframework.xd.dirt.plugins.job.ExpandedJobParametersConverter.getJobParametersForJsonString(ExpandedJobParametersConverter.java:166)
    ... 77 more

My job's config XML looks like this:

<context:component-scan base-package="com.me" use-default-filters="false">
        <context:include-filter type="annotation" expression="org.springframework.context.annotation.Configuration" />
    </context:component-scan>

    <int:channel id="input" />

    <batch:job id="filePollJob" restartable="true">
        <batch:step id="stepOne">
            <batch:tasklet>
                <batch:chunk reader="testReader" writer="testWriter" commit-interval="3"/>
            </batch:tasklet>
        </batch:step>
    </batch:job>

<bean id="testReader" class="com.me.TestItemReader" scope="step">
        <property name="resource" value="#{jobParameters['input.file']}"/>
        <property name="lineMapper" ref="testLineMapper" />
    </bean>

    <bean id="testLineMapper" class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
        <property name="lineTokenizer">
            <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer" />
        </property>
        <property name="fieldSetMapper">
            <bean class="org.springframework.batch.item.file.mapping.PassThroughFieldSetMapper" />
        </property>
    </bean>

    <bean id="testWriter" class="com.me.TestItemWriter" />

My XD shell commands:

xd:>job create myjob --definition "file-batch"
Successfully created job 'myjob'
xd:>job deploy myjob
Deployed job 'myjob'
xd:>stream create mystream --definition "file --dir='C:\Users\me\Desktop\testfiles\' --outputType=text/plain > queue:job:myjob" --deploy
Created and deployed new stream 'mystream'

Thank you for any help you can provide!

score 2 · Accepted Answer · answered Feb 06 '15 at 17:33

2

The file source, by default, sends the contents over the bus; you need to send the file itself, not its contents (ref=true).

This this sample for an example.

answered Feb 06 '15 at 17:33

Gary Russell

166,535
14
146
179

Thanks for your response! I just tried that and I'm getting pretty much the same issue. Now the String passed to ExpandedJobParametersMapConverter.getJobParametersForJsonString() is the path to my file. It is still throwing the exact same exception. I was originally following [this example](https://github.com/spring-projects/spring-xd/wiki/Batch-Jobs#launch-job-as-a-part-of-event-flow) which does not include the ref option. – Reid Harrison Feb 06 '15 at 19:10
1

Hmmm - I just tested it with a local and rabbit transport and it worked ok for me. It should not be a string when it gets to the converter. Turn on DEBUG logging for org.springframework to follow the message flow. By the way, after it gets past the converter, when the parameter ends up at the job, it's no longer a `File`, but an absolute path so instead of `#{jobParameters['input.file']}` you'll need `#{jobParameters['absoluteFilePath']}` (which is a String). – Gary Russell Feb 06 '15 at 21:57
1

or `#{new java.io.File(jobParameters['absolutePath'])}` – Gary Russell Feb 06 '15 at 22:06
I edited my question with debug logs. Thank you for the correction to the job params key. I have now tried both of your suggestions and have gotten the same exception. I'll look more into what's going on to cause the file's path to get passed as the job's parameters. Thanks! – Reid Harrison Feb 09 '15 at 07:38
`JobParameters are provided as 'String'` is very strange. I get `job.JobLaunchRequestTransformer - JobParameters are provided as 'File'. Convertering to Spring Batch JobParameters...`. Can you show your current stream definition? – Gary Russell Feb 09 '15 at 14:07
I've fixed it now. The first problem was that I had _'outputType=text/plain'_ in my stream definition which caused the AbstractMessageChannel to have String as the datatype so that AbstractMessageChannel.convertPayloadIfNecessary() converts the File payload to a String which in turn caused JobLaunchRequestTransformer.toJobLaunchRequest() to route to ExpandedJobParametersConverter.getJobParametersForJsonString() instead of getJobParametersForFile(). After than, I had to change the ItemReader's resource to _#{new org.springframework.core.io.FileSystemResource(jobParameters['absoluteFilePath'])}_ – Reid Harrison Feb 09 '15 at 17:50
With just wrapping the absoluteFilePath in a File, I was getting an exception when Spring was attempting to set that File as the FlatFileItemReader's Resource since File cannot be cast as a Resource. Thanks for all your help, I'll go ahead and accept your answer since this thread led to the solution. – Reid Harrison Feb 09 '15 at 17:53

Spring XD: pipe (>) from file source to batch job fails (IllegalArgumentException: Unable to convert provided JSON to Map)

1 Answers1