I use spring integration int-ws:outbound-gateway to read files(pdf/...) from a webservice with XOP/MTOM
It works and correctly get the stream from the attachment and I can write them to disk.
But S.I.spring-ws keeps the files in memory while transferring: getting 30 files of 60 MB together fills up the memory and fails (even if in the calling method i read the Stream with a bufferedStream 128kB at a time).
I'd like to be able to transfer N files with " N * fileSize > availableMemory ", streaming littler chunks.
The xml (and response object) contains other information like title and so on, and the "file" is an InputStream from the DataHandler.
EDIT: It was not Spring integration's fault. It happens also with a standalone spring-ws client.
my beans:
<bean id="mtomMarshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
<property name="contextPath" value="aaaaa.bb.filestreamer" />
<property name="mtomEnabled" value="true" />
</bean>
My S.I config:
<int:logging-channel-adapter id="logger" level="DEBUG" log-full-message="true"/>
<!-- All my channels are like this: -->
<int:channel id="singleFileRespChannel"><int:interceptors><int:wire-tap channel="logger"/></int:interceptors></int:channel>
<int:gateway id="filewsEntry"
service-interface="aaaaa.ws.FileServiceGateway"
default-reply-timeout="5000" >
<int:method name="getSingleFile" request-channel="singleFileReqChannel" reply-channel="singleFileRespChannel" />
</int:gateway>
<int-ws:outbound-gateway id="singleFileMarshallingGateway"
request-channel="singleFileReqChannel" reply-channel="singleFileRespChannel"
requires-reply="true"
uri="${filews.singleFile.wsURI}" marshaller="mtomMarshaller"
unmarshaller="mtomMarshaller" >
</int-ws:outbound-gateway>
I tried also with axiom:
<bean id="axiomMessageFactory" class="org.springframework.ws.soap.axiom.AxiomSoapMessageFactory">
<property name="payloadCaching" value="false" />
<property name="attachmentCaching" value="false" />
<property name="attachmentCacheThreshold" value="1024"/>
<property name="attachmentCacheDir" value="D:/tmp/cache"></property>
</bean>
But , apart caching the files on disk, it's the same: the JVM always uses the same memory (heap) amount.
I thought at a S-I. splitter, that would return the chunks in a map... but to to do this, it would have to read all the file anyway.
Can the result be "streamed" through spring integration channels to the caller without load it all in memory?
Any idea about how can i accomplish this?
Edit: adding requested info: in a Junit @Test i call the service in a Runnable.run() (starting 30 threads) to simulate concurrent requests by the clients. Memory used grows up to the maximum (2gb) meaning it's getting all the files in memory:
Here the caller the part of the code:
try{
FileResponse sfresp = service.getSingleFile(sfreq); // CALL S.I. GW
logger.warn("SINGLEFILE STREAM: "+ sfreq.getFile().getFileId());
InputStream in=new BufferedInputStream( sfresp.getFileAttachment().getFile().getInputStream());
logger.warn("writing the stream");
byte[] buffer = new byte[1024];
int bytesRead;
try{
bytesRead = in.read(buffer);
logger.warn("just read "+bytesRead+" Bytes from file : " + new String(buffer,"ASCII"));
}catch (IOException e) {
logger.error("FILE I/O ERROR"+e.getLocalizedMessage());
}finally{
in.close();
}
}catch....etc etc
First results appears after a lot of time (30 seconds) so it seems that the endpoint gets all the file before passing the stream back to the caller.
EDIT
I debugged a while i can't understand why it get all the 60-Megs file instead of just the few bytes i requested and it does it even before i start to read the stream:
2015-09-28 11:20:32,725|WebServiceTemplate|Sent request [AxiomSoapMessage]
2015-09-28 11:20:32,741|OMOutputFormat|Start getContentType: OMOutputFormat [ mimeBoundary =null rootContentId=null doOptimize=false doingSWA=false isSOAP11=true charSetEncoding=UTF-8 xmlVersion=null contentType=null ignoreXmlDeclaration=false autoCloseWriter=false actionProperty=null optimizedThreshold=0]
2015-09-28 11:20:32,741|OMOutputFormat|getContentType= {text/xml} OMOutputFormat [ mimeBoundary =null rootContentId=null doOptimize=false doingSWA=false isSOAP11=true charSetEncoding=UTF-8 xmlVersion=null contentType=text/xml ignoreXmlDeclaration=false autoCloseWriter=false actionProperty=null optimizedThreshold=0]
2015-09-28 11:20:32,741|MTOMXMLStreamWriter|Creating MTOMXMLStreamWriter
2015-09-28 11:20:32,741|MTOMXMLStreamWriter|OutputStream =class org.springframework.ws.transport.AbstractSenderConnection$RequestTransportOutputStream
2015-09-28 11:20:32,741|MTOMXMLStreamWriter|OMFormat = OMOutputFormat [ mimeBoundary =null rootContentId=null doOptimize=false doingSWA=false isSOAP11=true charSetEncoding=UTF-8 xmlVersion=null contentType=text/xml ignoreXmlDeclaration=false autoCloseWriter=false actionProperty=null optimizedThreshold=0]
2015-09-28 11:20:32,741|MTOMXMLStreamWriter|preserveAttachments = false
2015-09-28 11:20:32,741|StAXUtils|XMLStreamWriter is org.apache.axiom.util.stax.dialect.Woodstox4StreamWriterWrapper
2015-09-28 11:20:32,741|OMDataSourceExtBase|serialize xmlWriter=org.apache.axiom.om.impl.MTOMXMLStreamWriter@3799c784
2015-09-28 11:20:32,741|MTOMXMLStreamWriter|Returning access to the original output stream: org.springframework.ws.transport.AbstractSenderConnection$RequestTransportOutputStream@50f9c7a0
2015-09-28 11:20:32,741|MTOMXMLStreamWriter|Calling MTOMXMLStreamWriter.flush
2015-09-28 11:20:32,788|OMDataSourceExtBase|serialize OutputStream optimisation: true
2015-09-28 11:20:32,788|OMDataSourceExtBase|serialize output=org.springframework.ws.transport.AbstractSenderConnection$RequestTransportOutputStream@50f9c7a0 format=OMOutputFormat [ mimeBoundary =null rootContentId=null doOptimize=false doingSWA=false isSOAP11=true charSetEncoding=UTF-8 xmlVersion=null contentType=null ignoreXmlDeclaration=false autoCloseWriter=false actionProperty=null optimizedThreshold=0]
2015-09-28 11:20:32,788|ByteArrayDataSource|getXMLBytes encoding=UTF-8
2015-09-28 11:20:32,788|SOAPEnvelopeImpl|Could not close builder or parser due to:
2015-09-28 11:20:32,788|SOAPEnvelopeImpl|builder is null
2015-09-28 11:20:32,788|MTOMXMLStreamWriter|Calling MTOMXMLStreamWriter.flush
2015-09-28 11:20:32,788|MTOMXMLStreamWriter|close
*****
HERE IT STOPS WHILE TRANSFERRING THE DOC OVER NETWORK
*****
2015-09-28 11:20:38,322|MIMEMessage|Attachments contentLength=0, contentTypeString=Multipart/Related; start-info="text/xml"; type="application/xop+xml"; boundary="----=_Part_434_1482047736.1443432076967"
2015-09-28 11:20:38,342|MIMEMessage|getRootPartContentID rootContentID=null
2015-09-28 11:20:38,342|MIMEMessage|readHeaders
2015-09-28 11:20:38,342|MIMEMessage|addHeader: (Content-Type) value=(application/xop+xml; charset=utf-8; type="text/xml")
2015-09-28 11:20:38,342|PartImpl|getHeader name=(content-id) value=(null)
2015-09-28 11:20:38,358|MIMEMessage|getRootPartContentID rootContentID=null
2015-09-28 11:20:38,358|PartImpl|Using blob of type org.apache.axiom.blob.MemoryBlobImpl
2015-09-28 11:20:38,358|PartImpl|getHeader name=(Content-Transfer-Encoding) value=(null)
2015-09-28 11:20:38,358|DebugInputStream|EOF reached after reading 516 bytes in 1 chunks
2015-09-28 11:20:38,358|MIMEMessage|getRootPartContentID rootContentID=null
2015-09-28 11:20:38,358|PartImpl|getHeader name=(content-type) value=(application/xop+xml; charset=utf-8; type="text/xml")
2015-09-28 11:20:38,389|DefaultOMMetaFactoryLocator|Starting class path based discovery
2015-09-28 11:20:38,389|ImplementationFactory|Loading jar:file:/C:/Users/Max/.m2/repository/org/apache/ws/commons/axiom/axiom-impl/1.2.15/axiom-impl-1.2.15.jar!/META-INF/axiom.xml
2015-09-28 11:20:38,389|ImplementationFactory|Discovered implementations: [llom(metaFactory=org.apache.axiom.om.impl.llom.factory.OMLinkedListMetaFactory,features=[default(priority=100)])]
2015-09-28 11:20:38,405|PriorityBasedOMMetaFactoryLocator|Meta factories:
default: org.apache.axiom.om.impl.llom.factory.OMLinkedListMetaFactory
2015-09-28 11:20:38,405|StAXSOAPModelBuilder|Starting to process SOAP 1.1 message
2015-09-28 11:20:38,405|WebServiceTemplate|Received response [AxiomSoapMessage] for request [AxiomSoapMessage]
2015-09-28 11:20:38,405|PullSerializer|Pull serializer created; initial state is org.apache.axiom.om.impl.common.serializer.pull.Navigator@41ee0498[cache=false,document=false]
2015-09-28 11:20:38,421|StAXBuilder|Caching disabled; current element level is 3
2015-09-28 11:20:38,421|Navigator|Switching to pull-through mode; first event is START_ELEMENT; depth is 1
2015-09-28 11:20:38,421|PullSerializer|Switching to state org.apache.axiom.om.impl.common.serializer.pull.PullThroughWrapper@35c7d3d3[reader=org.apache.axiom.util.stax.xop.XOPDecodingStreamReader@5223dd3a]
2015-09-28 11:20:38,421|XOPDecodingStreamReader|processXopInclude - found href : cid:e8e49803-f357-4757-bcde-6c99abb48cf4%40ws.xxxx.com
2015-09-28 11:20:38,421|XOPDecodingStreamReader|processXopInclude - decoded contentID : e8e49803-f357-4757-bcde-6c99abb48cf4@ws.xxxx.com
2015-09-28 11:20:38,421|XOPDecodingStreamReader|Encountered xop:Include for content ID 'e8e49803-f357-4757-bcde-6c99abb48cf4@ws.xxxx.com'
2015-09-28 11:20:38,421|MIMEMessage|readHeaders
2015-09-28 11:20:38,421|MIMEMessage|addHeader: (Content-Type) value=(application/octet-stream)
2015-09-28 11:20:38,421|MIMEMessage|addHeader: (Content-ID) value=(<e8e49803-f357-4757-bcde-6c99abb48cf4@ws.xxxx.com>)
2015-09-28 11:20:38,421|MIMEMessage|addHeader: (Content-Transfer-Encoding) value=(binary)
2015-09-28 11:20:38,421|PartImpl|getHeader name=(content-id) value=(<e8e49803-f357-4757-bcde-6c99abb48cf4@ws.xxxx.com>)
2015-09-28 11:20:38,421|PartImpl|Using blob of type org.apache.axiom.blob.MemoryBlobImpl
2015-09-28 11:20:38,421|PartImpl|getHeader name=(Content-Transfer-Encoding) value=(binary)
2015-09-28 11:20:38,499|DebugInputStream|EOF reached after reading 64541873 bytes in 15938 chunks
2015-09-28 11:20:39,499|PullSerializer|Restoring state org.apache.axiom.om.impl.common.serializer.pull.Navigator@41ee0498[cache=false,document=false]
2015-09-28 11:20:39,499|StAXBuilder|Caching re-enabled; new element level: 2; done=false
2015-09-28 11:20:39,500|PullSerializer|Switching to state org.apache.axiom.om.impl.common.serializer.pull.EndDocumentState@1e229be
2015-09-28 11:20:39,501|AbstractReplyProducingMessageHandler|handler 'org.springframework.integration.ws.MarshallingWebServiceOutboundGateway#1' sending reply Message: [Payload=xxxx.ws.filestreamer.FileResponse@306a12cf][Headers={timestamp=1443432039501, id=d9ffd3d4-48f3-504e-00a1-9ccd5d1d56e8, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@15160f3c, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@15160f3c, useStub=false}]
2015-09-28 11:20:39,501|AbstractMessageChannel$ChannelInterceptorList|preSend on channel 'singleFileRespChannel', message: [Payload=xxxx.ws.filestreamer.FileResponse@306a12cf][Headers={timestamp=1443432039501, id=d9ffd3d4-48f3-504e-00a1-9ccd5d1d56e8, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@15160f3c, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@15160f3c, useStub=false}]
2015-09-28 11:20:39,501|AbstractMessageHandler|org.springframework.integration.handler.BridgeHandler@3f8ecde received message: [Payload=xxxx.ws.filestreamer.FileResponse@306a12cf][Headers={timestamp=1443432039501, id=d9ffd3d4-48f3-504e-00a1-9ccd5d1d56e8, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@15160f3c, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@15160f3c, useStub=false}]
2015-09-28 11:20:39,501|AbstractReplyProducingMessageHandler|handler 'org.springframework.integration.handler.BridgeHandler@3f8ecde' sending reply Message: [Payload=xxxx.ws.filestreamer.FileResponse@306a12cf][Headers={timestamp=1443432039501, id=d9ffd3d4-48f3-504e-00a1-9ccd5d1d56e8, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@15160f3c, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@15160f3c, useStub=false}]
2015-09-28 11:20:39,501|AbstractMessageChannel$ChannelInterceptorList|postSend (sent=true) on channel 'singleFileRespChannel', message: [Payload=xxxx.ws.filestreamer.FileResponse@306a12cf][Headers={timestamp=1443432039501, id=d9ffd3d4-48f3-504e-00a1-9ccd5d1d56e8, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@15160f3c, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@15160f3c, useStub=false}]
2015-09-28 11:20:39,501|AbstractMessageChannel$ChannelInterceptorList|postSend (sent=true) on channel 'singleFileReqChannel', message: [Payload=xxxx.ws.filestreamer.FileRequest@1f36f64][Headers={timestamp=1443432032632, id=0ce05572-05f6-5e5e-e819-63377ba4e0b1, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@15160f3c, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@15160f3c}]
2015-09-28 11:20:39,501|IntegrationObjectSupport|Unable to attempt conversion of Message payload types. Component 'filewsEntry' has no explicit ConversionService reference, and there is no 'integrationConversionService' bean within the context.
****
HERE THE SERVICE GATEWAY RETURNS THE CONTROL TO MY JAVA CODE
****
2015-09-28 11:20:39,501|TestFileWsInvocation|SINGLEFILE STREAM: "myFile"
2015-09-28 11:20:39,501|TestFileWsInvocation|writing the stream
2015-09-28 11:20:39,501|TestFileWsInvocation|just read 1024 Bytes from file : "1234567.......... "
USING SAAJ is the same:
2015-09-28 11:52:28,760|WebServiceAccessor|Opening [org.springframework.ws.transport.http.HttpUrlConnection@71a8da44] to [http://xxxx:8080/filestreamer/ws/]
2015-09-28 11:52:28,791|AbstractHeaderMapper|headerName=[useStub] WILL NOT be mapped
2015-09-28 11:52:28,807|WebServiceTemplate|Sent request [SaajSoapMessage {http://ws.xxxx.com/filestreamer}fileRequest]
******
STOPS WHILE GETTING ALL THE FILE
******
2015-09-28 11:52:34,948|WebServiceTemplate|Received response [SaajSoapMessage {http://ws.xxxx.com/filestreamer}fileResponse] for request [SaajSoapMessage {http://ws.xxxx.com/filestreamer}fileRequest]
2015-09-28 11:52:35,371|AbstractReplyProducingMessageHandler|handler 'org.springframework.integration.ws.MarshallingWebServiceOutboundGateway#1' sending reply Message: [Payload=com.xxxx.ws.filestreamer.FileResponse@797fc155][Headers={timestamp=1443433955371, id=0a41ab57-dac8-6f7e-3ad9-c71a745f5b0a, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, useStub=false}]
2015-09-28 11:52:35,371|AbstractMessageChannel$ChannelInterceptorList|preSend on channel 'singleFileRespChannel', message: [Payload=com.xxxx.ws.filestreamer.FileResponse@797fc155][Headers={timestamp=1443433955371, id=0a41ab57-dac8-6f7e-3ad9-c71a745f5b0a, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, useStub=false}]
2015-09-28 11:52:35,371|AbstractMessageHandler|org.springframework.integration.handler.BridgeHandler@326df1c4 received message: [Payload=com.xxxx.ws.filestreamer.FileResponse@797fc155][Headers={timestamp=1443433955371, id=0a41ab57-dac8-6f7e-3ad9-c71a745f5b0a, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, useStub=false}]
2015-09-28 11:52:35,371|AbstractReplyProducingMessageHandler|handler 'org.springframework.integration.handler.BridgeHandler@326df1c4' sending reply Message: [Payload=com.xxxx.ws.filestreamer.FileResponse@797fc155][Headers={timestamp=1443433955371, id=0a41ab57-dac8-6f7e-3ad9-c71a745f5b0a, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, useStub=false}]
2015-09-28 11:52:35,371|AbstractMessageChannel$ChannelInterceptorList|postSend (sent=true) on channel 'singleFileRespChannel', message: [Payload=com.xxxx.ws.filestreamer.FileResponse@797fc155][Headers={timestamp=1443433955371, id=0a41ab57-dac8-6f7e-3ad9-c71a745f5b0a, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, useStub=false}]
2015-09-28 11:52:35,371|AbstractMessageChannel$ChannelInterceptorList|postSend (sent=true) on channel 'singleFileWSReqChannel', message: [Payload=com.xxxx.ws.filestreamer.FileRequest@41216889][Headers={timestamp=1443433948760, id=42d187e8-c0c6-4077-1000-8840e2c531ba, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, useStub=false}]
2015-09-28 11:52:35,371|AbstractMessageChannel$ChannelInterceptorList|postSend (sent=true) on channel 'singleFileRouterChannel', message: [Payload=com.xxxx.ws.filestreamer.FileRequest@41216889][Headers={timestamp=1443433948760, id=42d187e8-c0c6-4077-1000-8840e2c531ba, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, useStub=false}]
2015-09-28 11:52:35,371|AbstractMessageChannel$ChannelInterceptorList|postSend (sent=true) on channel 'singleFileReqChannel', message: [Payload=com.xxxx.ws.filestreamer.FileRequest@41216889][Headers={timestamp=1443433948760, id=6ebcc8b3-0db4-1e67-9619-98716ff362ff, errorChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6, replyChannel=org.springframework.integration.core.MessagingTemplate$TemporaryReplyChannel@78561bc6}]
2015-09-28 11:52:35,371|IntegrationObjectSupport|Unable to attempt conversion of Message payload types. Component 'filewsEntry' has no explicit ConversionService reference, and there is no 'integrationConversionService' bean within the context.
2015-09-28 11:52:35,371|TestFileWsInvocation|SINGLEFILE STREAM: 0
2015-09-28 11:52:35,371|TestFileWsInvocation|writing the stream
2015-09-28 11:52:35,371|TestFileWsInvocation|just read 1024 Bytes from file : "12345........"
