3

I am trying to get a single file with .zip compression from a ftp server and trying to store it in S3 with .gzip compression using camel. Following is the route I currently have.

from("sftp://username@host/file_path/?password=<password>&noop=true&streamDownload=true")
    .routeId("route_id")
    .setExchangePattern(ExchangePattern.InOut)
    .unmarshal().zipFile()
    .marshal().gzip()
    .to("aws-s3://s3_bucket_name?amazonS3Client=#client");

This works fine for smaller files. But I have files that are ~700 MB in size when compressed. For files of that size I get OutOfMemoryError for Java heap space
I know there is a streaming option in camel (.split(body().tokenize("\n")).streaming()) but I am not sure if I can umarshal and marshal while streaming. (I see a similar solution here but in this case the source file is plain text / csv).
The second part to the problem is streaming the file back to S3. I am aware of the multiPartUpload option in camel-aws component but it seems to require the source to be a file. I do not know how to achieve that.

Can this be achieved without processing (unzipping and then gzipping) the file using java code in a custom processor ?

Environment: Camel 2.19.3, Java 8

Thanks

J. Doe
  • 33
  • 5

1 Answers1

1

I solved it using streamCaching(). So the way I would do that is

from('xyz')
.streamCaching()
.unmarshall().gzip()
.to('abc')
.end()
Eric Aya
  • 69,473
  • 35
  • 181
  • 253
log N
  • 925
  • 9
  • 33