Handling Large CSV file using Apache camel

Asked Dec 18 '14 at 23:24

Active May 15 '17 at 12:58

Viewed 764 times

I have already looked into the earlier post on handling large csv file using camel. In the below code I have used bindy to handle my CSV file. Also used streaming and tokenizer to further split the file into group of 2000 records.

Still the server gives OOM error when I run this in Production env, work fine in the development machine when processing the same file.

What am I missing?

<split streaming="true">
    <tokenize token="\n" group="2000"></tokenize>
        <unmarshal ref="bindyDataformat">
            <bindy type="Csv" classType="com.camel.bindy.input.ABC"/>
        </unmarshal>
        <bean ref="csvBean" method="processAbcCsv"/>
        <marshal ref="bindyDataformat">
             <bindy type="Csv" classType="com.camel.bindy.input.ABC"/>
        </marshal>
        <to uri="{{to.file.processed}}"/>
</split>

edited Oct 07 '16 at 22:03

gtonic

2,295
1
24
32

asked Dec 18 '14 at 23:24

Yoogi

How is this route deployed? Do you deploy in Karaf? ServiceMix? I had to tweak my jvm settings for my run time to handle a 250MB csv file to upload. – Namphibian Dec 19 '14 at 00:54
It is deployed as an OSGI bundle, on Jboss 6.0.Karaf console. – Yoogi Dec 19 '14 at 01:00
Which run time container? – Namphibian Dec 19 '14 at 01:58
java version "1.7.0_09-icedtea" OpenJDK Runtime Environment (rhel-2.3.4.1.el6_3-x86_64) OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode) – Yoogi Dec 19 '14 at 01:59
Sorry missed the JBoss part. You will need to edit the Karaf jvm settings this can be done by editting the karaf config files. – Namphibian Dec 19 '14 at 02:04
Can you please provide me with specific file name and path and which params to change – Yoogi Dec 19 '14 at 02:46

Handling Large CSV file using Apache camel

0 Answers0