Split and tokenize a large file with regex using Apache Camel

Asked May 05 '23 at 17:50

Active May 08 '23 at 17:42

Viewed 82 times

I am trying to read a very large flat file of around 1 GB using Camel. The records are separated by a set of strings, and I am trying the below route.

from(incomingFilePath+"?delete=true")
                .split()
                .tokenize("ENDRECORD")
                .streaming()
                .bean(bean, "method")
                .bean(bean,"method")
                .end();

The streaming works in the above case. But when I try to tokenize using a regex, my streaming fails and I am getting an Out of Memory Exception.

from(incomingFilePath+"?delete=true")
                .split()
                .tokenize("\\|END(\\r\\n|\\r|\\n)", true)
                .streaming()
                .bean(bean, "method")
                .bean(bean,"method")
                .end();

java.lang.OutOfMemoryError: Java heap space

Some references:

edited May 08 '23 at 17:42

asked May 05 '23 at 17:50

Halley

Showing the value of `RECORD_SEPARATOR_REGEX` might be helpful... – Jim Garrison May 05 '23 at 17:52
@JimGarrison added. Will that matter? – Halley May 05 '23 at 19:11
Did you try the regex manually on a small snippet? Do you really want to split by "|END" followed by any kind of newline? (CRLF, CR or LF) Just asking cause of the first `|`, but may be totally legit. – cyberbrain May 05 '23 at 20:02
@Halley Yes, if it's particularly complex. – Jim Garrison May 05 '23 at 21:52
@cyberbrain doesn't matter. The moment I use a regex, the streaming stops working. – Halley May 06 '23 at 20:31

Split and tokenize a large file with regex using Apache Camel

0 Answers0