How can I handle fixedLengthformat data?

Question

I handle some data with fixed length. So I use bindy component for treating this data.

This file has only one record. The record has header, multi bodies, and footer.

  header record (total length : 20)
  1 : record_type (length : 1)
  VOLTE : service_type (length : 5)
  20190515 : creation date (length : 8)
  3 : custom_flag (length : 6)

  3 body records (total length : 20)
  2 : record_type (length : 1)
  01012345678 : mobile number (length : 11)
  20190515 : call start date (length : 8)

  footer records (total length : 20)
  3 : record_type (length : 1)
  AAAA.DAT : FILE NAME (length : 19)

Real Data

  1VOLTE20190515     32010123456782019051520101234567820190516201012345678201905173AAAA.DAT

I defined dataformat like below.

Header

  @FixedLengthRecord(length=20, paddingChar=' ')
  public class VoLTEHeader {

  @DataField(pos=1, length=1, trim=true)
  String record_type;

  @DataField(pos=2, length=5, trim=true)
  String service_type;

  @DataField(pos=7, length=8, trim=true)
  String creation_date;

  @DataField(pos=15, length=6, trim=true, align="R")
  String custom_flag;

Footer

  @FixedLengthRecord(length=20, paddingChar=' ')
  public class VoLTEFooter {

  @DataField(pos=1, length=1, trim=true)
  String record_type;

  @DataField(pos=2, length=19, trim=true)
  String file_name;

Body

  @FixedLengthRecord(length=20, header=VoLTEHeader.class, footer=VoLTEFooter.class)
  public class VoLTEBody implements Serializable {

  @DataField(pos=1, length=1,trim=true)
  String record_type;

  @DataField(pos=2, length=11,trim=true)
  String mobile_number;

  @DataField(pos=13, length=8,trim=true)
  String call_start_date;

I executed camel route but the exception is occurred.

  java.lang.IllegalArgumentException: Size of the record: 100 is not equal to the value provided in the model: 20
  at org.apache.camel.dataformat.bindy.fixed.BindyFixedLengthDataFormat.createModel(BindyFixedLengthDataFormat.java:295) ~[camel-bindy-2.23.2.jar:2.23.2]
at org.apache.camel.dataformat.bindy.fixed.BindyFixedLengthDataFormat.unmarshal(BindyFixedLengthDataFormat.java:209) ~[camel-bindy-2.23.2.jar:2.23.2]
at org.apache.camel.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:69) ~[camel-core-2.23.2.jar:2.23.2]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.23.2.jar:2.23.2]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.23.2.jar:2.23.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.23.2.jar:2.23.2]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.23.2.jar:2.23.2]

I don't think that fixedLengthDataFormat necessarily needs to be created in multiple lines.

How can I fix this issue?

SebastianBrandt · Answer 1 · 2019-05-15T20:45:23.957

Camel bindy defines a header as "a single header record at the beginning of the file / stream" and a footer as "a single footer record at the end of the file / stream", see the [camel-bindy doc][1].

Your test data has meta data before and after a multi-payload part in the same line, you cannot use Bindy header/footer models to parse that.

Instead create independent Bindy models for header, footer and one body (essentially removing "header=VoLTEHeader.class, footer=VoLTEFooter.class" from VoLTEBody) and process header, footer and bodies individualy:

public class MyRouteBuilderTest extends CamelTestSupport {
    @Produce(uri="direct:start")
    private ProducerTemplate producer;

    @Override
    protected RouteBuilder createRouteBuilder() {
        return new RouteBuilder() {
            public void configure() {
                DataFormat bindyHeader = new BindyFixedLengthDataFormat(VoLTEHeader.class);
                DataFormat bindyFooter = new BindyFixedLengthDataFormat(VoLTEFooter.class);
                DataFormat bindyOneBody = new BindyFixedLengthDataFormat(VoLTEBody.class);

                from("direct:start")
                    // assuming the raw record is in the body, we keep a copy in a Camel-header
                    .setHeader("record", body())

                    // get the VoLTE header string into the Camel-body and unmarshal, then put VoLTE header object into Camel-header for later use
                    .setBody().groovy("request.body.substring(0,20)")
                    .unmarshal(bindyHeader)
                    .setHeader("header", body())

                    // restore VoLTE record string to Camel-body, get the VoLTE footer string into the Camel-body and unmarshal, then put footer VoLTE object into Camel-header for later use
                    .setBody().header("record")
                    .setBody().groovy("request.body.substring(request.body.length()-20,request.body.length())")
                    .unmarshal(bindyFooter)
                    .setHeader("footer", body())

                    // restore VoLTE record string to Camel-body, get the multi-bodies string into the Camel-body
                    .setBody().header("record")
                    .setBody().groovy("request.body.substring(20,request.body.length()-20)")

                    // Split VoLTE bodies string to each 20 char length, unmarshal each and finally put the list of VoLTE body objects into a Camel-header
                    .setBody().groovy("request.body.split('(?<=\\\\G.{20})')")
                    .split(body(), AggregationStrategies.flexible().storeInHeader("bodyList").accumulateInCollection(ArrayList.class))
                        .unmarshal(bindyOneBody)
                    .end()

                    // now do something with the unmarshalled objects in Camel-headers "header" (type VoLTEHeader), "footer" (type VoLTEFooter) and "bodyList" (type List<VoLTEBody>)
                    .log("VoLTEHeader: ${header.header}")
                    .log("VoLTEBody*: ${header.bodyList}")
                    .log("VoLTEFooter: ${header.footer}")
                    ;
            }
        };
    }

    @Test
    public void test() throws Exception {
        producer.sendBody("1VOLTE20190515     32010123456782019051520101234567820190516201012345678201905173AAAA.DAT           ");
    }
}

@FixedLengthRecord(length = 20, paddingChar = ' ')
public class VoLTEHeader {
    @DataField(pos = 1, length = 1, trim = true)
    String record_type;

    @DataField(pos = 2, length = 5, trim = true)
    String service_type;

    @DataField(pos = 7, length = 8, trim = true)
    String creation_date;

    @DataField(pos = 15, length = 6, trim = true, align = "R")
    String custom_flag;

    @Override
    public String toString() {
        return String.format("VoLTEHeader[record_type=%s, service_type=%s, creation_date=%s, custom_flag=%s]", record_type, service_type, creation_date, custom_flag);
    }
}

@FixedLengthRecord(length = 20)
public class VoLTEBody {
    @DataField(pos = 1, length = 1, trim = true)
    String record_type;

    @DataField(pos = 2, length = 11, trim = true)
    String mobile_number;

    @DataField(pos = 13, length = 8, trim = true)
    String call_start_date;

    @Override
    public String toString() {
        return String.format("VoLTEBody[record_type=%s, mobile_number=%s, call_start_date=%s]", record_type, mobile_number, call_start_date);
    }
}

@FixedLengthRecord(length = 20, paddingChar = ' ')
public class VoLTEFooter {
    @DataField(pos = 1, length = 1, trim = true)
    String record_type;

    @DataField(pos = 2, length = 19, trim = true)
    String file_name;

    @Override
    public String toString() {
        return String.format("VoLTEFooter[record_type=%s, file_name=%s]", record_type, file_name);
    }
}

Output:

[main] INFO route1 - VoLTEHeader: VoLTEHeader[record_type=1, service_type=VOLTE, creation_date=20190515, custom_flag=3]
[main] INFO route1 - VoLTEBody*: [VoLTEBody[record_type=2, mobile_number=01012345678, call_start_date=20190515], VoLTEBody[record_type=2, mobile_number=01012345678, call_start_date=20190516], VoLTEBody[record_type=2, mobile_number=01012345678, call_start_date=20190517]]
[main] INFO route1 - VoLTEFooter: VoLTEFooter[record_type=3, file_name=AAAA.DAT           ]

At the end of the route you should have an object of type VoLTEHeader in the Camel-header "header", an object of type VoLTEFooter in Camel-header "footer" and a List of VoLTEBody in Camel-header "bodyList". You could do your processing with these now.

Thank you for your advice. But I cannot define VOLTERecord class because this data can have multiple body records. So I cannot fix the position. — 이성규, May 15 '19 at 11:40
I see. You could split the record in three parts first using String#substring method: header (chars 1..20), footer (length-20..length) and bodies (21..length-20), see https://stackoverflow.com/questions/29636600/camel-substring-operation-n-characters-from-end-of-message-body You could then parse header and footer using their own bindy model. The bodies can be split using a custom Camel splitter cutting the bodies string at every 20th character and each body can be parsed independently using Bindy. The logic is more complex but it seems doable. — SebastianBrandt, May 15 '19 at 12:16
I don't understand your guidence. Please provide example about my scenario that the record has a header, a footer, and three body. — 이성규, May 15 '19 at 13:49

이성규 · Accepted Answer · 2019-05-16T06:34:10.457

Thank you. I implemented it though your guidance. I think that Bindy component support header, footer and body.

So, I considered below logic.

Incoming file -> split (20 length) -> aggregating split file -> unmarshalling

I use aggregated file with header, footer and bodies.

Below is code.

  from("direct:start")
                    // assuming the raw record is in the body, we keep a copy in a Camel-header
                    .setHeader("record", body())
                    .setBody().groovy("request.body.split('(?<=\\\\G.{20})')")
                    .split(body(), AggregationStrategies.flexible().storeInHeader("bodyList").accumulateInCollection(ArrayList.class))
                    .unmarshal(bindyOneBody)
                    .end()
                    .log("VoLTEBody*: ${header.bodyList}")
                   ...

VoLTEBody

       @FixedLengthRecord(length=20, header=VoLTEHeader.class, footer=VoLTETailer.class)
       public class VoLTEBody {

       @DataField(pos=1, length=1,trim=true)
       String record_type;

But there are errors like below.

       Stacktrace
       ------------------------------------------------------------------------------ 
       --------------------------------------------------------- 
       java.lang.IllegalArgumentException: No records have been defined in the file
               at org.apache.camel.dataformat.bindy.fixed.BindyFixedLengthDataFormat.unmarshal(BindyFixedLengthDataFormat.java:250) ~[camel-bindy-2.23.2.jar:2.23.2]
               at org.apache.camel.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:69) ~[camel-core-2.23.2.jar:2.23.2]
               at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.23.2.jar:2.23.2]
               at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.23.2.jar:2.23.2]
               at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.23.2.jar:2.23.2]

I think that this process is fine if annotation about header and footer is correctly defined in VoLTEBody class.

How can I handle this issue?

Additionally, I executed another test. In this case, there is no split.

Incoming file -> unmarshalling (bindyOneBody)

Router

    from("direct:start")
                    // assuming the raw record is in the body, we keep a copy in a Camel-header
                    .setHeader("record", body())
                    //.setBody().groovy("request.body.split('(?<=\\\\G.{20})')")
                    //.split(body(), AggregationStrategies.flexible().storeInHeader("bodyList").accumulateInCollection(ArrayList.class))
                    .unmarshal(bindyOneBody)

There is below result. In this result, we cannot find VoLTEHeader and VoLTEFooter structures. This is fine.

     2019-05-16 15:22:15,798 DEBUG BindyFixedLengthDataFormat - Graph of objects created: {camel.dataformat.volte.VoLTEBody_sample=VoLTEBody[record_type=2, mobile_number=01012345678, call_start_date=20190517]} 
     2019-05-16 15:22:15,798 INFO  route1 - [VoLTEBody[record_type=2, mobile_number=01012345678, call_start_date=20190515], VoLTEBody[record_type=2, mobile_number=01012345678, call_start_date=20190516], VoLTEBody[record_type=2, mobile_number=01012345678, call_start_date=20190517]] 
     2019-05-16 15:22:15,798 INFO  MyRouteBuilderTest - ********************************************************************************

Have you tried the implementation of my answer? I've updated it to a unit test, so you can directly execute it. It does not throw exceptions and produces the right results. — SebastianBrandt, May 16 '19 at 06:58
I implemented it through your answer. It is fine. Additionally, I am considering matter of my comment by my answer comment. Incoming file -> split (20 length) -> aggregating split file -> unmarshalling I have a problem in this case. Please consider this scenario. — 이성규, May 16 '19 at 07:02
How about sharing your solution so other readers could benefit from it? — SebastianBrandt, May 21 '19 at 07:52

How can I handle fixedLengthformat data?

2 Answers2