0

I have a use case where I am receiving random messages (with a JSON payload) via a queue (RabbitMQ) in Camel and I need to transform these JSON payloads into a CSV format to store in a file. What I have is mostly working, but when I try to add in the "generateHeaderColumns=true" into the @CsvRecord annotation on the POJO I have, I get the column headers repeated for each of the "rows" i end up with. As an example, consider the following route in Camel:

from("rabbitmq://localhost:5672/testexchange?routingKey=create")
    .unmarshal().json(JsonLibrary.Jackson, MyModel.class)
    .marshal().bindy(BindyType.Csv, MyModel.class)
    .to("file://c:/ftproot?fileName=export.$simple{date:now:MM.dd.yyyy}.csv&fileExist=Append")

The following is the MyModel.class POJO:

import org.apache.camel.dataformat.bindy.annotation.CsvRecord;
import org.apache.camel.dataformat.bindy.annotation.DataField;

@CsvRecord(separator=",", crlf="UNIX", generateHeaderColumns=true)
public class MyClass
{
    @DataField(pos = 1, columnName="System ID")
    private String identifier;

    @DataField(pos = 2, columnName="Full Name")
    private String name;

    public void setIdentifier(String identifier)
    {
        this.identifier = identifier;
    }

    public String getIdentifier()
    {
        return identifier;
    }

    public void setName(String name)
    {
        this.name = name;
    }

    public String getName()
    {
        return name;
    } 
}

However, what I end up with if I send multiple JSON payloads to the queue is the following:

System ID,Full Name
208557,Linus Van Pelt
System ID,Full Name
273053,Charlie Brown
System ID,Full Name
206541,Lucy Van Pelt
System ID,Full Name
279876,Sally Brown

When I look at the code for Bindy, it sort of makes sense what its doing, as the unmarshal step doesn't know that the final result is a file... but how do I get the resulting file to have the CSV record unmarshalled, but only have one "row" of column headers? Is there some other way to do this? In other words, how do I use the queue to receive random messages but have the file end up like this instead?

System ID,Full Name
208557,Linus Van Pelt
273053,Charlie Brown
206541,Lucy Van Pelt
279876,Sally Brown
jfoster67
  • 23
  • 4
  • Does the JSON message contain a single line in the CSV or does it contain an array? – Ralf Feb 19 '14 at 06:56
  • The JSON message contains a single line, a single record to be added. The idea is that the application sending them is tied to a business process, so a user essentially clicks a 'go' type button which does some other work and then sends the message. I was hoping to collect all of those 'one-offs' throughout the day in a file for nightly processing by other systems. – jfoster67 Feb 19 '14 at 15:39
  • Then you need to provide the context of a list to produce a file. Maybe, if you have enough RAM, the [aggregator](http://camel.apache.org/aggregator) is an option for you. You could aggregate all messages throughout the day in a list and then produce one file at the end of the day. – Ralf Feb 19 '14 at 15:44
  • Ok. So, Bindy is designed to work on collections, so as long as I can get all the JSON messages (or POJOs) into one place then Bindy can just marshal them all together at once, and produce one row for column header? – jfoster67 Feb 19 '14 at 16:14
  • It is not so much about Bindy. It marshals whatever is in the exchange body, if configured correctly. And the exchange contains the data of your user request, which represents one line in your case. If you want the route to maintain state across several user interactions, then you may use an aggregator to collect the inputs and then release them for further processing, e.g. marshaling, based on a criteria like number of requests, time elapsed, etc. – Ralf Feb 21 '14 at 08:54
  • What I ended up doing to solve this is to unmarshal each individual JSON payload into an entity class and use camel jpa to store each in a database table. Then have a separate route that uses camel jpa to query all the rows as entity objects and then use an aggregator to put them all together into a list (using the ArrayListAggregationStrategy on the aggregator2 page). Once the batch completes the aggregation, Bindy then marshals all of that together into the file I need. Thanks for all the help! – jfoster67 Feb 27 '14 at 18:45

0 Answers0