Read CSV file line by line in WSO2 ESB 5.0 using Smooks

Question

I have a CSV File that I am reading via VFS. After reading the file, I am using Smooks to convert the CSV data into XML and then sending it to another proxy. By default configuration, smooks converts the whole message into one xml payload.

Problem:

The default method is okay for small files, but I have a very large file to process and I want that I can read the file line by line and then send the message to the next component.

Proxy Configuration:

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="ParseTestCSV1"
       startOnLoad="true"
       statistics="disable"
       trace="disable"
       transports="vfs">
   <target>
      <inSequence>
         <log level="full" separator="*********Parsing Prroxy Started*****"/>
         <smooks config-key="gov:/repository/csv/smooks-config.xml">
            <input type="text"/>
            <output type="xml"/>
         </smooks>
         <log level="full" separator="********After Smooks*******"/>
         <property name="OUT_ONLY" value="true"/>
         <property action="remove" name="ClientApiNonBlocking" scope="axis2"/>
         <call/>
         <log level="full" separator="*******Message Sent*********"/>
      </inSequence>
      <outSequence/>
      <faultSequence/>
   </target>
   <parameter name="transport.PollInterval">60</parameter>
   <parameter name="transport.vfs.FileURI">vfs:file://C:\WSO2EnterpriseIntegrator\file\In</parameter>
   <parameter name="transport.vfs.ContentType">text/plain</parameter>
   <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
   <parameter name="transport.vfs.MoveAfterFailure">vfs:file://C:\WSO2EnterpriseIntegrator\file\Fail</parameter>
   <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   <parameter name="transport.vfs.FileNamePattern">convert.*.csv</parameter>
   <parameter name="transport.vfs.MoveAfterProcess">vfs:file://C:\WSO2EnterpriseIntegrator\file\Out</parameter>
   <description/>
</proxy>

Smooks Configuration File:

<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:csv="http://www.milyn.org/xsd/smooks/csv-1.2.xsd">

        <resource-config selector="org.xml.sax.driver">

            <resource>org.milyn.csv.CSVReader</resource>

            <param name="fields">$ignore$,firstname,lastname,age,street,address,statecode,postalcode,amount,code,date</param>

            <param name="rootElementName">record</param>

            <param name="recordElementName">csvRecord</param>

        </resource-config>

    </smooks-resource-list>

How can I transform my Smook's configuration file to read one line and then send it forward. Any insight would be appreciated.

score 1 · Answer 1 · answered Oct 30 '17 at 08:02

I've had a similar case, where I had to split a huge XML into a jms message for each order in the XML. You can achive this by using freemaker templates.

Basically you define your freemarker template and bind it to a router.

Example:

    <?xml version="1.0" encoding="UTF-8"?>
<!--
 ~ Copyright (c) 2005-2010, WSO2 Inc. (http://wso2.com) All Rights Reserved.
 ~
 ~ WSO2 Inc. licenses this file to you under the Apache License,
 ~ Version 2.0 (the "License"); you may not use this file except
 ~ in compliance with the License.
 ~ You may obtain a copy of the License at
 ~
 ~    http://www.apache.org/licenses/LICENSE-2.0
 ~
 ~ Unless required by applicable law or agreed to in writing,
 ~ software distributed under the License is distributed on an
 ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 ~ KIND, either express or implied.  See the License for the
 ~ specific language governing permissions and limitations
 ~ under the License.
 -->
<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
                      xmlns:core="http://www.milyn.org/xsd/smooks/smooks-core-1.3.xsd"
                      xmlns:jb="http://www.milyn.org/xsd/smooks/javabean-1.2.xsd"
                      xmlns:file="http://www.milyn.org/xsd/smooks/file-routing-1.1.xsd"
                      xmlns:ftl="http://www.milyn.org/xsd/smooks/freemarker-1.1.xsd"
                      xmlns:jms="http://www.milyn.org/xsd/smooks/jms-routing-1.2.xsd">


                <core:namespaces>
        <core:namespace prefix="soapenv" uri="http://schemas.xmlsoap.org/soap/envelope/"/>
        </core:namespaces>

    <core:filterSettings type="SAX" />
    <jb:bean beanId="recordsAsXml" class="java.util.Hashtable" createOnElement="records">
     <jb:value data="records/Id" decoder="String" property="Id"></jb:value>
     <jb:value data="records/Account_ID__c" decoder="String" property="accountID"></jb:value> 
     <!-- ...-->
 </jb:bean>

    <ftl:freemarker applyOnElement="records">
        <!--<ftl:template>/repository/resources/smooks/record_as_xml.ftl</ftl:template>-->
        <ftl:template><!--<order>
        <id>${recordsAsXml.Id}</id>
        <accountId>${recordsAsXml.accountID}</accountId>
        <!-- ...-->
   </order>-->
        </ftl:template>
        <ftl:use>
            <ftl:bindTo id="recordAsXmlOutput"/>
        </ftl:use>
    </ftl:freemarker>


    <jms:router routeOnElement="records" beanId="recordAsXmlOutput" destination="MyJMSQueue">
        <jms:message>
        </jms:message>
        <jms:jndi properties="/repository/resources/smooks/activemq.sr.jndi.properties" />
        <jms:highWaterMark mark="-1"/> 
    </jms:router>
</smooks-resource-list>

The following article might also help.

Message processing with smooks

Read CSV file line by line in WSO2 ESB 5.0 using Smooks

1 Answers1