0

Good afternoon! I have a directory with a two identically-structured Excel files (I can also use CSV, and there might be in theory more than two) with about 50,000 rows each. I need to combine all of the files in that directory, sort them by an ID, and then unflatten them for a call to a Web API.

In an answer to a previous question, I saw how to unflatten data from a SQL query, so my question is this: how to I combine and then sort two large Excel (or CSV) files? Thanks!

edit I found http://blogs.mulesoft.org/using-in-memory-database-to-help-with-flat-file-integration/, which I'll be using for the sorting and combining. My only remaining question is how I wait until all of the files have been loaded to continue with the flow?

Community
  • 1
  • 1
Becca Dee
  • 1,530
  • 1
  • 24
  • 51
  • 1
    We need to know a bit more about which OS, and which tools you'd be ready to use. CSV on linux would be very easy: `cat file1 file2 | sort`. – Guntram Blohm Dec 10 '13 at 19:12
  • This will be running under Windows. I'm hoping to use as little outside of Mule as possible. I could use a PowerShell script to combine the files if there isn't a more Mule way of doing in. – Becca Dee Dec 10 '13 at 19:28
  • @GuntramBlohm, can you take a look at the remaining portion of my question and let me know what you think? Thanks! – Becca Dee Dec 11 '13 at 14:47

1 Answers1

0

There is a module at https://github.com/mulesoft/mule-module-requester that allows the use of a global endpoint or other source that would not normally be usable in the middle of a flow (e.g., an inbound file endpoint) in the middle of the flow. If you use a global file endpoint, it will return a NullPayload if there is no file, otherwise it will return the first file. I created a recursive flow that has a choice to see if the payload is null. If it's not, it loads the file, puts it into Derby, and recurses. If it is null, it continues doing whatever I want with the data.

Becca Dee
  • 1,530
  • 1
  • 24
  • 51