1

I am trying to replicate a abinitio job in talend.

I have two starting point(conditional) in the job, and I am using tjava IF Trigger to make any one of the sub-job execute.

Now since, either one of the sub-job would execute, I am willing to put just one tFileInputDelimited and converge both the Subjob data flow to one.

I tried using tUnite, but by doing that, two-jobs are getting combined. And single subjob cannot have two trigger starting point. I am getting error in such case.

once way is to use intermediate file and then again read from it. But this won't be replication of abinitio job. So difficult to convince.

The input data could be very large, so I am not sure if joining of tmap would be a good approach.

Can someone suggest some better approach, please suggest.

enter image description here

enter image description here

NishantM
  • 141
  • 12

2 Answers2

2

If you only need to write to the same file and not process the rows with the same components you could

  1. Set up an OutputStream before calling the sub-job. A pre-job can be handy for that.
  2. Have the tFileOutputDelimeted on each sub-job write to that OutputStream instead of a file.
  3. After processing your input data, close the OutputStream. A post-job can be handy for that.

If you need to have the rows processed by the same logic you could call a common sub-job from each of the distinct sub-jobs you are having already.

use a common sub-job to merge data flows

Kriegel
  • 423
  • 5
  • 16
  • Outputstream willbe memory driven. DO you think we would use the same in case of large file, or say database unload? Basically I am trying to understand idea of flow merging. if Suppose I have teradatainput and MySQLinput in same job, and based on some context variable any one of them would execute, say there schema is same. can I have just one outputfile and merge flow? – NishantM Apr 14 '17 at 04:40
  • The Outputstream can be a FileOutputStream, it's handle will live in memory tho the data will be written to disc. It's not necessary to use a ByteArrayOutputStream which holds all the data in memory. – Kriegel Apr 17 '17 at 08:14
0

You could use the tHash components instead of an intermediary file, provided you can handle everything in memory.

Atari2600
  • 1,219
  • 2
  • 13
  • 26