-1

I have one use case to create incremental Data ingestion pipeline from one Database to AWS S3. I have created a pipeline and it is working fine except for the one scenario where no incremental data was found.

In case of zero record count, it is writing the file with a header-only (parquet file). I want to skip the target write when there is no incremental record. How I can implement this in IICS?

I have already tried to implement the router transformation where I have put the condition if record count > 0 then only write to target but still it is not working.

Maciejg
  • 3,088
  • 1
  • 17
  • 30
Nikhil Suthar
  • 2,289
  • 1
  • 6
  • 24

1 Answers1

0

First of all: the target file gets created even before any data is read from source. This is to ensure the process has write access to target location. So even if there will be no data to store, an empty file will get created.

The possible ways out here will be to:

  1. Have a command task check the number of lines in output file and delete it if there is just a header. This would require the file to be created locally, verified, and uploaded to S3 afterwards e.g. using Mass Ingestion task - all invoked sequentially via taskflow
  2. Have a session that will first check if there is any data available, and only then run the data extraction.
Maciejg
  • 3,088
  • 1
  • 17
  • 30