1

I have a job that will Collect Data from FTP using Get a file with FTP and I want it's only collect yesterday file or older than n day or base on specific date.

How do that? Is any way or possible?

What I know is Get a file with FTP only copy file directly from FTP to destination folder. So, I can't use any field and assign it into JavaScript variable to create condition.

My requirement is moving only yesterday or ... file from FTP into Location I need, not all of them because I have a lot of file about 30K-40K with various file size and it will took a lot of time if I do that. Below is the pic what I have design.

What I have create

Community
  • 1
  • 1
Rio Odestila
  • 125
  • 2
  • 19

2 Answers2

1

There a Scripting/Shell job entry on which you can put any shell script, including :

find . -mindepth 1 -maxdepth 1 -mtime -7  -exec mv -t /destination/path {} +

For eplanation on the shell script have a look there : https://unix.stackexchange.com/questions/207679/moving-files-modified-after-a-specific-date

AlainD
  • 6,187
  • 3
  • 17
  • 31
0

By using the 'Get File Names' step in a transformation, you can access your FTP files (via VFS) and their atributes, namely the 'lastmodifiedtime'.

With this information you can do a simple filter by dates, and only download the files which are older than N days, or any other filter you require. With that in hand you can move, download or any other file related action you desire.

Cristian Curti
  • 1,004
  • 1
  • 7
  • 13
  • It's work, I can access FTP from transformation now. Thanks. Now my question is if someone in my team change the yesterday file, today and then the attribute also change, right? Well, it will cause the `lastmodifieddate` attribute value also change into current date and the file can not be downloaded since I used that attribute and also it does not meet my requirement. NB:Sorry, my writing skill is bad – Rio Odestila Jul 17 '18 at 02:03
  • You are correct, if someone opens and saves the file, the lastmodified time changes. but if the filename has a date / timestamp you can use that for testing. – Cristian Curti Jul 18 '18 at 03:40
  • It have not but I change my CSV standard name and work good so far just sometime the FTP acccess via VFS using that `Get File Name` or `Text File Input` shows error about `Could not find ....csv from path ftp//....... Wildcard *\.csv` And sometime it just work. Idk why it's happen. – Rio Odestila Jul 18 '18 at 04:29
  • I would need the full error description to help you. Or a screenshot – Cristian Curti Jul 18 '18 at 13:06
  • Like this `https://imgur.com/a/O6ZpzF4`. It's first run, if I run it again for second time it work fine. It's problem for me because if I want make it automatically run using `Task Scheduler`, I should run it twice. – Rio Odestila Jul 19 '18 at 01:41
  • 1
    I used to have a problem with this before, for me, adding the parameter vfs.ftp.PassiveMode with Value true fixed it, add this parameter to the KTR where you use the Get File Names – Cristian Curti Jul 19 '18 at 17:09