In Talend how to fetch latest file

Question

I have three files which are having the same schema,

A1(file) received at 12:30:000.00,
A2(file) received at 12:35:000.00,
A3(file) received at 12:40:000.00.

Now I want to fetch the latest file which is A3.

Note: I have used to tfilelist component to fetch the file.

score 2 · Answer 1 · answered Nov 25 '17 at 11:56

2

Order by:

By modified date: most recent to least recent or least recent to most recent.

The Talend Knowledge Base has a load of information about components. Also, the components speak mostly for themselves if you examine them a bit.

answered Nov 25 '17 at 11:56

tobi6

Hi Tobi, Assume that in an existing table, A1 and A2 file records are there, now I want to load only A3 file records. – Naresh AR Nov 25 '17 at 18:25
Hey, you could use something like tAggregateRow. – tobi6 Nov 27 '17 at 08:09

score 1 · Answer 2 · answered Nov 26 '17 at 14:04

1

tFileList --> tFileProperties --> tJavaRow

tFileList to iterate over the file list
tFileProperties to get files properties
tJavaRow to save the filepath (using a global variable) for the file with the greatest value for mtime field

After that, tFileInputDelimited using the global variable for filename

answered Nov 26 '17 at 14:04

TRF

You may order the file list to get the most recent 1st, then in tJavaRow, set the global variable only if it is null to get the 1st filepath only. Does this help or not. A feedback would be appreciate. – TRF Nov 28 '17 at 09:23
As you said in above scenario, I have been trying to implement, but I couldn't able to get the results. 1.In tFileList I've selected the path. 2.I have Iterate to the tfile properties. 3.From tfile properties(In this component do I need to give the file name) to tJavaRow, In this component what should, I need to give??? 4.tfileinputdelimeter, I have given a current file path, and an input file delimiter it won't accept the main link. – Naresh AR Dec 11 '17 at 11:11

score 0 · Answer 3 · edited Mar 27 '18 at 07:07

You can create a job with these components:

tFileList -> tFileProperties -> tAggregateRow -> tLogRow (or any output component)

In tFileList provide the Directory Path.

tFileProperties contains schema corresponding to the properties of the file like basename, Modified time, Absolute Path etc.

In tFileProperties pass the global variable for the filepath ie ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH")).

In tAggregaterow under Operations section select the columns to be displayed & use Max function for mtime_string column.

score 0 · Answer 4 · answered Jul 24 '23 at 18:38

0

Talend - Fetch latest file from each directory

Please check the YouTube for reference and below are the high-level steps.

Get the list of directories using tFilelist.
For each directory - List all the files using tFilelist
Assign numbers of each file in the folder - Using sequence number
Fetch the latest file with seq no 1 and write the file path to new file using tFileoutputdelimited
Read the file name which has the file path for processing and process each file using tFileinputdelimited

answered Jul 24 '23 at 18:38

Anand Venkatesh

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jul 31 '23 at 06:48

4 Answers4